1 =================================================================
5 =================================================================
14 - 4. Open-iSCSI daemon
15 - 5. Open-iSCSI Configuration Utility
18 - 8. Advanced Configuration
19 - 9. iSCSI System Info
25 This file describes the Linux* Open-iSCSI Initiator. The software was
26 tested on AMD Opteron (TM) and Intel Xeon (TM).
28 The latest development release is available at:
29 http://www.open-iscsi.org
31 For questions, comments, contributions send e-mail to:
32 open-iscsi@googlegroups.com
36 - highly optimized and very small-footprint data path;
37 - persistent configuration database;
38 - SendTargets discovery;
46 Open-iSCSI project is a high-performance, transport independent,
47 multi-platform implementation of RFC3720 iSCSI.
49 Open-iSCSI is partitioned into user and kernel parts.
51 The kernel portion of Open-iSCSI is a from-scratch code
52 licensed under GPL. The kernel part implements iSCSI data path
53 (that is, iSCSI Read and iSCSI Write), and consists of three
54 loadable modules: scsi_transport_iscsi.ko, libiscsi.ko and iscsi_tcp.ko.
56 User space contains the entire control plane: configuration
57 manager, iSCSI Discovery, Login and Logout processing,
58 connection-level error processing, Nop-In and Nop-Out handling,
59 and (in the future:) Text processing, iSNS, SLP, Radius, etc.
61 The user space Open-iSCSI consists of a daemon process called
62 iscsid, and a management utility iscsiadm.
68 As of today, the Open-iSCSI Initiator requires a host running the
69 Linux operating system with kernel version 2.6.16, or later. 2.6.14 and
70 2.6.15 are partially supported. Known issues with 2.6.14 - .15 support:
72 - If the device is using a write back cache, during session logout
73 the cache sync command will fail.
74 - iscsiadm's -P 3 option will not print out scsi devices.
75 - iscsid will not automatically online devices.
77 You need to enable "Cryptographic API" under "Cryptographic options" in the
78 kernel config. And you must enable "CRC32c CRC algorithm" even if
79 you do not use header or data digests. They are the kernel options,
80 CONFIG_CRYPTO and CONFIG_CRYPTO_CRC32C, respectively.
82 By default the kernel source found at
83 /lib/modules/`uname -a`/build
84 will be used to compile the open-iscsi modules. To specify a different
85 kernel to build against use:
87 make KSRC=<kernel-src>
91 make KSRC=<kernel-src> KARCH="ARCH=um"
93 To compile on SUSE Linux you'll have to use
95 make KSRC=/usr/src/linux \
96 KBUILD_OUTPUT=/usr/src/linux-obj/<arch>/<config>
98 where <config> is the kernel configuration to use (eg. 'smp').
100 For Red Hat/Fedora and Debian distributions open-iscsi can be installed by
101 typing "make install". This will copy iscsid and iscsiadm to /usr/sbin, the
102 init script to /etc/init.d, and the kernel modules: iscsi_tcp.ko,
103 libiscsi_tcp.ko, libiscsi.ko and scsi_transport_iscsi to
104 /lib/modules/`uname -r`/kernel/drivers/scsi/ overwriting existing iscsi modules.
106 For Debian, be sure to install the linux-headers package that
107 corresponds to your kernel in order to compile the kernel modules
108 ('aptitude install linux-headers-`uname -r`'). You may also wish to
109 run 'make -C kernel/ dpkg_divert' before installing kernel modules if
110 you run a Debian-provided kernel. This will use dpkg-divert(8) to
111 move the packaged kernel modules out of the way, and ensure that
112 future kernel upgrades will not overwrite them.
114 Also, please be aware that the compatibility patches that enable these
115 iscsi modules to run on kernels older than 2.6.25 will not update the
116 ib_iser module; you may get warnings related to mismatched symbols on
117 this driver, in which case you'll be unable to load ib_iser and
118 open-iscsi simultaneously.
123 The daemon implements control path of iSCSI protocol, plus some management
124 facilities. For example, the daemon could be configured to automatically
125 re-start discovery at startup, based on the contents of persistent
126 iSCSI database (see next section).
132 Usage: iscsid [OPTION]
134 -c, --config=[path] Execute in the config file (/etc/iscsi/iscsid.conf).
135 -f, --foreground run iscsid in the foreground
136 -d, --debug debuglevel print debugging information
137 -u, --uid=uid run as uid, default is current user
138 -g, --gid=gid run as gid, default is current user group
139 -h, --help display this help and exit
140 -v, --version display version and exit
144 5. Open-iSCSI Configuration Utility
145 ===================================
147 Open-iSCSI persistent configuration is implemented as a DBM database
148 available on all Linux installations.
150 The database contains two tables:
152 - Discovery table (/etc/iscsi/send_targets);
153 - Node table (/etc/iscsi/nodes).
155 The regular place for iSCSI database files: /etc/iscsi/nodes
157 The iscsiadm utility is a command-line tool to manage (update, delete,
158 insert, query) the persistent database.
160 The utility presents set of operations that a user can perform
161 on iSCSI nodes, sessions, connections, and discovery records.
163 Open-iscsi does not use the term node as defined by the iSCSI RFC,
164 where a node is a single iSCSI initiator or target. Open-iscsi uses the
165 term node to refer to a portal on a target, so tools like iscsiadm
166 require that --targetname and --portal argument be used when in node mode.
168 For session mode, a session id (sid) is used. The sid of a session can be
169 found by running iscsiadm -m session -i. The session id is not currently
170 persistent and is partially determined by when the session is setup.
172 Note that some of the iSCSI Node and iSCSI Discovery operations
173 do not require iSCSI daemon (iscsid) loaded.
179 Usage: iscsiadm [OPTION]
181 -m, --mode <op> specify operational mode op =
182 <discovery|discoverydb|fw|iface|host|node>
184 -m discoverydb --type=[type] --interface=[iface...] --portal=[ip:port] \
186 --op=[op]=[NEW | UPDATE | DELETE | NONPERSISTENT] \ --discover
188 This command will use the discovery record settings
189 matching the record with type=type and
190 portal=ip:port]. If a record does not exist, it will
191 create a record using the iscsid.conf discovery
194 By default, it will then remove records for
195 portals no longer returned. And,
196 if a portal is returned by the target, then the
197 discovery command will create a new record or modify
198 an existing one with values from iscsi.conf and the
201 [op] can be passed in multiple times to this
202 command, and it will alter the node DB manipulation.
204 If [op] is passed in and the value is
205 "new", iscsiadm will add records for portals that do
206 not yet have records in the db.
208 If [op] is passed in and the value is
209 "update", iscsiadm will update node records using
210 info from iscsi.conf and the command line for portals
211 that are returned during discovery and have
214 If [op] is passed in and the value is "delete",
215 iscsiadm will delete records for portals that
216 were not returned during discovery.
218 If [op] is passed in and the value is
219 "nonpersistent" iscsiadm will not store
220 the portals found in the node DB. This
221 only useful with the --login command.
223 See the example section for more info.
225 See below for how to setup iscsi ifaces for
226 software iscsi or override the system defaults.
228 Multiple ifaces can be passed in during discovery.
230 For the above commands "print" is optional. If
231 used, N can be 0 or 1.
232 0 = The old flat style of output is used.
233 1 = The tree style with the inteface info is used.
235 If print is not used the old flay style is used.
236 -m discoverydb --interface=[iface...] --type=[type] --portal=[ip:port] \
238 --op=[op]=[NEW | UPDATE | DELETE | NONPERSISTENT] \
241 This works like the previous discoverydb command
242 with the --login argument passed in will also
243 log into the portals that are found.
244 -m discoverydb --portal=[ip:port] --type=[type] \
245 --op=[op] [--name=[name] --value=[value]]
247 Perform specific DB operation [op] for
248 discovery portal. It could be one of:
249 [new], [delete], [update] or [show]. In case of
250 [update], you have to provide [name] and [value]
253 op=NEW will create a new discovery record
254 using the iscsid.conf discovery settings. If it
255 already exists, it will be overwritten using
256 iscsid.conf discovery settings.
258 op=DELETE will delete the discovery record
259 and records for the targets found through
260 that discovery source.
262 op=SHOW will display the discovery record
263 values. The --show arguemnt can be used to
264 force the CHAP passwords to be displayed.
265 -m discovery --type=[type] --interface=iscsi_ifacename \
266 --portal=[ip:port] --login --print=[N] \
267 --op=[op]=[NEW | UPDATE | DELETE | NONPERSISTENT]
268 perform [type] discovery for target portal with
269 ip-address [ip] and port [port].
271 This command will not use the discovery record
272 settings. It will use the iscsid.conf discovery
273 settings and it will overwrite the discovery
274 record with iscsid.conf discovery settings if it
275 exists. By default, it will then remove records for
276 portals no longer returned. And,
277 if a portal is returned by the target, then the
278 discovery command will create a new record or modify
279 an existing one with values from iscsi.conf and the
282 [op] can be passed in multiple times to this
283 command, and it will alter the DB manipulation.
285 If [op] is passed in and the value is
286 "new", iscsiadm will add records for portals that do
287 not yet have records in the db.
289 If [op] is passed in and the value is
290 "update", iscsiadm will update node records using
291 info from iscsi.conf and the command line for portals
292 that are returned during discovery and have
295 If [op] is passed in and the value is "delete",
296 iscsiadm will delete records for portals that
297 were not returned during discovery.
299 If [op] is passed in and the value is
300 "nonpersistent" iscsiadm will not store
301 the portals found in the node DB.
303 See the example section for more info.
305 See below for how to setup iscsi ifaces for
306 software iscsi or override the system defaults.
308 Multiple ifaces can be passed in during discovery.
309 -m discovery --print=[N] display all discovery records from internal
310 persistent discovery database.
312 -m node display all discovered nodes from internal
313 persistent discovery database
314 -m node --targetname=[name] --portal=[ip:port] \
315 --interface=iscsi_ifacename] \
316 [--login|--logout|--rescan|--stats]
317 -m node --targetname=[name] --portal=[ip:port]
318 --interface=[driver,HWaddress] \
319 --op=[op] [--name=[name] --value=[value]]
320 -m node --targetname=[name] --portal=[ip:port]
321 --interface=iscsi_ifacename] \
323 perform specific DB operation [op] for specific
324 interface on host that will connect to portal on
325 target. targetname, portal and interface are optional.
326 See below for how to setup iscsi ifaces for
327 software iscsi or override the system defaults.
330 [new], [delete], [update] or [show]. In case of
331 [update], you have to provide [name] and [value]
333 [delete] - Note that if a session is using the
334 node record, the session will be logged out then
335 the record will be deleted.
337 Print level can be 0 to 1.
339 Rescan will perform a SCSI layer scan of the session
342 Stats prints the iSCSI stats for the session.
343 -m node --logoutall=[all|manual|automatic]
344 Logout "all" the running sessions or just the ones
345 with a node startup value manual or automatic.
346 Nodes marked as ONBOOT are skipped.
347 -m node --loginall=[all|manual|automatic]
348 Login "all" the running sessions or just the ones
349 with a node startup value manual or automatic.
350 Nodes marked as ONBOOT are skipped.
351 -m session display all active sessions and connections
352 -m session --sid=[sid] [ --print=level | --rescan | --logout ]
353 --op=[op] [--name=[name] --value=[value]]
354 perform operation for specific session with
355 session id sid. If no sid is given the operation
356 will be performed on all running sessions if possible.
357 --logout and --op work like they do in node mode,
358 but in session mode targetname and portal info is
361 Print level can be 0 to 2.
362 1 = Print basic session info like node we are
363 connected to and whether we are connected.
364 2 = Print iscsi params used.
365 3 = Print SCSI info like LUNs, device state.
366 If no sid and no operation is given print out the
368 -m iface --interface=iscsi_ifacename --op=[op] [--name=[name] --value=[value]]
370 perform operation on fiven interface with name
373 See below for examples.
374 -m host --host=hostno --print=level
375 Display information for a specific host if hostno
376 is passed in. If no hostno is passed in then info
377 for all hosts is printed.
379 Print level can be 0 to 4.
380 1 = Print info for how like its state, MAC, and
382 2 = Print basic session info for nodes the host
384 3 = Print iscsi params used.
385 4 = Print SCSI info like LUNs, device state.
386 -d, --debug debuglevel print debugging information
387 -V, --version display version and exit
388 -h, --help display this help and exit
391 5.1 iSCSI iface setup
392 =====================
394 The next sections describe how to setup iSCSI ifaces so you can bind
395 a session to a NIC port when using software iscsi (section 5.1.1), and
396 it describes how to setup ifaces for use with offload cards from Chelsio
397 and Broadcm (section 5.1.2).
400 5.1.1 How to setup iSCSI interfaces (iface) for binding
401 =======================================================
403 If you wish to allow the network susbsystem to figure out
404 the best path/NIC to use then you can skip this section. For example
405 if you have setup your portals and NICs on different subnets then
406 this the following is not needed for software iscsi.
409 This feature is experimental. The interface may change. When reporting
410 bugs, if you cannot do a "ping -I ethX target_portal", then check your
411 network settings first. Make sure the rp_filter setting is set to 0 or 2
412 (see Prep section below for more info). If you cannot ping the portal,
413 then you will not be able to bind a session to a NIC.
415 What is a scsi_host and iface for software, hardware and partial
418 Software iscsi, like iscsi_tcp and iser, allocate a scsi_host per session
419 and does a single connection per session. As a result
420 /sys/class_scsi_host and /proc/scsi will report a scsi_host for
421 each connection/session you have logged into. Offload iscsi, like
422 Chelsio cxgb3i, allocates a scsi_host for each PCI device (each
423 port on a HBA will show up as a different PCI device so you get
424 a scsi_host per HBA port).
426 To manage both types of initiator stacks, iscsiadm uses the interface (iface)
427 structure. For each HBA port or for software iscsi for each network
428 device (ethX) or NIC, that you wish to bind sessions to you must create
429 a iface config /etc/iscsi/ifaces.
433 The iface binding feature requires the sysctl setting
434 net.ipv4.conf.default.rp_filter to be set to 0 or 2. This can be set
435 in /etc/sysctl.conf by having the line:
437 net.ipv4.conf.default.rp_filter = N
439 where N is 0 or 2. Note that when setting this you may have to reboot
440 the box for the value to take effect.
443 rp_filter information from Documentation/networking/ip-sysctl.txt:
446 0 - No source validation.
447 1 - Strict mode as defined in RFC3704 Strict Reverse Path
448 Each incoming packet is tested against the FIB and if the interface
449 is not the best reverse path the packet check will fail.
450 By default failed packets are discarded.
451 2 - Loose mode as defined in RFC3704 Loose Reverse Path
452 Each incoming packet's source address is also tested against the FIB
453 and if the source address is not reachable via any interface
454 the packet check will fail.
460 iface0 qla4xxx,00:c0:dd:08:63:e8,20.15.0.7,default,iqn.2005-06.com.redhat:madmax
461 iface1 qla4xxx,00:c0:dd:08:63:ea,20.15.0.9,default,iqn.2005-06.com.redhat:madmax
463 Will report iface configurations that are setup in /etc/iscsi/ifaces.
466 iface_name transport_name,hwaddress,ipaddress,net_ifacename,initiatorname
468 For software iscsi, you can create the iface configs by hand, but it is
469 reccomended that you use iscsiadm's iface mode. There is a iface.example in
470 /etc/iscsi/ifaces which can be used as a template for the daring.
472 For each network object you wish to bind a session to you must create
473 a seperate iface config in /etc/iscsi/ifaces and each iface config file
474 must have a unique name which is less than or equal to 64 characters.
478 If you have NIC1 with MAC address 00:0F:1F:92:6B:BF and NIC2 with
479 MAC address 00:C0:DD:08:63:E7 and you wanted to do software iscsi over
480 TCP/IP. Then in /etc/iscsi/ifaces/iface0 you would enter:
482 iface.transport_name = tcp
483 iface.hwaddress = 00:0F:1F:92:6B:BF
485 and in /etc/iscsi/ifaces/iface1 you would enter:
487 iface.transport_name = tcp
488 iface.hwaddress = 00:C0:DD:08:63:E7
490 Warning: Do not name a iface config file "default" or "iser".
491 They are special value/file that is used by the iscsi tools for
492 backward compatibility. If you name a iface default or iser, then
493 the behavior is not defined.
495 To use iscsiadm to create iface0 above for you run:
497 (This will create a new empty iface config. If there was already a iface
498 with the name "iface0" this command will overwrite it.)
499 # iscsiadm -m iface -I iface0 --op=new
501 (This will set the hwaddress.)
502 # iscsiadm -m iface -I iface0 --op=update -n iface.hwaddress -v 00:0F:1F:92:6B:BF
504 If you had sessions logged in iscsiadm will not update, overwrite
505 a iface. You must log out first. If you have a iface bound to a node/portal
506 but you have not logged in then, iscsiadm will update the config and
507 all existing bindings.
510 You should now skip to 5.1.3 to see how to log in using the iface and for
511 some helpful management commands.
515 5.1.2 Setting up a iface for a iSCSI offload card
516 =================================================
518 This section describes how to setup ifaces for use with Chelsio
521 By default, iscsiadm will create a iface for each Broadcom and Chelsio
522 port. The iface name will be of the form:
524 $transport/driver_name.$MAC_ADDRESS
529 default tcp,<empty>,<empty>,<empty>,<empty>
530 iser iser,<empty>,<empty>,<empty>,<empty>
531 cxgb3i.00:07:43:05:97:07 cxgb3i,00:07:43:05:97:07,<empty>,<empty>,<empty>
534 Will report iface configurations that are setup in /etc/iscsi/ifaces.
537 iface_name transport_name,hwaddress,ipaddress,net_ifacename,initiatorname
539 iface_name: name of iface
540 transport_name: name of driver
541 hwaddress: MAC address
542 ipaddress: IP address to use for this port
543 net_iface_name: Net_ifacename will be <empty> because change between
544 reboots. It is used for software iSCSI's vlan or alias binding.
545 initiatorname: Initiatorname to be used if you want to override the
546 default one in /etc/iscsi/initiatorname.iscsi.
550 To display these values in a more friendly run:
552 iscsiadm -m iface -I cxgb3i.00:07:43:05:97:07
553 # BEGIN RECORD 2.0-871
554 iface.iscsi_ifacename = cxgb3i.00:07:43:05:97:07
555 iface.net_ifacename = <empty>
556 iface.ipaddress = <empty>
557 iface.hwaddress = 00:07:43:05:97:07
558 iface.transport_name = cxgb3i
559 iface.initiatorname = <empty>
563 Before you can use the iface, you must set the IP address for the port
564 with the following command:
566 iscsiadm -m iface -I cxgb3i.00:07:43:05:97:07 -o update -n iface.ipaddress -v 20.15.0.66
570 For the name of the value we want to update we use the name from
571 the "iscsiadm -m iface -I cxgb3i.00:07:43:05:97:07" command which is
575 Now, we can use this iface to login into targets, which is described in the
579 5.1.3 Discoverying iSCSI targets/portals
580 ========================================
582 Be aware that iscsiadm will use the default route to do discovery. It will
583 not use the iface specified. So if you are using a offload card, you will
584 need a seperate network connection to the target for discovery purposes.
585 *This will be fixed in the next version of open-iscsi*
587 For compatibility reasons, when you run iscsiadm to do discovery, it
588 will check for interfaces in /etc/iscsi/iscsi/ifaces that are using
589 tcp for the iface.transport and it will bind the portals that are discovered
590 so that they will be logged in through those ifaces. This behavior can also
591 be overriden by passing in the interfaces you want to use. For the case
592 of offload like with cxgb3i and bnx2i this is required because the transport
595 For example if you had defined two interface but only wanted to use one
596 you can use the --interface/-I argument:
598 iscsiadm -m discoverydb -t st -p ip:port -I iface1 --discover -P 1
600 If you had defined interfaces but wanted the old behavior, where
601 we do not bind a session to a iface, then you can use the special iface
604 iscsiadm -m discoverydb -t st -p ip:port -I default --discover -P 1
606 And if you did not define any interfaces in /etc/iscsi/ifaces and do
607 not pass anything into iscsiadm, running iscsiadm will do the default
608 behavior, where we allow the network subsystem to decide which
611 If you later want to remove the bindings for a specific target and
612 iface then you can run:
614 iscsiadm -m node -T my_target -I iface0 --op=delete
616 To do this for a specific portal on a target run:
618 iscsiadm -m node -T my_target -p ip:port -I iface0 --op=delete
620 If you wanted to delete all bindinds for iface0 then you can run
622 iscsiadm -m node -I iface0 --op=delete
624 And for equalogic targets it is sometimes useful to remove by just portal
626 iscsiadm -m node -p ip:port -I iface0 --op=delete
629 To now log into targets it is the same as with sofware iscsi. See section
630 7 for how to get started.
635 5.2 iscsiadm examples
636 =====================
637 Usage examples using the one-letter options (see iscsiadm man page
642 - SendTargets iSCSI Discovery using the default driver and interface and
643 using the discovery settings for the discovery record with the
644 ID [192.168.1.1:3260].
646 ./iscsiadm -m discoverydb -t st -p 192.168.1.1:3260 --discover
648 This will search /etc/iscsi/send_targets for a record with the
649 ID [portal = 192.168.1.1:3260 and type = sendtargets. If found it
650 will perform discovery using the settings stored in the record.
651 If a record does not exist, it will be created using the iscsid.conf
654 The argument to -p may also be a hostname instead of an address.
655 ./iscsiadm -m discoverydb -t st -p smoehost --discover
657 For the ifaces, iscsiadm will first search /etc/iscsi/ifaces for
658 interfaces using software iscsi. If any are found then nodes found
659 during discovery will be setup so that they can logged in through
660 those interfaces. To specify a specific iface, pass the
661 -I argument for each iface.
663 - SendTargets iSCSI Discovery updating existing target records:
665 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 \
668 If there is a record for targetX and portalY exists in the DB, and
669 is returned during discovery, it will be updated with the info
670 from the iscsi.conf. No new portals will be added and stale
671 portals will not be removed.
673 - SendTargets iSCSI Discovery deleting existing target records:
675 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 \
678 If there a record for targetX and portalY exists in the DB, but
679 is not returned during discovery it will be removed from the DB.
680 No new portals will be added and existing portal records will not
683 Note: If a session is logged into portal we are going to delete
684 a record for, it will be logged out then the record will be
687 - SendTargets iSCSI Discovery adding new records:
689 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 \
692 If there targetX and portalY is returned during discovery and does
693 not have a record, it will be added. Existing records are not
696 - SendTargets iSCSI Discovery using multiple ops:
698 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 \
699 -o new -o delete --discover
701 This command will add new portals and delete records for portals
702 no longer returned. It will not change the record information for
705 - SendTargets iSCSI Discovery in nonpersistent mode:
707 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 \
708 -o nonpersistent --discover
710 This command will perform discovery, but not manipulate the node DB.
712 - SendTargets iSCSI Discovery with a specific interface. If you
713 wish to only use a subset of the interfaces in /etc/iscsi/ifaces
714 then you can pass them in during discovery:
716 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 \
717 --interface=iface0 --interface=iface1 --discover
719 Note that for software iscsi, we let the network layer select
720 which NIC to use for discovery, but for later logins iscsiadm
721 will use the NIC defined in the iface config.
723 qla4xxx support is very basic and experimental. It does not store
724 the record info in the card's FLASH or the node DB, so you must
725 rerun discovery every time the driver is reloaded.
727 - Manipulate SendTargets DB.
729 Create new SendTargets discovery record or overwrite an existing
730 discovery record with iscsid.conf discovery settings.
731 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 -o new
733 See discovery settings.
734 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 -o show
736 See hidden discovery settings like CHAP passwords
737 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 \
740 Set discovery setting.
741 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 \
742 -o update -n name -v value
744 Delete discovery record. This will also delete the records for
745 the targets found through the discovery source.
746 ./iscsiadm -m discoverydb -t sendtargets -p 192.168.1.1:3260 -o delete
748 Node mode. In node mode you can specify which records you want to log
749 into by specifying the targetname, ip address, port or interface
750 (if specifying the interface it must already be setup in the node db).
751 iscsiadm will search the node db, for records which match the values
752 you pass in, so if you pass in the targetname and interface, iscsiadm
753 will search for records with those values and operate on only them.
754 Passing in none of them will result in all node records being operated on.
756 - iSCSI Login to all portals on every node/starget through each interface
759 ./iscsiadm -m node -l
761 - iSCSI login to all portals on a node/target through each interface set
764 ./iscsiadm -m node -T iqn.2005-03.com.max -l
766 - iSCSI login to a specific portal through each interface set in the db:
768 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 -l
770 To specify a IPv6 address the following can be used:
772 ./iscsiadm -m node -T iqn.2005-03.com.max \
773 -p 2001:c90::211:9ff:feb8:a9e9 -l
775 The above command would use the default port, 3260. To specify a
776 port use the following:
778 ./iscsiadm -m node -T iqn.2005-03.com.max \
779 -p [2001:c90::211:9ff:feb8:a9e9]:3260 -l
781 To specify a hostname the following can be used:
783 ./iscsiadm -m node -T iqn.2005-03.com.max -p somehost -l
785 - iSCSI Login to a specific portal through the NIC setup as iface0:
787 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 \
790 - iSCSI Logout to all portals on every node/starget through each interface
793 ./iscsiadm -m node -u
795 Warning: this does not check startup values like the logout/login all
796 option. Do not use this if you are running iscsi on your root disk.
798 - iSCSI logout to all portals on a node/target through each interface set
801 ./iscsiadm -m node -T iqn.2005-03.com.max -u
803 - iSCSI logout to a specific portal through each interface set in the db:
805 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 -u
807 - iSCSI Logout to a specific portal through the NIC setup as iface0:
809 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 \
812 - Changing iSCSI parameter:
814 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 \
815 -o update -n node.cnx[0].iscsi.MaxRecvDataSegmentLength -v 65536
817 You can also change paramaters for multiple records at once, by
818 specifying different combinations of the target, portal and interface
821 - Adding custom iSCSI portal:
823 ./iscsiadm -m node -o new -T iqn.2005-03.com.max \
824 -p 192.168.0.1:3260,2 -I iface4
826 The -I/--interface is optional. If not passed in, "default" is used.
827 For tcp or iser, this would allow the network layer to decide what is
830 Note that for this command the target portal group tag (TPGT) should
831 be passed in. If it is not passed in on the initial creation command
832 then the user must run iscsiadm again to set the value. Also
833 if the TPGT is not initially passed in, the old behavior of not
834 tracking whether the record was statically or dynamically created
837 - Adding custom NIC config to multiple targets:
839 ./iscsiadm -m node -o new -I iface4
841 This command will add a interface config using the iSCSI and SCSI
842 settings from iscsid.conf to every target that is in the node db.
844 - Removing iSCSI portal:
846 ./iscsiadm -m node -o delete -T iqn.2005-03.com.max -p 192.168.0.4:3260
848 You can also delete multiple records at once, by specifying different
849 combinations of the target, portal and interface like above.
851 - Display iSCSI portal onfiguration:
853 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260
857 ./iscsiadm -m node -o show -T iqn.2005-03.com.max -p 192.168.0.4:3260
858 You can also display multiple records at once, by specifying different
859 combinations of the target, portal and interface like above.
861 Note: running "iscsiadm -m node" will only display the records. It
862 will not display the configuration info. You must run,
863 "iscsiadm -m node -o show".
865 - Show all node records:
869 This will print the nodes using the old flat format where the
870 interface and driver are not displayed. To display that info
871 use the -P argument with the arguent "1":
873 ./iscsiadm -m node -P 1
875 - Show all records in discovery database:
877 ./iscsiadm -m discovery
879 - Show all records in discovery database and show the targets that
880 were discovered from each record:
882 ./iscsiadm -m discovery -P 1
884 - Display session statistics:
886 ./iscsiadm -m session -r 1 --stats
888 This function also works in node mode. Instead of the "-r $sid"
889 argument, you would pass in the node info like targetname and/or portal,
892 - Perform a SCSI scan on a session
894 ./iscsiadm -m session -r 1 --rescan
896 This function also works in node mode. Instead of the "-r $sid"
897 argument, you would pass in the node info like targetname and/or portal,
900 Note: Rescanning does not delete old LUNs. It will only pick up new
903 - Display running sessions:
905 ./iscsiadm -m session -P 1
910 The default configuration file is /etc/iscsi/iscsid.conf. This file contains
911 only configuration that could be overwritten by iSCSI Discovery,
912 or manualy updated via iscsiadm utility. Its OK if this file does not
913 exist in which case compiled-in default configuration will take place
914 for newer discovered Target nodes.
916 See the man page and the example file for the current syntax.
917 The manpages for iscsid, iscsiadm are in the doc subdirectory and can be
918 installed in the appropriate man page directories and need to be manually
919 copied into e.g. /usr/local/share/man8.
923 There are three steps needed to set up a system to use iSCSI storage:
924 7.1. iSCSI startup using the init script or manual startup.
925 7.2. Discover targets.
926 7.3. Automate target logins for future system reboots.
928 The init scripts will start the iSCSI daemon and log into any
929 portals that are set up for automatic login (discussed in 7.2)
930 or discovered through the discover daemon iscsid.conf params
931 (discussed in 7.1.2).
933 If your distro does not have a init script, then you will have to start the
934 daemon and log into the targets manually.
937 7.1.1 iSCSI startup using the init script
938 -----------------------------------------------
942 To start open-iscsi in Red Hat/Fedora you can do:
944 service open-iscsi start
946 To get open-iscsi to automatically start at run time you may have to
948 chkconfig --level <levels> open-iscsi on
949 Where <levels> are the run levels.
951 And, to automatically mount a file system during startup
952 you must have the partition entry in /etc/fstab marked with the "_netdev"
953 option. For example this would mount a iscsi disk sdb:
955 /dev/sdb /mnt/iscsi ext3 _netdev 0 0
959 Otherwise, if there is a initd script for your distro in etc/initd that
960 gets installed with "make install"
962 /etc/init.d/open-iscsi start
964 will usually get you started.
967 7.1.2 Manual Startup:
968 ---------------------
970 7.1.2.1 Starting up the iSCSI daemon (iscsid) and loading modules:
971 -----------------------------------------------------------------
972 If there is no initd script, you must start the tools by hand. First load the
975 modprobe -q iscsi_tcp
977 after that start iSCSI daemon process:
981 or alternatively, start it with debug enabled and with output
982 redirected to the current console:
986 7.1.2.2 Logging into Targets:
987 ---------------------------
988 Use the configuration utility, iscsiadm, to add/remove/update Discovery
989 records, iSCSI Node records or monitor active iSCSI sessions (see above or the
990 iscsiadm man files and see section 7.2 below for how to discover targets).
994 will print out the nodes that have been discovered as:
996 10.15.85.19:3260,3 iqn.1992-08.com.netapp:sn.33615311
997 10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311
1001 ip:port,target_portal_group_tag targetname
1003 If you are using the iface argument or want to see the driver
1004 info use the following:
1006 ./iscsiadm -m node -P 1
1008 Target: iqn.1992-08.com.netapp:sn.33615311
1009 Portal: 10.15.84.19:3260,2
1011 Portal: 10.15.85.19:3260,3
1017 Portal ip_address:port,tpgt
1020 where targetname is the name of the target and ip_address:port is the address
1021 and port of the portal. tpgt, is the portal group tag of
1022 the portal, and is not used in iscsiadm commands except for static
1023 record creation. And iface name is the name of the iscsi interface
1024 defined in /etc/iscsi/ifaces. If no interface was defined in
1025 /etc/iscsi/ifaces or passed in, the default behavior is used.
1026 Default here is iscsi_tcp/tcp to be used over which ever NIC the
1027 network layer decides is best.
1029 To login, take the ip, port and targetname from above and run:
1031 ./iscsiadm -m node -T targetname -p ip:port -l
1033 In this example we would run
1035 ./iscsiadm -m node -T iqn.1992-08.com.netapp:sn.33615311 -p 10.15.84.19:3260 -l
1037 Note: drop the portal group tag from the "iscsiadm -m node" output.
1039 7.2. Discover Targets
1040 ---------------------
1041 Once the iSCSI service is running, you can perform discovery using
1044 iscsiadm -m discoverydb -t sendtargets -p ip:port --discover
1046 where "ip" is the address of the portal and port is the port.
1048 To use iSNS you can run the discovery command with the type as "isns"
1049 and pass in the ip:port:
1051 iscsiadm -m discoverydb -t isns -p ip:port --discover
1053 Both commands will print out the list of all discovered targets and their
1056 # iscsiadm -m discoverydb -t st -p 10.15.85.19:3260 --discover
1057 10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311
1058 10.15.85.19:3260,3 iqn.1992-08.com.netapp:sn.33615311
1060 The format for the output is:
1062 ip:port,tpgt targetname
1064 In this example, for the first target the ip address is 10.15.85.19.
1065 The port is 3260. The target portal group is 3. The target name
1066 is iqn.1992-08.com.netapp:sn.33615311.
1068 If you would also like to see the iscsi inteface which will be used
1069 for each session then use the --print[N] option.
1071 iscsiadm -m discoverydb -t sendtargets -p ip:port -P 1 --discover
1074 Target: iqn.1992-08.com.netapp:sn.33615311
1075 Portal: 10.15.84.19:3260,2
1077 Portal: 10.15.85.19:3260,3
1080 In this example, The IP address of the first portal is 10.15.84.19.
1081 The port is 3260. The target portal group is 3. The target name
1082 is iqn.1992-08.com.netapp:sn.33615311. The iface being used is iface2.
1084 While discovery targets are kept in the discovery db, they are
1085 useful only for re-discovery. The discovered targets (a.k.a. nodes)
1086 are stored as records in the node db.
1088 The discovered targets are not logged into yet. Rather than logging
1089 into the discovered nodes (making LUs from those nodes available as
1090 storage), it is better to automate the login to the nodes we need.
1092 If you wish to log into a target manually now, see section
1093 "7.1.2.2 Logging in targets" above.
1095 7.3. Automate Target Logins for Future System Statups
1096 -----------------------------------------------------
1097 Note: this may only work for distros with init scripts.
1099 To automate login to a node, use the following with the record ID
1100 (record ID is the targetname and portal) of the node discovered in the
1102 iscsiadm -m node -T targetname -p ip:port --op update -n node.startup -v automatic
1104 To set the automatic setting to all portals on a target through every
1105 interface setup for each protal, the following can be run:
1106 iscsiadm -m node -T targetname --op update -n node.startup -v automatic
1108 Or to set the "node.startup" attribute to "startup" as default for
1109 all sessions add the following to the /etc/iscsi/iscsid.conf:
1111 node.startup = automatic
1113 Setting this in iscsid.conf, will not affect existing nodes. It will only
1114 affect nodes that are discovered after setting the value.
1116 To login to all the automated nodes, simply restart the iscsi service:
1117 e.g /etc/init.d/open-iscsi restart. On your next startup the nodes will
1118 be logged into autmotically.
1121 7.4 Automatic Discovery and Login
1122 -----------------------------------
1124 Instead of running the iscsiadm discovery command and editing the
1125 startup setting, iscsid can be configured so that every X seconds
1126 it performs discovery and logs in and out of the portals return or
1127 no longer returned. In this mode, when iscsid starts it will check the
1128 discovery db for iSNS records with:
1130 discovery.isns.use_discoveryd = Yes
1132 and it will check for SendTargets discovery records that have the setting:
1134 discovery.sendtargets.use_discoveryd = Yes
1136 If set, iscsid will perform discovery to the address every
1137 discovery.isns.discoveryd_poll_inval or
1138 discovery.sendtargets.discoveryd_poll_inval seconds,
1139 and it will log into any portals found from the discovery source using
1140 the ifaces in /etc/iscsi/ifaces.
1142 Note that for iSNS the poll_interval does not have to be set. If not set,
1143 iscsid will only perform rediscovery when it gets a SCN from the server.
1146 # For servers like Microsofts where they allow SCN registrations, but do not
1147 # send SCN events, discovery.isns.poll_interval should be set to a non zero
1148 # value to auto discover new targets. This is also useful for servers like
1149 # linux-isns (SLES's iSNS server) where it sometimes does not send SCN
1150 # events in the proper format, so they may not get handled.
1158 - Create a SendTargets record by passing iscsiadm the "-o new" argument in
1160 # iscsiadm -m discoverydb -t st -p 20.15.0.7:3260 -o new
1161 New discovery record for [20.15.0.7,3260] added.
1163 - Set the use_discoveryd setting for the record.
1164 # iscsiadm -m discoverydb -t st -p 20.15.0.7:3260 -o update -n discovery.sendtargets.use_discoveryd -v Yes
1166 - Set the polling interval.
1167 # iscsiadm -m discoverydb -t st -p 20.15.0.7:3260 -o update -n discovery.sendtargets.discoveryd_poll_inval -v 30
1169 To have the new settings take effect restart iscsid by restarting the
1173 When iscsiadm is run with the -o new argument, it will use the
1174 discovery.sendtargets.use_discoveryd and
1175 discovery.sendtargets.discoveryd_poll_inval
1176 settings in iscsid.conf for the records initial settings. So if those
1177 are set in iscsid.conf, then you can skip the iscsiadm -o update
1183 - Create a iSNS record by passing iscsiadm the "-o new" argument in
1185 # iscsiadm -m discoverydb -t isns -p 20.15.0.7:3205 -o new
1186 New discovery record for [20.15.0.7,3205] added.
1188 - Set the use_discoveryd setting for the record.
1189 # iscsiadm -m discoverydb -t isns -p 20.15.0.7:3205 -o update -n discovery.isns.use_discoveryd -v Yes
1191 - [OPTIONAL: see iSNS note above] Set the polling interval if needed.
1192 # iscsiadm -m discoverydb -t st -p 20.15.0.7:3205 -o update -n discovery.isns.discoveryd_poll_inval -v 30
1194 To have the new settings take effect restart iscsid by restarting the
1198 When iscsiadm is run with the -o new argument, it will use the
1199 discovery.isns.use_discoveryd and discovery.isns.discoveryd_poll_inval
1200 settings in iscsid.conf for the records initial settings. So if those
1201 are set in iscsid.conf, then you can skip the iscsiadm -o update
1205 8. Advanced Configuration
1206 =========================
1208 8.1 iSCSI settings for dm-multipath
1209 -----------------------------------
1211 When using dm-multipath, the iSCSI timers should be set so that commands
1212 are quickly failed to the dm-multipath layer. For dm-multipath you should
1213 then set values like queue if no path, so that IO errors are retried and
1214 queued if all paths are failed in the multipath layer.
1217 8.1.1 iSCSI ping/Nop-Out settings
1218 ---------------------------------
1219 To quickly detect problems in the network, the iSCSI layer will send iSCSI
1220 pings (iSCSI NOP-Out requests) to the target. If a NOP-Out times out the
1221 iSCSI layer will respond by failing running commands and asking the SCSI
1222 layer to requeue them if possible (SCSI disk commands get 5 retries if not
1223 using multipath). If dm-multipath is being used the SCSI layer will fail
1224 the command to the multipath layer instead of retrying. The multipath layer
1225 will then retry the command on another path.
1227 To control how often a NOP-Out is sent the following value can be set:
1229 node.conn[0].timeo.noop_out_interval = X
1231 Where X is in seconds and the default is 10 seconds. To control the
1232 timeout for the NOP-Out the noop_out_timeout value can be used:
1234 node.conn[0].timeo.noop_out_timeout = X
1236 Again X is in seconds and the default is 15 seconds.
1238 Normally for these values you can use:
1240 node.conn[0].timeo.noop_out_interval = 5
1241 node.conn[0].timeo.noop_out_timeout = 10
1243 If there are a lot of IO error messages, then the above values may be too
1244 aggressive and you may need to increase the values for your network conditions
1245 and workload, or you may need to check your network for possible problems.
1248 8.1.2 replacement_timeout
1249 -------------------------
1250 The next iSCSI timer that will need to be tweaked is:
1252 node.session.timeo.replacement_timeout = X
1254 Here X is in seconds.
1256 replacement_timeout will control how long to wait for session re-establishment
1257 before failing pending SCSI commands and commands that are being operated on by
1258 the SCSI layer's error handler up to a higher level like multipath or to
1259 an application if multipath is not being used.
1262 8.1.2.1 Running Commands, the SCSI Error Handler, and replacement_timeout
1263 -------------------------------------------------------------------------
1264 Remember, from the Nop-out discussion that if a network problem is detected,
1265 the running commands are failed immediately. There is one exception to this
1266 and that is when the SCSI layer's error handler is running. To check if
1267 the SCSI error handler is running iscsiadm can be run as:
1269 iscsiadm -m session -P 3
1273 Host Number: X State: Recovery
1275 When the SCSI EH is running, commands will not be failed until
1276 node.session.timeo.replacement_timeout seconds.
1278 To modify the timer that starts the SCSI EH, you can either write
1279 directly to the device's sysfs file:
1281 echo X > /sys/block/sdX/device/timeout
1283 where X is in seconds or on most distros you can modify the udev rule.
1285 To modify the udev rule open /etc/udev/rules.d/50-udev.rules, and find the
1288 ACTION=="add", SUBSYSTEM=="scsi" , SYSFS{type}=="0|7|14", \
1289 RUN+="/bin/sh -c 'echo 60 > /sys$$DEVPATH/timeout'"
1291 And change the echo 60 part of the line to the value that you want.
1293 The default timeout for normal File System commands is 30 seconds when udev
1294 is not being used. If udev is used the default is the above value which
1295 is normally 60 seconds.
1298 8.1.2.2 Pending Commands and replacement_timeout
1299 ------------------------------------------------
1300 Commonly, the SCSI/BLOCK layer will queue 256 commands, but the path can
1301 only take 32. When a network problem is detected, the 32 commands
1302 in flight will be sent back to the SCSI layer immediately and because
1303 multipath is being used this will cause the commands to be sent to the multipath
1304 layer for execution on another path. However the other 96 commands that were
1305 still in the SCSI/BLOCK queue, will remain here until the session is
1306 re-established or until node.session.timeo.replacement_timeout seconds has
1307 gone by. After replacement_timeout seconds, the pending commands will be
1308 failed to the multipath layer, and all new incoming commands will be
1309 immediately failed back to the multipath layer. If a session is later
1310 re-established, then new commands will be queued and executed. Normally,
1311 multipathd's path tester mechanism will detect that the session has been
1312 re-established and the path is accessible again, and it will inform
1316 8.1.3 Optimal replacement_timeout Value
1317 ---------------------------------------
1319 The default value for replacement_timeout is 120 seconds, but because
1320 multipath's queue_if_no_path and no_path_retry setting can prevent IO errors
1321 from being propagated to the application, replacement_timeout can be set to a
1322 shorter value like 5 to 15 seconds. By setting it lower pending IO is quickly
1323 sent to a new path and executed while the iSCSI layer attempts
1324 re-establishment of the session. If all paths end up being failed, then the
1325 multipath and device mapper layer will internally queue IO based on the
1326 multipath.conf settings, instead of the iSCSI layer.
1329 8.2 iSCSI settings for iSCSI root
1330 ---------------------------------
1332 When accessing the root partition directly through a iSCSI disk, the
1333 iSCSI timers should be set so that iSCSI layer has several chances to try to
1334 re-establish a session and so that commands are not quickly requeued to
1335 the SCSI layer. Basically you want the opposite of when using dm-multipath.
1337 For this setup, you can turn off iSCSI pings by setting:
1339 node.conn[0].timeo.noop_out_interval = 0
1340 node.conn[0].timeo.noop_out_timeout = 0
1342 And you can turn the replacement_timer to a very long value:
1344 node.session.timeo.replacement_timeout = 86400
1347 9. iSCSI System Info
1348 ====================
1350 To get information about the running sessions: including the session and
1351 device state, session ids (sid) for session mode, and some of the
1352 negotiated parameters, run:
1354 iscsiadm -m session -P 2
1356 If you are looking for something shorter like just the sid to node mapping
1359 iscsiadm -m session -P 0
1363 This will print the list of running sessions with the format:
1365 driver [sid] ip:port,target_portal_group_tag targetname
1367 # iscsiadm -m session
1368 tcp [2] 10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311
1369 tcp [3] 10.15.85.19:3260,3 iqn.1992-08.com.netapp:sn.33615311
1371 To print the hw address info use the -P option with "1":
1373 iscsiadm -m session -P 1
1375 This will print the sessions with the following format:
1377 Current Portal: portal currently logged into
1378 Persistent Portal: portal we would fall back to if we had got redirected during login
1379 Iface Transport: driver/transport_name
1380 Iface IPaddress: IP address of iface being used
1381 Iface HWaddress: HW address used to bind session
1382 Iface Netdev: netdev value used to bind session
1383 SID: iscsi sysfs session id
1384 iSCSI Connection State: iscsi state
1386 Note: if a older kernel is being used or if the session is not bound
1387 then the keyword "default" is print to indicate that the default
1388 network behavior is being used.
1391 #iscsiadm -m session -P 1
1392 Target: iqn.1992-08.com.netapp:sn.33615311
1393 Current Portal: 10.15.85.19:3260,3
1394 Persistent Portal: 10.15.85.19:3260,3
1395 Iface Transport: tcp
1396 Iface IPaddress: 10.11.14.37
1397 Iface HWaddress: default
1398 Iface Netdev: default
1400 iSCSI Connection State: LOGGED IN
1401 Internal iscsid Session State: NO CHANGE
1403 The connection state is currently not available for qla4xxx.
1406 To get a HBA/Host view of the session there is the host mode.
1411 cxgb3i: [7] 10.10.15.51,[00:07:43:05:97:07],eth3 <empty>
1413 This prints the list of iSCSI hosts in the system with the format:
1414 driver [hostno] ipaddress,[hwaddress],net_ifacename,initiatorname
1417 To print this info in a more user friendly way the -P argument can be used:
1419 iscsiadm -m host -P 1
1423 Initiatorname: <empty>
1424 IPaddress: 10.10.15.51
1425 HWaddress: 00:07:43:05:97:07
1428 Here, you can also see the sate of the host.
1430 You can also pass in any value from 1 - 4 to print more info like the
1431 sessions running through the host, what ifaces are being used and what
1432 devices are accessed through it.
1434 To print the info for a specific host then you can pass in the -H argument
1435 with the host number:
1436 iscsiadm -m host -P 1 -H 7