1 .\" $NetBSD: raidctl.8,v 1.55 2008/08/26 21:08:08 oster Exp $
3 .\" Copyright (c) 1998, 2002 The NetBSD Foundation, Inc.
4 .\" All rights reserved.
6 .\" This code is derived from software contributed to The NetBSD Foundation
9 .\" Redistribution and use in source and binary forms, with or without
10 .\" modification, are permitted provided that the following conditions
12 .\" 1. Redistributions of source code must retain the above copyright
13 .\" notice, this list of conditions and the following disclaimer.
14 .\" 2. Redistributions in binary form must reproduce the above copyright
15 .\" notice, this list of conditions and the following disclaimer in the
16 .\" documentation and/or other materials provided with the distribution.
18 .\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
19 .\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
20 .\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
21 .\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
22 .\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
23 .\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
24 .\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
25 .\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
26 .\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
27 .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
28 .\" POSSIBILITY OF SUCH DAMAGE.
31 .\" Copyright (c) 1995 Carnegie-Mellon University.
32 .\" All rights reserved.
34 .\" Author: Mark Holland
36 .\" Permission to use, copy, modify and distribute this software and
37 .\" its documentation is hereby granted, provided that both the copyright
38 .\" notice and this permission notice appear in all copies of the
39 .\" software, derivative works or modified versions, and any portions
40 .\" thereof, and that both notices appear in supporting documentation.
42 .\" CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS"
43 .\" CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND
44 .\" FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
46 .\" Carnegie Mellon requests users of this software to return to
48 .\" Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU
49 .\" School of Computer Science
50 .\" Carnegie Mellon University
51 .\" Pittsburgh PA 15213-3890
53 .\" any improvements or extensions that they make and grant Carnegie the
54 .\" rights to redistribute these changes.
61 .Nd configuration utility for the RAIDframe disk driver
65 .Fl a Ar component Ar dev
68 .Fl A Op yes | no | root
75 .Fl c Ar config_file Ar dev
78 .Fl C Ar config_file Ar dev
81 .Fl f Ar component Ar dev
84 .Fl F Ar component Ar dev
87 .Fl g Ar component Ar dev
96 .Fl I Ar serial_number Ar dev
105 .Fl r Ar component Ar dev
108 .Fl R Ar component Ar dev
120 is the user-land control program for
122 the RAIDframe disk device.
124 is primarily used to dynamically configure and unconfigure RAIDframe disk
126 For more information about the RAIDframe disk device, see
129 This document assumes the reader has at least rudimentary knowledge of
130 RAID and RAID concepts.
132 The command-line options for
135 .Bl -tag -width indent
136 .It Fl a Ar component Ar dev
139 as a hot spare for the device
141 Component labels (which identify the location of a given
142 component within a particular RAID set) are automatically added to the
143 hot spare after it has been used and are not required for
146 .It Fl A Ic yes Ar dev
147 Make the RAID set auto-configurable.
148 The RAID set will be automatically configured at boot
150 the root file system is mounted.
151 Note that all components of the set must be of type
154 .It Fl A Ic no Ar dev
155 Turn off auto-configuration for the RAID set.
156 .It Fl A Ic root Ar dev
157 Make the RAID set auto-configurable, and also mark the set as being
158 eligible to be the root partition.
159 A RAID set configured this way will
161 the use of the boot disk as the root device.
162 All components of the set must be of type
165 Note that only certain architectures
166 .Pq currently alpha, i386, pmax, sparc, sparc64, and vax
167 support booting a kernel directly from a RAID set.
169 Initiate a copyback of reconstructed data from a spare disk to
171 This is performed after a component has failed,
172 and the failed drive has been reconstructed onto a spare drive.
173 .It Fl c Ar config_file Ar dev
174 Configure the RAIDframe device
176 according to the configuration given in
178 A description of the contents of
181 .It Fl C Ar config_file Ar dev
184 but forces the configuration to take place.
185 This is required the first time a RAID set is configured.
186 .It Fl f Ar component Ar dev
187 This marks the specified
189 as having failed, but does not initiate a reconstruction of that component.
190 .It Fl F Ar component Ar dev
193 of the device, and immediately begin a reconstruction of the failed
194 disk onto an available hot spare.
195 This is one of the mechanisms used to start
196 the reconstruction process if a component does have a hardware failure.
197 .It Fl g Ar component Ar dev
198 Get the component label for the specified component.
200 Generate the configuration of the RAIDframe device in a format suitable for
207 Initialize the RAID device.
208 In particular, (re-)write the parity on the selected device.
213 RAID sets before the RAID device is labeled and before
214 file systems are created on the RAID device.
215 .It Fl I Ar serial_number Ar dev
216 Initialize the component labels on each component of the device.
218 is used as one of the keys in determining whether a
219 particular set of components belong to the same RAID set.
220 While not strictly enforced, different serial numbers should be used for
224 be performed when a new RAID set is created.
226 Check the status of the parity on the RAID set.
227 Displays a status message,
228 and returns successfully if the parity is up-to-date.
230 Check the status of the parity on the RAID set, and initialize
231 (re-write) the parity if the parity is not known to be up-to-date.
232 This is normally used after a system crash (and before a
234 to ensure the integrity of the parity.
235 .It Fl r Ar component Ar dev
236 Remove the spare disk specified by
238 from the set of available spare components.
239 .It Fl R Ar component Ar dev
242 if necessary, and immediately begins a reconstruction back to
244 This is useful for reconstructing back onto a component after
245 it has been replaced following a failure.
247 Display the status of the RAIDframe device for each of the components
250 Check the status of parity re-writing, component reconstruction, and
252 The output indicates the amount of progress
253 achieved in each of these areas.
255 Unconfigure the RAIDframe device.
256 This does not remove any component labels or change any configuration
257 settings (e.g. auto-configuration settings) for the RAID set.
260 For operations such as reconstructions, parity
261 re-writing, and copybacks, provide a progress indicator.
269 may be either the full name of the device, e.g.,
271 for the i386 architecture, or
273 for many others, or just simply
276 .Pa /dev/rraid0[cd] ) .
277 It is recommended that the partitions used to represent the
278 RAID device are not used for file systems.
279 .Ss Configuration file
280 The format of the configuration file is complex, and
281 only an abbreviated treatment is given here.
282 In the configuration files, a
284 indicates the beginning of a comment.
286 There are 4 required sections of a configuration file, and 2
288 Each section begins with a
290 followed by the section name,
291 and the configuration parameters associated with that section.
292 The first section is the
294 section, and it specifies
295 the number of rows, columns, and spare disks in the RAID set.
297 .Bd -literal -offset indent
302 indicates an array with 1 row, 3 columns, and 0 spare disks.
303 Note that although multi-dimensional arrays may be specified, they are
305 supported in the driver.
307 The second section, the
309 section, specifies the actual components of the device.
311 .Bd -literal -offset indent
318 specifies the three component disks to be used in the RAID device.
319 If any of the specified drives cannot be found when the RAID device is
320 configured, then they will be marked as
322 and the system will operate in degraded mode.
325 that the order of the components in the configuration file does not
326 change between configurations of a RAID device.
327 Changing the order of the components will result in data loss
328 if the set is configured with the
331 In normal circumstances, the RAID set will not configure if only
333 is specified, and the components are out-of-order.
335 The next section, which is the
337 section, is optional, and, if present, specifies the devices to be used as
339 \(em devices which are on-line,
340 but are not actively used by the RAID driver unless
341 one of the main components fail.
345 .Bd -literal -offset indent
350 for a configuration with a single spare component.
351 If no spare drives are to be used in the configuration, then the
353 section may be omitted.
355 The next section is the
358 This section describes the general layout parameters for the RAID device,
359 and provides such information as
360 sectors per stripe unit,
361 stripe units per parity unit,
362 stripe units per reconstruction unit,
363 and the parity configuration to use.
364 This section might look like:
365 .Bd -literal -offset indent
367 # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level
371 The sectors per stripe unit specifies, in blocks, the interleave
372 factor; i.e., the number of contiguous sectors to be written to each
373 component for a single stripe.
374 Appropriate selection of this value (32 in this example)
375 is the subject of much research in RAID architectures.
376 The stripe units per parity unit and
377 stripe units per reconstruction unit are normally each set to 1.
378 While certain values above 1 are permitted, a discussion of valid
379 values and the consequences of using anything other than 1 are outside
380 the scope of this document.
381 The last value in this section (5 in this example)
382 indicates the parity configuration desired.
383 Valid entries include:
387 No parity, only simple striping.
391 The parity is the mirror.
394 Striping across components, with parity stored on the last component.
397 Striping across components, parity distributed across all components.
400 There are other valid entries here, including those for Even-Odd
401 parity, RAID level 5 with rotated sparing, Chained declustering,
402 and Interleaved declustering, but as of this writing the code for
403 those parity operations has not been tested with
406 The next required section is the
409 This is most often specified as:
410 .Bd -literal -offset indent
415 where the queuing method is specified as fifo (first-in, first-out),
416 and the size of the per-component queue is limited to 100 requests.
417 Other queuing methods may also be specified, but a discussion of them
418 is beyond the scope of this document.
420 The final section, the
422 section, is optional.
423 For more details on this the reader is referred to
424 the RAIDframe documentation discussed in the
430 for a more complete configuration file example.
432 .Bl -tag -width /dev/XXrXraidX -compact
433 .It Pa /dev/{,r}raid*
435 device special files.
438 It is highly recommended that before using the RAID driver for real
439 file systems that the system administrator(s) become quite familiar
442 and that they understand how the component reconstruction process works.
443 The examples in this section will focus on configuring a
444 number of different RAID sets of varying degrees of redundancy.
445 By working through these examples, administrators should be able to
446 develop a good feel for how to configure a RAID set, and how to
447 initiate reconstruction of failed components.
449 In the following examples
451 will be used to denote the RAID device.
452 Depending on the architecture,
456 may be used in place of
458 .Ss Initialization and Configuration
459 The initial step in configuring a RAID set is to identify the components
460 that will be used in the RAID set.
461 All components should be the same size.
462 Each component should have a disklabel type of
464 and a typical disklabel entry for a RAID component might look like:
465 .Bd -literal -offset indent
466 f: 1800000 200495 RAID # (Cyl. 405*- 4041*)
471 will also work as the component type, the type
473 is preferred for RAIDframe use, as it is required for features such as
475 As part of the initial configuration of each RAID set,
476 each component will be given a
477 .Sq component label .
480 contains important information about the component, including a
481 user-specified serial number, the row and column of that component in
482 the RAID set, the redundancy level of the RAID set, a
483 .Sq modification counter ,
484 and whether the parity information (if any) on that
485 component is known to be correct.
486 Component labels are an integral part of the RAID set,
487 since they are used to ensure that components
488 are configured in the correct order, and used to keep track of other
489 vital information about the RAID set.
490 Component labels are also required for the auto-detection
491 and auto-configuration of RAID sets at boot time.
492 For a component label to be considered valid, that
493 particular component label must be in agreement with the other
494 component labels in the set.
495 For example, the serial number,
496 .Sq modification counter ,
497 number of rows and number of columns must all be in agreement.
498 If any of these are different, then the component is
499 not considered to be part of the set.
502 for more information about component labels.
504 Once the components have been identified, and the disks have
507 is then used to configure the
510 To configure the device, a configuration file which looks something like:
511 .Bd -literal -offset indent
513 # numRow numCol numSpare
525 # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5
532 is created in a file.
533 The above configuration file specifies a RAID 5
534 set consisting of the components
543 in case one of the three main drives should fail.
544 A RAID 0 set would be specified in a similar way:
545 .Bd -literal -offset indent
547 # numRow numCol numSpare
557 # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_0
564 In this case, devices
570 are the components that make up this RAID set.
571 Note that there are no hot spares for a RAID 0 set,
572 since there is no way to recover data if any of the components fail.
574 For a RAID 1 (mirror) set, the following configuration might be used:
575 .Bd -literal -offset indent
577 # numRow numCol numSpare
585 # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_1
596 are the two components of the mirror set.
597 While no hot spares have been specified in this
598 configuration, they easily could be, just as they were specified in
599 the RAID 5 case above.
600 Note as well that RAID 1 sets are currently limited to only 2 components.
601 At present, n-way mirroring is not possible.
603 The first time a RAID set is configured, the
606 .Bd -literal -offset indent
607 raidctl -C raid0.conf raid0
612 is the name of the RAID configuration file.
615 forces the configuration to succeed, even if any of the component
616 labels are incorrect.
619 option should not be used lightly in
620 situations other than initial configurations, as if
621 the system is refusing to configure a RAID set, there is probably a
622 very good reason for it.
623 After the initial configuration is done (and
624 appropriate component labels are added with the
626 option) then raid0 can be configured normally with:
627 .Bd -literal -offset indent
628 raidctl -c raid0.conf raid0
631 When the RAID set is configured for the first time, it is
632 necessary to initialize the component labels, and to initialize the
633 parity on the RAID set.
634 Initializing the component labels is done with:
635 .Bd -literal -offset indent
636 raidctl -I 112341 raid0
641 is a user-specified serial number for the RAID set.
642 This initialization step is
645 As well, using different serial numbers between RAID sets is
646 .Em strongly encouraged ,
647 as using the same serial number for all RAID sets will only serve to
648 decrease the usefulness of the component label checking.
650 Initializing the RAID set is done via the
657 RAID sets, since among other things it verifies that the parity (if
658 any) on the RAID set is correct.
659 Since this initialization may be quite time-consuming, the
661 option may be also used in conjunction with
663 .Bd -literal -offset indent
667 This will give more verbose output on the
668 status of the initialization:
669 .Bd -literal -offset indent
670 Initiating re-write of parity
671 Parity Re-write status:
672 10% |**** | ETA: 06:03 /
675 The output provides a
677 in both a numeric and graphical format, as well as an estimated time
678 to completion of the operation.
680 Since it is the parity that provides the
682 part of RAID, it is critical that the parity is correct as much as possible.
683 If the parity is not correct, then there is no
684 guarantee that data will not be lost if a component fails.
686 Once the parity is known to be correct, it is then safe to perform
691 on the device or its file systems, and then to mount the file systems
694 Under certain circumstances (e.g., the additional component has not
695 arrived, or data is being migrated off of a disk destined to become a
696 component) it may be desirable to configure a RAID 1 set with only
698 This can be achieved by using the word
700 to indicate that a particular component is not present.
702 .Bd -literal -offset indent
704 # numRow numCol numSpare
712 # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_1
720 is the real component, and will be the second disk of a RAID 1 set.
721 The first component is simply marked as being absent.
726 as above) proceeds normally, but initialization of the RAID set will
727 have to wait until all physical components are present.
728 After configuration, this set can be used normally, but will be operating
730 Once a second physical component is obtained, it can be hot-added,
731 the existing data mirrored, and normal operation resumed.
733 The size of the resulting RAID set will depend on the number of data
734 components in the set.
735 Space is automatically reserved for the component labels, and
736 the actual amount of space used
737 for data on a component will be rounded down to the largest possible
738 multiple of the sectors per stripe unit (sectPerSU) value.
739 Thus, the amount of space provided by the RAID set will be less
740 than the sum of the size of the components.
741 .Ss Maintenance of the RAID set
742 After the parity has been initialized for the first time, the command:
743 .Bd -literal -offset indent
747 can be used to check the current status of the parity.
748 To check the parity and rebuild it necessary (for example,
749 after an unclean shutdown) the command:
750 .Bd -literal -offset indent
755 Note that re-writing the parity can be done while
756 other operations on the RAID set are taking place (e.g., while doing a
758 on a file system on the RAID set).
759 However: for maximum effectiveness of the RAID set, the parity should be
760 known to be correct before any data on the set is modified.
762 To see how the RAID set is doing, the following command can be used to
763 show the RAID set's status:
764 .Bd -literal -offset indent
768 The output will look something like:
769 .Bd -literal -offset indent
776 Component label for /dev/sd1e:
777 Row: 0 Column: 0 Num Rows: 1 Num Columns: 3
778 Version: 2 Serial Number: 13432 Mod Counter: 65
780 sectPerSU: 32 SUsPerPU: 1 SUsPerRU: 1
781 RAID Level: 5 blocksize: 512 numBlocks: 1799936
783 Last configured as: raid0
784 Component label for /dev/sd2e:
785 Row: 0 Column: 1 Num Rows: 1 Num Columns: 3
786 Version: 2 Serial Number: 13432 Mod Counter: 65
788 sectPerSU: 32 SUsPerPU: 1 SUsPerRU: 1
789 RAID Level: 5 blocksize: 512 numBlocks: 1799936
791 Last configured as: raid0
792 Component label for /dev/sd3e:
793 Row: 0 Column: 2 Num Rows: 1 Num Columns: 3
794 Version: 2 Serial Number: 13432 Mod Counter: 65
796 sectPerSU: 32 SUsPerPU: 1 SUsPerRU: 1
797 RAID Level: 5 blocksize: 512 numBlocks: 1799936
799 Last configured as: raid0
801 Reconstruction is 100% complete.
802 Parity Re-write is 100% complete.
803 Copyback is 100% complete.
806 This indicates that all is well with the RAID set.
807 Of importance here are the component lines which read
812 .Sq Parity status: clean
813 indicates that the parity is up-to-date for this RAID set,
814 whether or not the RAID set is in redundant or degraded mode.
815 .Sq Parity status: DIRTY
816 indicates that it is not known if the parity information is
817 consistent with the data, and that the parity information needs
819 Note that if there are file systems open on the RAID set,
820 the individual components will not be
822 but the set as a whole can still be clean.
824 To check the component label of
826 the following is used:
827 .Bd -literal -offset indent
828 raidctl -g /dev/sd1e raid0
831 The output of this command will look something like:
832 .Bd -literal -offset indent
833 Component label for /dev/sd1e:
834 Row: 0 Column: 0 Num Rows: 1 Num Columns: 3
835 Version: 2 Serial Number: 13432 Mod Counter: 65
837 sectPerSU: 32 SUsPerPU: 1 SUsPerRU: 1
838 RAID Level: 5 blocksize: 512 numBlocks: 1799936
840 Last configured as: raid0
842 .Ss Dealing with Component Failures
844 (perhaps to test reconstruction) it is necessary to pretend a drive
845 has failed, the following will perform that function:
846 .Bd -literal -offset indent
847 raidctl -f /dev/sd2e raid0
850 The system will then be performing all operations in degraded mode,
851 where missing data is re-computed from existing data and the parity.
852 In this case, obtaining the status of raid0 will return (in part):
853 .Bd -literal -offset indent
862 Note that with the use of
864 a reconstruction has not been started.
865 To both fail the disk and start a reconstruction, the
868 .Bd -literal -offset indent
869 raidctl -F /dev/sd2e raid0
874 option may be used first, and then the
876 option used later, on the same disk, if desired.
877 Immediately after the reconstruction is started, the status will report:
878 .Bd -literal -offset indent
881 /dev/sd2e: reconstructing
884 /dev/sd4e: used_spare
887 Reconstruction is 10% complete.
888 Parity Re-write is 100% complete.
889 Copyback is 100% complete.
892 This indicates that a reconstruction is in progress.
893 To find out how the reconstruction is progressing the
896 This will indicate the progress in terms of the
897 percentage of the reconstruction that is completed.
898 When the reconstruction is finished the
901 .Bd -literal -offset indent
907 /dev/sd4e: used_spare
910 Reconstruction is 100% complete.
911 Parity Re-write is 100% complete.
912 Copyback is 100% complete.
915 At this point there are at least two options.
918 is known to be good (i.e., the failure was either caused by
922 or the failed disk was replaced), then a copyback of the data can
923 be initiated with the
926 In this example, this would copy the entire contents of
930 Once the copyback procedure is complete, the
931 status of the device would be (in part):
932 .Bd -literal -offset indent
941 and the system is back to normal operation.
943 The second option after the reconstruction is to simply use
947 in the configuration file.
948 For example, the configuration file (in part) might now look like:
949 .Bd -literal -offset indent
961 is completely interchangeable with
964 Note that extreme care must be taken when
965 changing the order of the drives in a configuration.
966 This is one of the few instances where the devices and/or
967 their orderings can be changed without loss of data!
968 In general, the ordering of components in a configuration file should
972 If a component fails and there are no hot spares
973 available on-line, the status of the RAID set might (in part) look like:
974 .Bd -literal -offset indent
982 In this case there are a number of options.
983 The first option is to add a hot spare using:
984 .Bd -literal -offset indent
985 raidctl -a /dev/sd4e raid0
988 After the hot add, the status would then be:
989 .Bd -literal -offset indent
998 Reconstruction could then take place using
1002 A second option is to rebuild directly onto
1004 Once the disk containing
1006 has been replaced, one can simply use:
1007 .Bd -literal -offset indent
1008 raidctl -R /dev/sd2e raid0
1014 As the rebuilding is in progress, the status will be:
1015 .Bd -literal -offset indent
1018 /dev/sd2e: reconstructing
1023 and when completed, will be:
1024 .Bd -literal -offset indent
1032 In circumstances where a particular component is completely
1033 unavailable after a reboot, a special component name will be used to
1034 indicate the missing component.
1036 .Bd -literal -offset indent
1043 indicates that the second component of this RAID set was not detected
1044 at all by the auto-configuration code.
1047 can be used anywhere a normal component name would be used.
1048 For example, to add a hot spare to the above set, and rebuild to that hot
1049 spare, the following could be done:
1050 .Bd -literal -offset indent
1051 raidctl -a /dev/sd3e raid0
1052 raidctl -F component1 raid0
1055 at which point the data missing from
1057 would be reconstructed onto
1060 When more than one component is marked as
1062 due to a non-component hardware failure (e.g., loss of power to two
1063 components, adapter problems, termination problems, or cabling issues) it
1064 is quite possible to recover the data on the RAID set.
1065 The first thing to be aware of is that the first disk to fail will
1066 almost certainly be out-of-sync with the remainder of the array.
1067 If any IO was performed between the time the first component is considered
1069 and when the second component is considered
1071 then the first component to fail will
1073 contain correct data, and should be ignored.
1074 When the second component is marked as failed, however, the RAID device will
1075 (currently) panic the system.
1076 At this point the data on the RAID set
1077 (not including the first failed component) is still self consistent,
1078 and will be in no worse state of repair than had the power gone out in
1079 the middle of a write to a file system on a non-RAID device.
1080 The problem, however, is that the component labels may now have 3 different
1081 .Sq modification counters
1082 (one value on the first component that failed, one value on the second
1083 component that failed, and a third value on the remaining components).
1084 In such a situation, the RAID set will not autoconfigure,
1085 and can only be forcibly re-configured
1089 To recover the RAID set, one must first remedy whatever physical
1090 problem caused the multiple-component failure.
1091 After that is done, the RAID set can be restored by forcibly
1092 configuring the raid set
1094 the component that failed first.
1099 fail (in that order) in a RAID set of the following configuration:
1100 .Bd -literal -offset indent
1111 # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5
1119 then the following configuration (say "recover_raid0.conf")
1120 .Bd -literal -offset indent
1131 # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5
1139 .Bd -literal -offset indent
1140 raidctl -C recover_raid0.conf raid0
1143 to force the configuration of raid0.
1145 .Bd -literal -offset indent
1146 raidctl -I 12345 raid0
1149 will be required in order to synchronize the component labels.
1150 At this point the file systems on the RAID set can then be checked and
1152 To complete the re-construction of the RAID set,
1154 is simply hot-added back into the array, and reconstructed
1155 as described earlier.
1157 RAID sets can be layered to create more complex and much larger RAID sets.
1158 A RAID 0 set, for example, could be constructed from four RAID 5 sets.
1159 The following configuration file shows such a setup:
1160 .Bd -literal -offset indent
1162 # numRow numCol numSpare
1172 # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_0
1179 A similar configuration file might be used for a RAID 0 set
1180 constructed from components on RAID 1 sets.
1181 In such a configuration, the mirroring provides a high degree
1182 of redundancy, while the striping provides additional speed benefits.
1183 .Ss Auto-configuration and Root on RAID
1184 RAID sets can also be auto-configured at boot.
1185 To make a set auto-configurable,
1186 simply prepare the RAID set as above, and then do a:
1187 .Bd -literal -offset indent
1188 raidctl -A yes raid0
1191 to turn on auto-configuration for that set.
1192 To turn off auto-configuration, use:
1193 .Bd -literal -offset indent
1197 RAID sets which are auto-configurable will be configured before the
1198 root file system is mounted.
1199 These RAID sets are thus available for
1200 use as a root file system, or for any other file system.
1201 A primary advantage of using the auto-configuration is that RAID components
1202 become more independent of the disks they reside on.
1203 For example, SCSI ID's can change, but auto-configured sets will always be
1204 configured correctly, even if the SCSI ID's of the component disks
1205 have become scrambled.
1207 Having a system's root file system
1209 on a RAID set is also allowed, with the
1211 partition of such a RAID set being used for
1213 To use raid0a as the root file system, simply use:
1214 .Bd -literal -offset indent
1215 raidctl -A root raid0
1218 To return raid0a to be just an auto-configuring set simply use the
1222 Note that kernels can only be directly read from RAID 1 components on
1223 architectures that support that
1224 .Pq currently alpha, i386, pmax, sparc, sparc64, and vax .
1225 On those architectures, the
1227 file system is recognized by the bootblocks, and will properly load the
1228 kernel directly from a RAID 1 component.
1229 For other architectures, or to support the root file system
1230 on other RAID sets, some other mechanism must be used to get a kernel booting.
1231 For example, a small partition containing only the secondary boot-blocks
1232 and an alternate kernel (or two) could be used.
1233 Once a kernel is booting however, and an auto-configuring RAID set is
1234 found that is eligible to be root, then that RAID set will be
1235 auto-configured and used as the root device.
1236 If two or more RAID sets claim to be root devices, then the
1237 user will be prompted to select the root device.
1238 At this time, RAID 0, 1, 4, and 5 sets are all supported as root devices.
1240 A typical RAID 1 setup with root on RAID might be as follows:
1243 wd0a - a small partition, which contains a complete, bootable, basic
1247 wd1a - also contains a complete, bootable, basic
1251 wd0e and wd1e - a RAID 1 set, raid0, used for the root file system.
1253 wd0f and wd1f - a RAID 1 set, raid1, which will be used only for
1256 wd0g and wd1g - a RAID 1 set, raid2, used for
1259 or other data, if desired.
1261 wd0h and wd1h - a RAID 1 set, raid3, if desired.
1264 RAID sets raid0, raid1, and raid2 are all marked as auto-configurable.
1265 raid0 is marked as being a root file system.
1266 When new kernels are installed, the kernel is not only copied to
1268 but also to wd0a and wd1a.
1269 The kernel on wd0a is required, since that
1270 is the kernel the system boots from.
1271 The kernel on wd1a is also
1272 required, since that will be the kernel used should wd0 fail.
1273 The important point here is to have redundant copies of the kernel
1274 available, in the event that one of the drives fail.
1276 There is no requirement that the root file system be on the same disk
1278 For example, obtaining the kernel from wd0a, and using
1279 sd0e and sd1e for raid0, and the root file system, is fine.
1282 critical, however, that there be multiple kernels available, in the
1283 event of media failure.
1285 Multi-layered RAID devices (such as a RAID 0 set made
1286 up of RAID 1 sets) are
1288 supported as root devices or auto-configurable devices at this point.
1289 (Multi-layered RAID devices
1291 supported in general, however, as mentioned earlier.)
1292 Note that in order to enable component auto-detection and
1293 auto-configuration of RAID devices, the line:
1294 .Bd -literal -offset indent
1295 options RAID_AUTOCONFIG
1298 must be in the kernel configuration file.
1302 .Ss Swapping on RAID
1303 A RAID device can be used as a swap device.
1304 In order to ensure that a RAID device used as a swap device
1305 is correctly unconfigured when the system is shutdown or rebooted,
1306 it is recommended that the line
1307 .Bd -literal -offset indent
1314 The final operation performed by
1319 This is accomplished via a simple:
1320 .Bd -literal -offset indent
1324 at which point the device is ready to be reconfigured.
1325 .Ss Performance Tuning
1326 Selection of the various parameter values which result in the best
1327 performance can be quite tricky, and often requires a bit of
1328 trial-and-error to get those values most appropriate for a given system.
1329 A whole range of factors come into play, including:
1332 Types of components (e.g., SCSI vs. IDE) and their bandwidth
1334 Types of controller cards and their bandwidth
1336 Distribution of components among controllers
1340 file system access patterns
1345 As with most performance tuning, benchmarking under real-life loads
1346 may be the only way to measure expected performance.
1347 Understanding some of the underlying technology is also useful in tuning.
1348 The goal of this section is to provide pointers to those parameters which may
1349 make significant differences in performance.
1351 For a RAID 1 set, a SectPerSU value of 64 or 128 is typically sufficient.
1352 Since data in a RAID 1 set is arranged in a linear
1353 fashion on each component, selecting an appropriate stripe size is
1354 somewhat less critical than it is for a RAID 5 set.
1355 However: a stripe size that is too small will cause large IO's to be
1356 broken up into a number of smaller ones, hurting performance.
1357 At the same time, a large stripe size may cause problems with
1358 concurrent accesses to stripes, which may also affect performance.
1359 Thus values in the range of 32 to 128 are often the most effective.
1361 Tuning RAID 5 sets is trickier.
1362 In the best case, IO is presented to the RAID set one stripe at a time.
1363 Since the entire stripe is available at the beginning of the IO,
1364 the parity of that stripe can be calculated before the stripe is written,
1365 and then the stripe data and parity can be written in parallel.
1366 When the amount of data being written is less than a full stripe worth, the
1371 means only a portion of the stripe on the components is going to
1372 change, the data (and parity) on the components must be updated
1373 slightly differently.
1378 must be read from the components.
1379 Then the new parity is constructed,
1380 using the new data to be written, and the old data and old parity.
1381 Finally, the new data and new parity are written.
1382 All this extra data shuffling results in a serious loss of performance,
1383 and is typically 2 to 4 times slower than a full stripe write (or read).
1384 To combat this problem in the real world, it may be useful
1385 to ensure that stripe sizes are small enough that a
1387 from the system will use exactly one large stripe write.
1388 As is seen later, there are some file system dependencies
1389 which may come into play here as well.
1393 is often (currently) only 32K or 64K, on a 5-drive RAID 5 set it may
1394 be desirable to select a SectPerSU value of 16 blocks (8K) or 32
1396 Since there are 4 data sectors per stripe, the maximum
1397 data per stripe is 64 blocks (32K) or 128 blocks (64K).
1398 Again, empirical measurement will provide the best indicators of which
1399 values will yeild better performance.
1401 The parameters used for the file system are also critical to good performance.
1404 for example, increasing the block size to 32K or 64K may improve
1405 performance dramatically.
1406 As well, changing the cylinders-per-group
1407 parameter from 16 to 32 or higher is often not only necessary for
1408 larger file systems, but may also have positive performance implications.
1410 Despite the length of this man-page, configuring a RAID set is a
1411 relatively straight-forward process.
1412 All that needs to be done is the following steps:
1417 to create the components (of type RAID).
1419 Construct a RAID configuration file: e.g.,
1422 Configure the RAID set with:
1423 .Bd -literal -offset indent
1424 raidctl -C raid0.conf raid0
1428 Initialize the component labels with:
1429 .Bd -literal -offset indent
1430 raidctl -I 123456 raid0
1434 Initialize other important parts of the set with:
1435 .Bd -literal -offset indent
1440 Get the default label for the RAID set:
1441 .Bd -literal -offset indent
1442 disklabel raid0 \*[Gt] /tmp/label
1447 .Bd -literal -offset indent
1452 Put the new label on the RAID set:
1453 .Bd -literal -offset indent
1454 disklabel -R -r raid0 /tmp/label
1458 Create the file system:
1459 .Bd -literal -offset indent
1464 Mount the file system:
1465 .Bd -literal -offset indent
1466 mount /dev/raid0e /mnt
1471 .Bd -literal -offset indent
1472 raidctl -c raid0.conf raid0
1475 To re-configure the RAID set the next time it is needed, or put
1479 where it will automatically be started by the
1488 RAIDframe is a framework for rapid prototyping of RAID structures
1489 developed by the folks at the Parallel Data Laboratory at Carnegie
1490 Mellon University (CMU).
1491 A more complete description of the internals and functionality of
1492 RAIDframe is found in the paper "RAIDframe: A Rapid Prototyping Tool
1493 for RAID Systems", by William V. Courtright II, Garth Gibson, Mark
1494 Holland, LeAnn Neal Reilly, and Jim Zelenka, and published by the
1495 Parallel Data Laboratory of Carnegie Mellon University.
1499 command first appeared as a program in CMU's RAIDframe v1.1 distribution.
1502 is a complete re-write, and first appeared in
1506 The RAIDframe Copyright is as follows:
1508 Copyright (c) 1994-1996 Carnegie-Mellon University.
1509 All rights reserved.
1511 Permission to use, copy, modify and distribute this software and
1512 its documentation is hereby granted, provided that both the copyright
1513 notice and this permission notice appear in all copies of the
1514 software, derivative works or modified versions, and any portions
1515 thereof, and that both notices appear in supporting documentation.
1517 CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS"
1518 CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND
1519 FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
1521 Carnegie Mellon requests users of this software to return to
1523 Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU
1524 School of Computer Science
1525 Carnegie Mellon University
1526 Pittsburgh PA 15213-3890
1528 any improvements or extensions that they make and grant Carnegie the
1529 rights to redistribute these changes.
1532 Certain RAID levels (1, 4, 5, 6, and others) can protect against some
1533 data loss due to component failure.
1534 However the loss of two components of a RAID 4 or 5 system,
1535 or the loss of a single component of a RAID 0 system will
1536 result in the entire file system being lost.
1539 a substitute for good backup practices.
1541 Recomputation of parity
1543 be performed whenever there is a chance that it may have been compromised.
1544 This includes after system crashes, or before a RAID
1545 device has been used for the first time.
1546 Failure to keep parity correct will be catastrophic should a
1547 component ever fail \(em it is better to use RAID 0 and get the
1548 additional space and speed, than it is to use parity, but
1549 not keep the parity correct.
1550 At least with RAID 0 there is no perception of increased data security.
1552 Hot-spare removal is currently not available.