1 INCREMENTAL BACKUP HOWTO
3 This document describes one of several ways to set up a LAN backup and
4 an off-site WAN backup system using cpdup's hardlinking capabilities.
6 The features described in this document are also encapsulated in scripts
7 which can be found in the scripts/ directory. These scripts can be used
8 to automate all backup steps except for the initial preparation of the
9 backup and off-site machine's directory topology. Operation of these
10 scripts is described in the last section of this document.
13 PART 1 - PREPARE THE LAN BACKUP BOX
15 The easiest way to create a LAN backup box is to NFS mount all your
16 backup clients onto the backup box. It is also possible to use cpdup's
17 remote host feature to access your client boxes but that requires root
18 access to the client boxes and is not described here. (But see the
19 sections "OFF-SITE BACKUPS" and "SSH SECURITY TIPS" below.)
21 Create a directory on the backup machine called /nfs, a subdirectory
22 foreach remote client, and subdirectories for each partition on each
23 client. Remember that cpdup does not cross mount points so you will
24 need a mount for each partition you wish to backup. For example:
33 Before you actually do the NFS mount, create a dummy file for each
34 mount point that can be used by scripts to detect when an NFS mount
35 has not been done. Scripts can thus avoid a common failure scenario
36 and not accidentally cpdup an empty mount point to the backup partition
37 (destroying that day's backup in the process).
39 touch /nfs/box1/home/NOT_MOUNTED
40 touch /nfs/box1/var/NOT_MOUNTED
42 Once the directory structure has been set up, do your NFS mounts and
43 also add them to your fstab. Since you will probably wind up with a
44 lot of mounts it is a good idea to use 'ro,bg' (readonly, background
45 mount) in the fstab entries.
47 mount box1:/home /nfs/box1/home
48 mount box1:/var /nfs/box1/var
50 You should create a huge /backup partition on your backup machine which
51 is capable of holding all your mirrors. Create a subdirectory called
52 /backup/mirrors in your huge backup partition.
54 mount <huge_disk> /backup
58 PART 2 - DOING A LEVEL 0 BACKUP
60 (If you use the supplied scripts, a level 0 backup can be accomplished
61 simply by running the 'do_mirror' script with an argument of 0).
63 Create a level 0 backup using a standard cpdup with no special arguments
64 other then -i0 -s0 (tell it not to ask questions and turn off the
65 file-overwrite-with-directory safety feature). Name the mirror with
66 the date in a string-sortable format.
68 set date = `date "+%Y%m%d"`
69 mkdir /backup/mirrors/box1.${date}
70 cpdup -i0 -s0 /nfs/box1/home /backup/mirrors/box1.${date}/home
71 cpdup -i0 -s0 /nfs/box1/var /backup/mirrors/box1.${date}/var
73 Create a softlink to the most recently completed backup, which is your
74 level 0 backup. Note that using 'ln -sf' will create a link in the
75 subdirectory pointed to by the current link, not replace the current
76 link. 'ln -shf' can be used to replace the link but is not portable.
77 'mv -f' has the same problem.
80 rm -f /backup/mirrors/box1
81 ln -s /backup/mirrors/box1.${date} /backup/mirrors/box1
83 PART 3 - DO AN INCREMENTAL BACKUP
85 An incremental backup is exactly the same as a level 0 backup EXCEPT
86 you use the -H option to specify the location of the most recent
87 completed backup. We simply maintain the handy softlink pointing at
88 the most recent completed backup and the cpdup required to do this
91 Each day's incremental backup will reproduce the ENTIRE directory topology
92 for the client, but cpdup will hardlink files from the most recent backup
93 instead of copying them and this is what saves you all the disk space.
95 set date = `date "+%Y%m%d"`
96 mkdir /backup/mirrors/box1.${date}
97 if ( "`readlink /backup/mirrors/box1`" == "box1.${date}" ) then
98 echo "silly boy, an incremental already exists for today"
101 cpdup -H /backup/mirrors/box1 \
102 -i0 -s0 /nfs/box1/home /backup/mirrors/box1.${date}/home
104 Be sure to update your 'most recent backup' softlink, but only do it
105 if the cpdup's for all the partitions for that client have succeeded.
106 That way the next incremental backup will be based on the previous one.
108 rm -f /backup/mirrors/box1
109 ln -s /backup/mirrors/box1.${date} /backup/mirrors/box1
111 Since these backups are mirrors, locating a backup is as simple
112 as CDing into the appropriate directory. If your filesystem has a
113 hardlink limit and cpdup hits it, cpdup will 'break' the hardlink
114 and copy the file instead. Generally speaking only a few special cases
115 will hit the hardlink limit for a filesystem. For example, the
116 CVS/Root file in a checked out cvs repository is often hardlinked, and
117 the sheer number of hardlinked 'Root' files multiplied by the number
118 of backups can often hit the filesystem hardlink limit.
120 PART 4 - DO AN INCREMENTAL VERIFIED BACKUP
122 Since your incremental backups use hardlinks heavily the actual file
123 might exist on the physical /backup disk in only one place even though
124 it may be present in dozens of daily mirrors. To ensure that the
125 file being hardlinked does not get corrupted cpdup's -f option can be
126 used in conjunction with -H to force cpdup to validate the contents
127 of the file, even if all the stat info looks identical.
129 cpdup -f -H /backup/mirrors/box1 ...
131 You can create completely redundant (non-hardlinked-dependent) backups
132 by doing the equivalent of your level 0, i.e. not using -H. However I
133 do NOT recommend that you do this, or that you do it very often (maybe
134 once every 6 months at the most), because each mirror created this way
135 will have a distinct copy of all the file data and you will quickly
136 run out of space in your /backup partition.
138 MAINTENANCE OF THE "/backup" DIRECTORY
140 Now, clearly you are going to run out of space in /backup if you keep
141 doing this, but you may be surprised at just how many daily incrementals
142 you can create before you fill up your /backup partition.
144 If /backup becomes full, simply start rm -rf'ing older mirror directories
145 until enough space is freed up. You do not have to remove the oldest
146 directory first. In fact, you might want to keep it around and remove
147 a day's backup here, a day's backup there, etc, until you free up enough
152 Making an off-site backup involves similar methodology, but you use
153 cpdup's remote host capability to generate the backup. To avoid
154 complications it is usually best to take a mirror already generated on
155 your LAN backup box and copy that to the remote box.
157 The remote backup box does not use NFS, so setup is trivial. Just
158 create your super-large /backup partition and mkdir /backup/mirrors.
159 Your LAN backup box will need root access via ssh to your remote backup
160 box. See the section "SSH SECURITY TIPS" below.
162 You can use the handy softlink to get the latest 'box1.date' mirror
163 directory and since the mirror is all in one partition you can just
164 cpdup the entire machine in one command. Use the same dated directory
165 name on the remote box, so:
167 # latest will wind up something like 'box1.20060915'
168 set latest = `readlink /backup/mirrors/box1`
169 cpdup -i0 -s0 /backup/mirrors/$latest remote.box:/backup/mirrors/$latest
171 As with your LAN backup, create a softlink on the backup box denoting the
172 latest mirror for any given site.
174 if ( $status == 0 ) then
176 "rm -f /backup/mirrors/box1; ln -s /backup/mirrors/$latest /backup/mirrors/box1"
179 Incremental backups can be accomplished using the same cpdup command,
180 but adding the -H option to the latest backup on the remote box. Note
181 that the -H path is relative to the remote box, not the LAN backup box
182 you are running the command from.
184 set latest = `readlink /backup/mirrors/box1`
185 set remotelatest = `ssh remote.box -n "readlink /backup/mirrors/box1"`
186 if ( "$latest" == "$remotelatest" ) then
187 echo "silly boy, you already made a remote incremental backup today"
190 cpdup -H /backup/mirrors/$remotelatest \
191 -i0 -s0 /backup/mirrors/$latest remote.box:/backup/mirrors/$latest
192 if ( $status == 0 ) then
194 "rm -f /backup/mirrors/box1; ln -s /backup/mirrors/$latest /backup/mirrors/box1"
197 Cleaning out the remote directory works the same as cleaning out the LAN
201 RESTORING FROM BACKUPS
203 Each backup is a full filesystem mirror, and depending on how much space
204 you have you should be able to restore it simply by cd'ing into the
205 appropriate backup directory and using 'cpdup blah box1:blah' (assuming
206 root access), or you can export the backup directory via NFS to your
207 client boxes and use cpdup locally on the client to extract the backup.
208 Using NFS is probably the most efficient solution.
211 PUTTING IT ALL TOGETHER - SOME SCRIPTS
213 Please refer to the scripts in the script/ subdirectory. These scripts
214 are EXAMPLES ONLY. If you want to use them, put them in your ~root/adm
215 directory on your backup box and set up a root crontab.
217 First follow the preparation rules in PART 1 above. The scripts do not
218 do this automatically. Edit the 'params' file that the scripts use
219 to set default paths and such.
221 ** FOLLOW DIRECTIONS IN PART 1 ABOVE TO SET UP THE LAN BACKUP BOX **
223 Copy the scripts to ~/adm. Do NOT install a crontab yet (but an example
224 can be found in scripts/crontab).
226 Do a manual lavel 0 LAN BACKUP using the do_mirror script.
231 Once done you can do incremental backups using './do_mirror 1' to do a
232 verified incremental, or './do_mirror 2' to do a stat-optimized
233 incremental. You can enable the cron jobs that run do_mirror and
238 Setting up an off-site backup box is trivial. The off-site backup box
239 needs to allow root ssh logins from the LAN backup box (at least for
240 now, sorry!). Set up the off-site backup directory, typically
241 /backup/mirrors. Then do a level 0 backup from your LAN backup box
242 to the off-site box using the do_remote script.
247 Once done you can do incremental backups using './do_remote 1' to do a
248 verified incremental, or './do_mirror 2' to do a stat-optimized
249 incremental. You can enable the cron jobs that run do_remote now.
251 NOTE! It is NOT recommended that you use verified-incremental backups
252 over a WAN, as all related data must be copied over the wire every single
253 day. Instead, I recommend sticking with stat-optimized backups
256 You will also need to set up a daily cleaning script on the off-site
259 SCRIPT TODOS - the ./do_cleanup script is not very smart. We really
260 should do a tower-of-hanoi removal
265 To allow root access via ssh, add the following line to your sshd
266 configuration on the client boxes (typically /etc/ssh/sshd_config):
268 PermitRootLogin forced-commands-only
270 If your OpenSSH version is too old to recognize that setting, you
271 should update to a more recent version immediately.
272 Restart sshd for the settings to take effect.
274 On the backup machine, create a special backup key for root:
276 mkdir /root/.ssh # if it doesn't already exist
278 ssh-keygen -t dsa -N "" -f backup-key
280 You now have a key pair, consisting of a secret key called "backup-key"
281 and a public key called "backup-key.pub". The secret key must *NEVER*
282 leave the backup machine nor be disclosed in any way! Note that we
283 haven't procted the secret key with a passphrase (-N "") because it
284 will be used by cron jobs where no passphrase can be entered.
286 On the client boxes, create a file /root/.ssh/authorized_keys.
287 It should contain just this line:
289 command="/usr/local/bin/cpdup -S",from="<BAKHOST>",no-pty,
290 no-port-forwarding,no-X11-forwarding,no-agent-forwarding <PUBKEY>
292 This must be on one long line; it has been broken up here for
293 readability only. Note that the options must be separated by commas
294 *ONLY* (no spaces). Replace <BAKHOST> with the IP address or DNS name
295 of the backup machine. Replace <PUBKEY> with the contents of the
296 file /root/.ssh/backup-key.pub from the backup machine (the public key,
297 not the secret key!). It typically starts with "ssh-dss" followed by
298 a long character sequence that looks like line noise, followed by a
299 comment that typically indicates who created the key.
301 The format of the authorized_keys file is documented in the sshd(8)
302 manual page. Please refer to it for more details.
304 If you have done all of the above correctly, then the root user on the
305 backup machine will be able to log into the client boxes as root and
306 execute "/usr/local/bin/cpdup -S", but nothing else.
308 To further improve security, you can place the slave cpdup on the client
309 machine into read-only mode by adding the -R option. In this case, the
310 line from the authorized_keys file should begin as follows:
312 command="/usr/local/bin/cpdup -RS",from="<BAKHOST>",etc...
314 If you do that, your backup server can only pull backups from the client
315 machine, but it won't be able to change anything on it. That is, you
316 cannot use the client machine as a remote target. So, if an attacker
317 manages to be able to execute commands on your backup machine, he won't
318 be able to do any harm to your clients. This also protects against
319 human errors, e.g. accidentally swapping source and destination.
321 By the way, it doesn't really matter much whether you specify the -R
322 option when running cpdup on the backup machine. If you do it, then
323 the -R option will be passed to the slave, but the command="..." entry
324 from the authorized_keys file overides it anyway, so the slave always
325 runs with the -R option.
327 When using cpdup on the backup machine, make sure that the right key is
328 used by passing the -i option to the ssh command:
330 cpdup -F -i/root/.ssh/backup-key ...
332 If one or both of the machines involved has a slow processor, it might
333 be worthwhile to use a faster encryption algorithm, for example:
335 cpdup -F -cblowfish-cbc ...
337 If your OpenSSH version has been patched to support unencrypted transfers
338 *AND* you trust the physical network between the machines involved, you
339 might want to disable encryption altogether: