2 # -*- coding: ascii -*-
3 ###########################################################################
4 # clive, command line video extraction utility.
5 # Copyright 2007, 2008, 2009 Toni Gundogdu.
7 # This file is part of clive.
9 # clive is free software: you can redistribute it and/or modify it under
10 # the terms of the GNU General Public License as published by the Free
11 # Software Foundation, either version 3 of the License, or (at your option)
14 # clive is distributed in the hope that it will be useful, but WITHOUT ANY
15 # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
16 # FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
19 # You should have received a copy of the GNU General Public License along
20 # with this program. If not, see <http://www.gnu.org/licenses/>.
21 ###########################################################################
34 clive - command line video extraction utility
38 clive [options]... [URL]...
42 clive is a command line video extraction utility for Youtube and other
43 video-sharing websites. It was written to work around the Adobe Flash
46 clive is not an universal video extraction utility. It only works with
47 the video hosts that it supports.
51 -h, --help print help and exit
52 -v, --version print version and exit
53 --hosts print supported hosts and exit
54 --home-dir=HOMEDIR use HOMEDIR instead of $HOME
56 --emit-csv emit video details in CSV to stdout
57 --debug print cURL debug messages
58 -q, --quiet turn off all output
59 -l, --recall recall last input
60 --recall-file=FILE use FILE instead of default ~/.clivelast
61 --stderr redirect all output to stderr
63 --agent=STRING identify as STRING to http server
64 --connect-timeout=SECS max time allowed connection to take
65 --connect-timeout-socks=S same as above, tries to workaround SOCKS
66 --proxy=ADDR use ADDR for http proxy
67 --no-proxy do not use http proxy
69 --cache-file=FILE use FILE instead of default ~/.clivecache
70 -r, --cache-read enable reading from cache
71 -d, --cache-dump dump cache records to stdout
72 --cache-dump-format=STRING print format for dumping cache records
73 --cache-grep=PATTERN grep cache records for PATTERN
74 -i, --cache-ignore-case ignore case while matching records
75 -D, --cache-remove-record remove matched records from cache
76 --cache-clear truncate cache records
77 --no-cache disable cache all (read and write) use
79 -f, --format=FORMAT extract FORMAT of video
80 -O, --output-file=FNAME write video to file
81 -n, --no-extract do not extract any videos
82 -c, --continue continue partially downloaded file
83 --save-dir=DIR save video files to DIR
84 --cclass=CLASS use character CLASS to filter titles
85 -C, --no-cclass do not apply character class
86 --filename-format=STRING use STRING to format output filename
87 --exec=CMD run CMD subsequently after file transfer
88 --stream=PERCENT run stream command (below) at PERCENT
89 --stream-exec=CMD stream command to run
90 --limit-rate=AMOUNT limit transfer rate to AMOUNT (KB/s)
91 --stop-after=SIZE|PERCENT stop file transfer after SIZE or PERCENT
92 -R, --raw process video page html on as-is basis
94 See clive manual page for EXAMPLES.
98 You may freely mix different option styles and specify options after
99 the command line arguments, e.g.:
100 % clive -c URL --format=best
102 You may also put several options together that do not require arguments:
103 % clive -cnrf best URL
105 Note that the "dashed" options have aliases available which are not
106 documented in the L</OPTIONS> section. For example:
107 % clive --no-extract --no_extract --noextract
108 % clive --cache-read --cache_read --cacheread
110 =head1 OPTION DESCRIPTIONS
118 =item B<-v, --version>
120 Print version and exit.
124 Print supported hosts, with precompiled regexps used to identify links,
127 =item B<--home-dir>=I<homedir>
129 Use I<homedir> instead of the default $HOME (and $CLIVE_HOME) environment
130 variable for all non-config files (e.g. cache).
140 Print (emit) video details in CSV format to standard output.
141 Implies --no-extract.
145 Print cURL debug (or verbose) messages to standard error.
149 Turn off all output to standard output and error.
151 =item B<-l, --recall>
153 Recall (or reuse) the last input.
155 =item B<--recall-file>=I<file>
157 Use I<file> instead of I<homedir>/.clivelast.
161 Direct all output to standard error.
169 =item B<--agent>=I<string>
171 Identify clive as I<string> to HTTP servers. Defaults to "Mozilla/5.0".
173 =item B<--connect-timeout>=I<seconds>
175 Maximum time in I<seconds> allowed for connection to take. Defaults to 30.
177 =item B<--connect-timeout-socks>=I<seconds>
179 Same as above but tries to workaround the SOCKS proxy bug in cURL. Defaults
182 =item B<--proxy>=I<address>
184 Use I<address> for HTTP proxy. Example: "http://foo:1234".
188 Disable HTTP proxy use even if http_proxy environment variable is set.
196 =item B<--cache-file>=I<file>
198 Use I<file> instead of I<homedir>/.clivecache.
200 =item B<-r, --cache-read>
202 Read video details from cache record if it exists. Allows clive to
203 skip fetching the video page again. See L</CACHE> section for more
206 =item B<-d, --cache-dump>
208 Dump cache records to standard output.
210 =item B<--cache-dump-format>=I<format-string>
212 Used to format the output of the above. Defaults to "%n: %t [%f, %mMB]".
215 % clive --cache-dump --cache-dump-format="%d: %t"
217 Supported format specifiers:
218 %t .. video page title
221 %l .. video file length (bytes)
222 %m .. video file length (MB)
223 %d .. date (last update)
224 %T .. time (last update)
225 %s .. time stamp (same as "%d %T")
226 %f .. video file format
229 =item B<--cache-grep>=I<pattern>
231 Grep stored cache records for I<pattern>. See also L</EXAMPLES - ADVANCED USE>.
233 =item B<-i, --cache-ignore-case>
235 Ignore case-differences while matching records.
237 =item B<-D, --cache-remove-record>
239 Remove matched records from cache.
241 =item B<--cache-clear>
243 Truncate cache records.
247 Disable all (read and write) cache use.
255 =item B<-f, --format>=I<format>
257 Extract I<format> of the video. If I<format> is set to I<best>, clive
258 will attempt to extract the best video quality.
260 Note that the I<format> is strictly host specific. See the L</FORMATS>
261 section for more on this.
263 =item B<-O, --output-file>=I<filename>
265 Write video to I<filename>. Not to be used when extracting
268 =item B<-n, --no-extract>
270 Do not extract the video.
272 =item B<-c, --continue>
274 Continue partially downloaded video file.
276 =item B<--sav-dir>=I<dir>
278 Save extracted videos to I<dir>.
280 =item B<--cclass>=I<class>
282 Use character-I<class> to filter video page titles. Defaults to "\w".
283 Refer to the Perl regular expressions (character classes) for more on
286 =item B<-C, --no-cclass>
288 Negates the use of the character-class.
290 =item B<filename-format>=<format-string>
292 Use <format-string> to format output video filenames. Defaults to "%t.%s".
294 Supported format specifiers:
295 %t .. video page title (after applying character-class filter)
296 %s .. video file suffix (e.g. "flv")
300 =item B<--exec>=I<command>;
302 Execute I<command> subsequently after video file transfer. Optional
303 arguments may be passed to the command. The expression must be
304 terminated by a semicolon (";"). If the specifier "%i" appears
305 anywhere in the I<command>, it is replaced by the pathname of the
306 extracted video file.
308 =item B<--exec>=I<command>+
310 Same as above but "%i" is replaced with as many pathnames as
311 possible for the invocation of I<command>.
313 =item B<--stream>=I<percent>
315 Execute --stream-exec=I<command> when file transfer reaches
318 =item B<--stream-exec>=I<command>
320 Execute (fork a child process) I<command> while transferring
321 video file. This "simulates" streaming the media but does so
322 without checking for buffer underruns so make sure you set
323 --stream=I<percent> high enough and that you have a fast
326 clive will not attempt to re-execute the command if it
327 terminates before the file transfer finishes.
329 clive will wait that the child process terminates before
330 it moves on to extract another file or exits if there
333 Note that some video file formats (namely Google mp4) are
334 known to B<not> to work with this feature.
336 =item B<--limit-rate>=I<amount>
338 Limit transfer rate to I<amount> KB/s.
340 =item B<--stop-after>=I<size|percent>
342 Stop file transfer after I<size> or I<percent>. The value must
343 be terminated by either '%' or 'M'.
347 Process video page HTML as it is. Technically, this means that
348 clive will not try to decode the HTML using Encode::decode_utf8.
349 This may be useful with some of the supported host such as
350 Cctv which uses GBK as its HTML encoding.
354 =head1 EXAMPLES - BASIC USE
358 =item clive "http://youtube.com/watch?v=3HD220e0bx4"
360 Extracts video (flv) from the above video page link. You
361 can then play the flv video file in a media player.
363 =item cat E<gt> url.lst
365 http://en.sevenload.com/videos/IUL3gda-Funny-Football-Clips
366 http://youtube.com/watch?v=3HD220e0bx4
367 http://break.com/index/beach-tackle-whip-lash.html
368 http://www.liveleak.com/view?i=704_1228511265
370 =item cat url.lst | clive
372 You can feed clive multiple video page links like this
375 =item clive URL URL2 URL3 URL4
377 When you are using the pipes, be sure to separate each link
380 =item xclip -o | clive
382 Many X clipboard utilities exist. This example uses C<xclip(1)>
383 and a pipe to paste the contents of the clipboard to clive.
387 Recall last video page link input.
391 =head1 EXAMPLES - ADVANCED USE
395 =item clive -f best "http://youtube.com/watch?v=3HD220e0bx4"
397 Extract the best format of the video.
399 =item clive -r -f best "http://youtube.com/watch?v=3HD220e0bx4"
401 Causes clive to skip re-fetching the video page and parsing it.
402 Instead it reuses the record saved in the cache from the previous
405 =item clive --cache-dump
407 Dump cache records to stdout. You can use --cache-dump-format
408 to format the output.
412 Grep for "3hd2" pattern in cache records. If pattern
413 matches, clive continues to extract the matched videos.
414 Note the use of "-i" (--cache-ignore-case).
416 =item clive -ig 3hd2 -D
418 Same as above but removes the record from cache instead of
419 extracting the video.
421 =item clive --exec="ffmpeg -i %i %i.mp3;" URL
423 Extract video and use C<ffmpeg(1)> to copy audio from it
426 =item clive --stream-exec="mplayer -really-quiet %i"
429 Start playing the video being extracted when the transfer
430 reaches 25% complete.
436 clive extracts flv (typically 320x240) by default from all
437 supported websites. Some of them support also other formats.
445 Format: (flv|fmt17|fmt18|fmt22|fmt35)
447 flv (fmt34) and fmt18 (mp4) are usually available. Others may be
448 available. At the time of writing this, the following formats
449 are recognized by both clive and Youtube:
451 fmt22 .. mp4 (1280x720) (HD)
452 fmt35 .. flv (640x380) (HQ)
453 fmt18 .. mp4 (480x360)
454 flv .. fmt34 (320x180)
455 fmt17 .. 3gp (176x144)
457 Some of the videos available at last.fm are actually Youtube
458 videos. clive can handle such video links.
460 =item B<video.google.com>
464 mp4 format is available for a limited number of videos.
466 =item B<dailymotion.com>
468 Format: (flv|spak-mini|vp6-hq|vp6-hd|vp6|h264)
470 The HD and HQ videos may not always be available.
472 ON2-1280x720 (vp6-hd)
476 FLV-320x240 (flv/spark)
477 FLV-80x60 (spak-mini)
483 HD should be available for the vimeo.com/hd channel videos at least.
484 Note that "flv" only means "default" here, as some of the hosted
485 videos are encoded (as default) in other video formats such as
486 "mp4" rather than "flv".
489 http://vimeo.com/help/hd
493 =item B<liveleak.com>
497 =item B<sevenload.com>
513 User configuration file. You can append the command line options to
514 the config file, e.g.:
517 --proxy=http://foo:1234
521 File containing last user input.
525 BerkeleyDB based cache containing the records of fetched and
530 Note that the default path (~) for all non-config files can be altered
531 with CLIVE_HOME environment variable and --home-dir option.
538 The purpose of the cache is to allow clive to skip fetching
539 and parsing the video page again. It does not contain any
540 actual video data so one should not expect to recover a deleted
541 video file from the cache. Only some of the parsed details
542 are stored as records to the cache.
544 By now, it is should be a well known fact that the cache fails
545 with some of the supported hosts. For example Youtube video links
546 expire after some time, this causes the re-extraction to fail if
547 the cached video link is used later again.
549 This was the main reason why in 2.2.0 reading from cache was
550 disabled by default. Many users reported the reuse of expired
551 video links as a bug previously even though it was well documented
552 in the manual page explaining that most of the HTTP 403/404 errors
553 were actually caused by expired video links.
555 It is, of course, still possible to read from cache. You can
556 enable this by invoking the --cache-read option. This causes
557 clive to look up a saved cache record and reuse the stored
558 video details if they are found instead of fetching the video
561 The use of the cache can be disabled with the --no-cache option.
562 This disables both read and write. Note that if the BerkeleyDB Perl
563 module is not installed, clive will not use the cache.
565 See also the --cache-grep option.
569 You are welcome to submit bug reports via the Issue tracker at:
570 <http://code.google.com/p/clive/issues/>
574 clive exits 0 on success, and E<gt>0 if an error occurs.
577 CLIVE_NOTHINGTODO = 1 # file already retrieved
578 CLIVE_NOSUPPORT = 2 # host not supported
579 CLIVE_READ = 3 # file open/read error
580 CLIVE_GREP = 4 # grep: nothing matched in cache
581 CLIVE_OPTARG = 5 # invalid option argument
582 CLIVE_SYSTEM = 6 # system call failed (e.g. fork)
583 CLIVE_REGEXP = 7 # regexp pattern matching failed
584 CLIVE_FORMAT = 8 # requested format unavailable
585 CLIVE_NET = 9 # network error
586 CLIVE_STOP = 10 # --stop-after
591 <http://code.google.com/p/clive/>
594 <http://code.google.com/p/abby/>
596 Development repository can be cloned with:
597 % git clone git://repo.or.cz/clive.git
601 clive was originally written in Python but later (2.0.0)
602 rewritten in Perl. The project started as a workaround
603 to Youtube Adobe flash-plugin requirement which was
604 poorly supported on Unix-like systems.
608 Toni Gundogdu <legatvs@gmail.com>
610 Thanks to all those who have contributed to the project over the
611 years by sending patches, reporting bugs and writing feedback.
612 You know who you are.