3 clive - the non-interactive video extraction utility
7 clive [option]... [URL]...
11 clive is a command line utility for extracting videos from Youtube and other
12 video sharing Web sites. It was originally written to bypass the Adobe Flash
13 requirement needed to view the hosted videos.
15 clive can be non-interactive. This means it can work in the background
16 while the user is not logged on, allowing the user to start an extraction
17 and disconnect from the system and letting clive finish the work.
19 clive was influenced by the GNU L<wget(1)> utility. Kudos to the wget team
20 for their original work.
24 You may freely mix different option styles, or specify options after the
25 command-line arguments. For example:
26 % clive -c --format=mp4 URL -n
28 You may also put several options together that do not require arguments.
32 Which is equivalent to:
41 Print this help and exit.
45 Print version and exit.
49 Print supported hosts and exit.
51 =item B<-b --background>
53 Go to background immediately after startup. If no output file is specified
54 using the B<--output> or B<--append>, the output is redirected to I<clive-log>
55 file. Implies B<--progress=>I<dot>.
57 =item B<-e --emit-csv>
59 Do not extract any videos, instead print the video details as CSV to stdout.
60 Ignores the B<--quiet> option for CSV.
62 =item B<-E --emit-xml>
64 Equivalent to B<--emit-csv> in every way but the output is XML. Negates
67 =item B<-V --clivepass=>I<path>
69 I<path> to the L<clivepass(1)> utility. If CLIVEPASS_PATH environment
70 variable is set, it will will used.
78 =item B<-U --agent=>I<string>
80 Identify as I<string> to the HTTP server. Defaults is "Mozilla/5.0".
82 =item B<-y --proxy=>I<address>
84 Use I<address> for HTTP proxy, e.g. http://foo:1234. If undefined,
85 uses the http_proxy environment setting if present.
87 =item B<-X --no-proxy>
89 Do not use HTTP proxy.
91 =item B<--connect-timeout=>I<seconds>
93 Maximum time in I<seconds> allowed the connection to the server to take.
94 This limits only the connection phase. B<clive> will only timeout
95 on the system's internal timeouts by default.
97 http://curl.haxx.se/docs/knownbugs.html:
99 "34. The SOCKS4 connection codes don't properly acknowledge
100 (connect) timeouts. Also see #12. According to bug #1556528,
101 even the SOCKS5 connect code does not do it right:
102 http://curl.haxx.se/bug/view.cgi?id=1556528"
104 "12. When connecting to a SOCKS proxy, the (connect) timeout
105 is not properly acknowledged after the actual TCP connect
106 (during the SOCKS 'negotiate' phase)."
108 See also B<--connect-timeout-socks>.
110 =item B<--connect-timeout-socks=>I<seconds>
112 Same as B<--connect-timeout> but works around the SOCKS proxy bug by using
113 CURLOPT_TIMEOUT instead of CURLOPT_CONNECTIMEOUT.
115 Ignored for file transfers, otherwise curl could interrupt started transfers.
117 More about these curl options:
118 http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
128 Renew cache entries for the re-visited URLs. See L</CACHE> in the manual page.
132 Print all cache entries to standard output.
134 =item B<-H --show-format=>I<string>
136 Format printed entries using the I<string>. Default is '%D: "%t" | %mMB'.
137 The following specifiers are supported:
138 %t = video page title
140 %l = video file length (bytes)
142 %x = video extraction url
143 %D = video extraction date
144 %T = video extraction time
145 %S = video extraction timestamp (same as: %D %T)
147 =item B<-g --grep=>I<pattern>
149 Grep cache entries for I<pattern>. Matching is B<not> restricted to any
150 specific cache entry field (e.g. title or id).
152 % clive --grep=^git --ignore-case
154 The matched entries are then extracted. To only show the matches, use the
155 B<--show> option. For example:
158 =item B<-i --ignore-case>
160 Causes clive to ignore case-differences with B<--grep>.
164 Delete grep-matched entries from the cache. If used with the B<--show>
165 option, clive will prompt before deleting, e.g.:
170 Clear cache and exit.
174 B<Logging and Input Options>
178 =item B<-o --output=>I<logfile>
180 Log all messages to I<logfile>. The messages are normally reported to
183 =item B<-a --append=>I<logfile>
185 Append to I<logfile>. This is the same as B<--output> but appends to
186 I<logfile> instead of overwriting it. If the I<logfile> does not exist,
191 Causes the program to print Curl debug messages.
199 Recall the last URL batch from the previous runtime. Note that clive overwrites
200 this information at every runtime. See also L</CACHE>.
202 =item B<-T --savebatch=>I<file>
204 Save current URL batch to I<file>.
208 Paste input from clipboard. The pasted URLs are expected to be separated with
213 Redirect all input to stderr, even if no TTY is available.
221 =item B<-O --output-video=>I<file>
223 Write video to I<file>.
225 =item B<-n --no-extract>
227 Do not actually extract any videos.
229 =item B<-c --continue>
231 Continue extraction of a partially downloaded file. Note that this works only
232 with HTTP servers that support the "Range" header. Ignored unless I<localfile>
235 The "requested range was not delivered" error typically implies that the
236 host does not allow continuing partially extracted video files. You will
237 see this error if you attempt to continue a partially downloaded flv video
238 from Youtube, for example.
240 =item B<-W --overwrite>
242 Causes clive to overwrite already existing video files. By default clive
243 appends a numeric suffix to the output filename.
245 =item B<-G --progress=>I<type>
247 Select the I<type> of the progress indicator. Accepted types are "bar",
250 The "bar" indicator is used by default. It spews out some details indicating
251 the status of retrieval. If the output is not a TTY, the "dot" bar will be
254 Use --progress=dot to switch to the "dot" display. It traces the retrieval
255 by printing dots on the screen, each dot representing a fixed amount of
258 When using the dotted retrieval, you may also set the style by specifying
259 the I<type> as dot:style.
260 --progress=dot:default
261 each dot 1K, 10-dots clusters, 50-dots lines
262 --progress=dot:binary
263 each dot 8K, 16-dots clusters, 48-dots lines
265 each dot 64K, 8-dots clusters, 48-dots lines
267 Progress indication can be disabled completely by setting I<type> to "none".
269 =item B<-u --youtube-user=>I<username>
271 Causes B<clive> to log into Youtube as I<username>. This is done I<before>
272 extraction, even if the batch does not contain any Youtube video links.
274 =item B<-t --youtube-pass=>I<password>
276 Use I<password> for Youtube login. If undefined, B<clive> prompts for it.
278 =item B<-L --no-login>
280 Do not log into Youtube.
282 =item B<-S --savedir=>I<dir>
284 Save extracted videos to I<dir>.
286 =item B<-f --format=>I<format>
288 Extract I<format> of the video. All supported websites have at least the flv
289 format available. See L</FORMATS> for more.
291 =item B<-l --cclass=>I<character-class>
293 Use I<character-class> to filter the video page titles. Default is B<\w>.
294 The filtered video page titles are used to name the extracted videos.
296 =item B<-N --filename-format=>I<string>
298 Use I<string> to construct the name of the extracted video. Default is
299 "%t.%s". The following specifiers are supported:
300 %t = video title (after applying --cclass)
306 %S = timestamp (same as %D %T)
308 =item B<-x --exec=>I<command>B<;>
310 Execute I<command>. Optional arguments may be passed to the command.
311 The expression must be terminated by a semicolon (";"). If the specifier
312 "%i" appears anywhere in I<command> name or the arguments it is replaced
313 by the pathname of the extracted video file.
316 % clive --exec="mplayer %i;" URL
318 =item B<-x, --exec=>I<command>B<+>
320 Same as B<--exec>, except that "%i" is replaced with as many pathnames
321 as possible for the invocation of I<command>.
323 =item B<--stream-exec=>I<command>
325 Execute I<command> when file transfer progress reaches I<percent> as defined
326 with the B<--stream> option. If a "%i" specifier is used in the I<command>
327 expression, it will be replaced with video pathname.
329 B<clive> forks the I<command> and continues with the transfer. Note that
330 B<clive> will not continue to download another file before the child process
333 Some players and formats may not work properly with this feature.
334 L<mplayer(1)> is known to work properly with most flv videos.
336 =item B<--stream=>I<percent>
338 Execute the I<command> defined with the B<--stream-exec> option when the
339 file transfer progress reaches I<percent>.
342 % clive --stream-exec="mplayer -really-quiet %i" --stream=20 URL
344 Note that B<clive> does nothing to check if there is enough data buffered
345 before forking the stream process. If the transfer rate drops significantly
346 after starting the process and it runs out of data, B<clive> does nothing
349 =item B<--limit-rate=>I<amount>
351 Limit video download rate to I<amount> KB/s.
361 % clive "http://youtube.com/watch?v=3HD220e0bx4"
363 =item Multiple video URLs:
365 % cat E<gt>E<gt> url.lst
366 http://en.sevenload.com/videos/IUL3gda-Funny-Football-Clips
367 http://youtube.com/watch?v=3HD220e0bx4
368 http://break.com/index/beach-tackle-whip-lash.html
369 http://www.liveleak.com/view?i=704_1228511265
371 % cat url.lst | clive
373 =item Combine input from multiple sources:
377 =item Recall the last URL batch, extract mp4 video format:
381 =item Grep for the pattern in cache, extract matched videos:
383 % clive -g 3HD220e0bx4
385 =item Same as above but I<delete> the matched entries from cache:
389 Note the use of -i (--ignore-case).
391 =item Print cache contents to stdout:
395 =item Combine matching and printing cache entries:
399 Note that -s negates video extraction.
401 =item Match, redirect output to I<my.log>, go to background:
403 % clive -big ^3hd2 -o my.log
405 =item Extract video from URL, re-encode audio to mp3:
407 % clive --exec="ffmpeg -y -i %i %i.mpg;" URL
409 =item Login to Youtube to extract a video:
411 % clive -u myusername -t mypassword URL
413 This isn't typically needed unless you are trying to extract videos flagged
416 =item Use protected login password:
418 Otherwise same as above but the login info is saved to a clivepass password
419 repository file. The saved passwords are encrypted.
421 Note that you will need to install the clive-utils for this. See the L</OTHER>
424 Create new password repository:
427 Add new login name "myusername":
428 % clivepass --add myusername
430 Use the created login info from clive:
431 % clive -u myusername -t - -V /usr/bin/clivepass URL
433 This causes B<clive> to request for the login password for "myusername" from
434 L<clivepass(1)>. The password is then used to login to Youtube before extracting
435 the video from the URL.
441 clive defaults to extract the flv format unless the B<--format> option is
442 used. The requested format may not always be available. The server typically
443 returns one of the HTTP/40x errors if the requested format is not available.
445 The quality of the video depends on the uploaded video quality. Each website
446 typically recompresses the uploaded videos to 320x240 resolution (sometimes
447 higher). Because this varies per video and website, you should treat the
448 list below only as a rough reference of what formats should be available.
452 =item B<www.youtube.com>
456 Formats: (flv|mp4|mp4_hd|3gpp|xflv) Continue: Yes (except flv)
458 mp4 and 3gpp formats are typically, or will soon become, available.
459 xflv appears to be very rare. Videos dating back to 2006 may only be
460 available as flv. The high-definition mp4_hd format is currently available
461 for a limited number of videos, which are usually explicitly marked as "HD"
464 Some of the videos displayed on last.fm site are actually Youtube videos.
465 You can pass the lastfm'ed video URLs to B<clive>.
471 =item B<video.google.com>
473 Formats: (flv|mp4) Continue: Only mp4
475 The mp4 may not always be available. GoogleVideo mp4 are encoded using a
476 different codec than Youtube mp4 videos. You can typically B<--stream-exec>
477 Youtube mp4 videos but this is not case with GoogleVideo mp4 videos. If you
478 find a way, drop an email.
484 =item B<www.sevenload.com>
486 Formats: flv Continue: Yes
492 =item B<www.break.com>
494 Formats: flv Continue: Yes
500 =item B<www.liveleak.com>
502 Formats: flv Continue: Yes
504 =item B<www.evisor.tv>
506 Formats: flv Continue: Yes
508 =item B<www.dailymotion.com>
510 Formats: (spark|spak-mini|vp6-hq|vp6-hd|vp6|h264) Continue: Yes
512 spark is the "regular" 320x240 flv. B<clive> defaults to spark
513 if requested format is not available.
517 Formats: flv Continue: Yes
523 The cache has two purposes:
529 Keep reusable video data available for fast re-extraction.
533 Keep a record of extracted videos.
537 Each cache entry contains information about a video, including, but not limited
538 to, page title, file length and extraction URL. You can use the B<--grep> option
539 to match these details when you are searching for videos from the cache. See
540 the L</EXAMPLES> section for examples of how to do this.
542 Some entries may need to be renewed from time to time as certain websites have
543 their extraction URLs expire after awhile. Youtube is an example of this.
544 Youtube servers usually return the HTTP/410 error if the extraction URL has
545 expired. You can use the B<--renew> option to fix this.
547 Note that B<clive> renews the cache entry automatically if the video format
548 does not match the cached video format string. This is done for two reasons:
554 Cached video extraction URL points to a different location
558 Cached video file length is likely to be incorrect
564 clive searches the ~/.config/clive directory for the config file. You can
565 override this by setting the B<CLIVE_HOME> environment variable.
569 =item ~/.config/clive/config
571 Configuration file for clive.
573 =item ~/.config/clive/cache
575 Contains the cache entries of the visited URLs. A Berkeley DB (Hash) file.
577 =item ~/.config/clive/recall
579 Contains the last URL batch. You can B<--recall> this file.
585 clive exits 0 on success, and E<gt>0 if an error occurs.
589 L<clivefeed(1)> L<clivescan(1)> L<clivepass(1)>
594 <http://clive.sourceforge.net/>
597 <http://code.google.com/p/clive/>
599 Development repository can be cloned with:
600 % git clone git://repo.or.cz/clive.git
602 For release announcements, subscribe to the project at:
603 <http://freshmeat.net/projects/clive/>
605 Additional utilities (clive-utils):
606 <http://code.google.com/p/clive-utils/>
610 Written by Toni Gundogdu <legatvs@gmail.com>.