bin/clive

   1 #!/usr/bin/perl
   2 # -*- coding: ascii -*-
   3 ###########################################################################
   4 # clive, command line video extraction utility.
   5 # Copyright 2007, 2008, 2009 Toni Gundogdu.
   6 #
   7 # This file is part of clive.
   8 #
   9 # clive is free software: you can redistribute it and/or modify it under
  10 # the terms of the GNU General Public License as published by the Free
  11 # Software Foundation, either version 3 of the License, or (at your option)
  12 # any later version.
  13 #
  14 # clive is distributed in the hope that it will be useful, but WITHOUT ANY
  15 # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
  16 # FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
  17 # details.
  18 #
  19 # You should have received a copy of the GNU General Public License along
  20 # with this program. If not, see <http://www.gnu.org/licenses/>.
  21 ###########################################################################
  22 use warnings;
  23 use strict;
  24
  25 use clive::App;
  26
  27 clive::App->main;
  28
  29
  30 __END__
  31
  32 =head1 NAME
  33
  34 clive - command line video extraction utility
  35
  36 =head1 SYNOPSIS
  37
  38 clive [options]... [URL]...
  39
  40 =head1 DESCRIPTION
  41
  42 clive is a command line video extraction utility for Youtube and other
  43 video-sharing websites. It was written to work around the Adobe Flash
  44 plugin requirement.
  45
  46 clive is not an universal video extraction utility. It only works with
  47 the video hosts that it supports.
  48
  49 =head1 OPTIONS
  50
  51  -h, --help                     print help and exit
  52  -v, --version                  print version and exit
  53      --hosts                    print supported hosts and exit
  54      --home-dir=HOMEDIR         use HOMEDIR instead of $HOME
  55 Output Options:
  56      --emit-csv                 emit video details in CSV to stdout
  57      --debug                    print cURL debug messages
  58  -q, --quiet                    turn off all output
  59  -l, --recall                   recall last input
  60      --recall-file=FILE         use FILE instead of default ~/.clivelast
  61      --stderr                   redirect all output to stderr
  62 HTTP Options:
  63      --agent=STRING             identify as STRING to http server
  64      --connect-timeout=SECS     max time allowed connection to take
  65      --connect-timeout-socks=S  same as above, tries to workaround SOCKS
  66      --proxy=ADDR               use ADDR for http proxy
  67      --no-proxy                 do not use http proxy
  68 Cache Options:
  69      --cache-file=FILE          use FILE instead of default ~/.clivecache
  70  -r, --cache-read               enable reading from cache
  71  -d, --cache-dump               dump cache records to stdout
  72      --cache-dump-format=STRING print format for dumping cache records
  73      --cache-grep=PATTERN       grep cache records for PATTERN
  74  -i, --cache-ignore-case        ignore case while matching records
  75  -D, --cache-remove-record      remove matched records from cache
  76      --cache-clear              truncate cache records
  77      --no-cache                 disable cache all (read and write) use
  78 Download Options:
  79  -f, --format=FORMAT            extract FORMAT of video
  80  -O, --output-file=FNAME        write video to file
  81  -n, --no-extract               do not extract any videos
  82  -c, --continue                 continue partially downloaded file
  83      --save-dir=DIR             save video files to DIR
  84      --cclass=CLASS             use character CLASS to filter titles
  85  -C, --no-cclass                do not apply character class
  86      --filename-format=STRING   use STRING to format output filename
  87      --exec=CMD                 run CMD subsequently after file transfer
  88      --stream=PERCENT           run stream command (below) at PERCENT
  89      --stream-exec=CMD          stream command to run
  90      --limit-rate=AMOUNT        limit transfer rate to AMOUNT (KB/s)
  91      --stop-after=SIZE|PERCENT  stop file transfer after SIZE or PERCENT
  92  -R, --raw                      process video page html on as-is basis
  93
  94 See clive manual page for EXAMPLES.
  95
  96 =head1 OPTION SYNTAX
  97
  98 You may freely mix different option styles and specify options after
  99 the command line arguments, e.g.:
 100   % clive -c URL --format=best
 101
 102 You may also put several options together that do not require arguments:
 103   % clive -cnrf best URL
 104
 105 Note that the "dashed" options have aliases available which are not
 106 documented in the L</OPTIONS> section. For example:
 107   % clive --no-extract --no_extract --noextract
 108   % clive --cache-read --cache_read --cacheread
 109
 110 =head1 OPTION DESCRIPTIONS
 111
 112 =over 4
 113
 114 =item B<-h, --help>
 115
 116 Print help and exit.
 117
 118 =item B<-v, --version>
 119
 120 Print version and exit.
 121
 122 =item B<--hosts>
 123
 124 Print supported hosts, with precompiled regexps used to identify links,
 125 and exit.
 126
 127 =item B<--home-dir>=I<homedir>
 128
 129 Use I<homedir> instead of the default $HOME (and $CLIVE_HOME) environment
 130 variable for all non-config files (e.g. cache).
 131
 132 =back
 133
 134 B<Output options>
 135
 136 =over 4
 137
 138 =item B<--emit-csv>
 139
 140 Print (emit) video details in CSV format to standard output.
 141 Implies --no-extract.
 142
 143 =item B<--debug>
 144
 145 Print cURL debug (or verbose) messages to standard error.
 146
 147 =item B<-q, --quiet>
 148
 149 Turn off all output to standard output and error.
 150
 151 =item B<-l, --recall>
 152
 153 Recall (or reuse) the last input.
 154
 155 =item B<--recall-file>=I<file>
 156
 157 Use I<file> instead of I<homedir>/.clivelast.
 158
 159 =item B<--stderr>
 160
 161 Direct all output to standard error.
 162
 163 =back
 164
 165 B<HTTP Options>
 166
 167 =over 4
 168
 169 =item B<--agent>=I<string>
 170
 171 Identify clive as I<string> to HTTP servers. Defaults to "Mozilla/5.0".
 172
 173 =item B<--connect-timeout>=I<seconds>
 174
 175 Maximum time in I<seconds> allowed for connection to take. Defaults to 30.
 176
 177 =item B<--connect-timeout-socks>=I<seconds>
 178
 179 Same as above but tries to workaround the SOCKS proxy bug in cURL. Defaults
 180 to 30.
 181
 182 =item B<--proxy>=I<address>
 183
 184 Use I<address> for HTTP proxy. Example: "http://foo:1234".
 185
 186 =item B<--no-proxy>
 187
 188 Disable HTTP proxy use even if http_proxy environment variable is set.
 189
 190 =back
 191
 192 B<Cache Options>
 193
 194 =over 4
 195
 196 =item B<--cache-file>=I<file>
 197
 198 Use I<file> instead of I<homedir>/.clivecache.
 199
 200 =item B<-r, --cache-read>
 201
 202 Read video details from cache record if it exists. Allows clive to
 203 skip fetching the video page again. See L</CACHE> section for more
 204 on this.
 205
 206 =item B<-d, --cache-dump>
 207
 208 Dump cache records to standard output.
 209
 210 =item B<--cache-dump-format>=I<format-string>
 211
 212 Used to format the output of the above. Defaults to "%n: %t [%f, %mMB]".
 213
 214 Example:
 215   % clive --cache-dump --cache-dump-format="%d: %t"
 216
 217 Supported format specifiers:
 218   %t .. video page title
 219   %i .. video id
 220   %h .. video host
 221   %l .. video file length (bytes)
 222   %m .. video file length (MB)
 223   %d .. date (last update)
 224   %T .. time (last update)
 225   %s .. time stamp (same as "%d %T")
 226   %f .. video file format
 227   %n .. index
 228
 229 =item B<--cache-grep>=I<pattern>
 230
 231 Grep stored cache records for I<pattern>. See also L</EXAMPLES - ADVANCED USE>.
 232
 233 =item B<-i, --cache-ignore-case>
 234
 235 Ignore case-differences while matching records.
 236
 237 =item B<-D, --cache-remove-record>
 238
 239 Remove matched records from cache.
 240
 241 =item B<--cache-clear>
 242
 243 Truncate cache records.
 244
 245 =item B<--no-cache>
 246
 247 Disable all (read and write) cache use.
 248
 249 =back
 250
 251 B<Download Options>
 252
 253 =over 4
 254
 255 =item B<-f, --format>=I<format>
 256
 257 Extract I<format> of the video. If I<format> is set to I<best>, clive
 258 will attempt to extract the best video quality.
 259
 260 Note that the I<format> is strictly host specific. See the L</FORMATS>
 261 section for more on this.
 262
 263 =item B<-O, --output-file>=I<filename>
 264
 265 Write video to I<filename>. Not to be used when extracting
 266 multiple videos.
 267
 268 =item B<-n, --no-extract>
 269
 270 Do not extract the video.
 271
 272 =item B<-c, --continue>
 273
 274 Continue partially downloaded video file.
 275
 276 =item B<--sav-dir>=I<dir>
 277
 278 Save extracted videos to I<dir>.
 279
 280 =item B<--cclass>=I<class>
 281
 282 Use character-I<class> to filter video page titles. Defaults to "\w".
 283 Refer to the Perl regular expressions (character classes) for more on
 284 this.
 285
 286 =item B<-C, --no-cclass>
 287
 288 Negates the use of the character-class.
 289
 290 =item B<filename-format>=<format-string>
 291
 292 Use <format-string> to format output video filenames. Defaults to "%t.%s".
 293
 294 Supported format specifiers:
 295   %t .. video page title (after applying character-class filter)
 296   %s .. video file suffix (e.g. "flv")
 297   %i .. video id
 298   %h .. video host
 299
 300 =item B<--exec>=I<command>;
 301
 302 Execute I<command> subsequently after video file transfer. Optional
 303 arguments may be passed to the command. The expression must be
 304 terminated by a semicolon (";"). If the specifier "%i" appears
 305 anywhere in the I<command>, it is replaced by the pathname of the
 306 extracted video file.
 307
 308 =item B<--exec>=I<command>+
 309
 310 Same as above but "%i" is replaced with as many pathnames as
 311 possible for the invocation of I<command>.
 312
 313 =item B<--stream>=I<percent>
 314
 315 Execute --stream-exec=I<command> when file transfer reaches
 316 I<percent>.
 317
 318 =item B<--stream-exec>=I<command>
 319
 320 Execute (fork a child process) I<command> while transferring
 321 video file. This "simulates" streaming the media but does so
 322 without checking for buffer underruns so make sure you set
 323 --stream=I<percent> high enough and that you have a fast
 324 internet connection.
 325
 326 clive will not attempt to re-execute the command if it
 327 terminates before the file transfer finishes.
 328
 329 clive will wait that the child process terminates before
 330 it moves on to extract another file or exits if there
 331 are not any left.
 332
 333 Note that some video file formats (namely Google mp4) are
 334 known to B<not> to work with this feature.
 335
 336 =item B<--limit-rate>=I<amount>
 337
 338 Limit transfer rate to I<amount> KB/s.
 339
 340 =item B<--stop-after>=I<size|percent>
 341
 342 Stop file transfer after I<size> or I<percent>. The value must
 343 be terminated by either '%' or 'M'.
 344
 345 =item <-R, --raw>
 346
 347 Process video page HTML as it is. Technically, this means that
 348 clive will not try to decode the HTML using Encode::decode_utf8.
 349 This may be useful with some of the supported host such as
 350 Cctv which uses GBK as its HTML encoding.
 351
 352 =back
 353
 354 =head1 EXAMPLES - BASIC USE
 355
 356 =over 4
 357
 358 =item clive "http://youtube.com/watch?v=3HD220e0bx4"
 359
 360 Extracts video (flv) from the above video page link. You
 361 can then play the flv video file in a media player.
 362
 363 =item cat E<gt> url.lst
 364
 365   http://en.sevenload.com/videos/IUL3gda-Funny-Football-Clips
 366   http://youtube.com/watch?v=3HD220e0bx4
 367   http://break.com/index/beach-tackle-whip-lash.html
 368   http://www.liveleak.com/view?i=704_1228511265
 369
 370 =item cat url.lst | clive
 371
 372 You can feed clive multiple video page links like this
 373 or via command line:
 374
 375 =item clive URL URL2 URL3 URL4
 376
 377 When you are using the pipes, be sure to separate each link
 378 with a newline.
 379
 380 =item xclip -o | clive
 381
 382 Many X clipboard utilities exist. This example uses C<xclip(1)>
 383 and a pipe to paste the contents of the clipboard to clive.
 384
 385 =item clive -l
 386
 387 Recall last video page link input.
 388
 389 =back
 390
 391 =head1 EXAMPLES - ADVANCED USE
 392
 393 =over 4
 394
 395 =item clive -f best "http://youtube.com/watch?v=3HD220e0bx4"
 396
 397 Extract the best format of the video.
 398
 399 =item clive -r -f best "http://youtube.com/watch?v=3HD220e0bx4"
 400
 401 Causes clive to skip re-fetching the video page and parsing it.
 402 Instead it reuses the record saved in the cache from the previous
 403 time.
 404
 405 =item clive --cache-dump
 406
 407 Dump cache records to stdout. You can use --cache-dump-format
 408 to format the output.
 409
 410 =item clive -ig 3hd2
 411
 412 Grep for "3hd2" pattern in cache records. If pattern
 413 matches, clive continues to extract the matched videos.
 414 Note the use of "-i" (--cache-ignore-case).
 415
 416 =item clive -ig 3hd2 -D
 417
 418 Same as above but removes the record from cache instead of
 419 extracting the video.
 420
 421 =item clive --exec="ffmpeg -i %i %i.mp3;" URL
 422
 423 Extract video and use C<ffmpeg(1)> to copy audio from it
 424 to mp3 file.
 425
 426 =item clive --stream-exec="mplayer -really-quiet %i"
 427     --stream=25 URL
 428
 429 Start playing the video being extracted when the transfer
 430 reaches 25% complete.
 431
 432 =back
 433
 434 =head1 FORMATS
 435
 436 clive extracts flv (typically 320x240) by default from all
 437 supported websites. Some of them support also other formats.
 438
 439 =over 4
 440
 441 =item B<youtube.com>
 442
 443 =item B<last.fm>
 444
 445 Format: (flv|fmt17|fmt18|fmt22|fmt35)
 446
 447 flv (fmt34) and fmt18 (mp4) are usually available. Others may be
 448 available. At the time of writing this, the following formats
 449 are recognized by both clive and Youtube:
 450
 451   fmt22 .. mp4    (1280x720) (HD)
 452   fmt35 .. flv     (640x380) (HQ)
 453   fmt18 .. mp4     (480x360)
 454   flv   .. fmt34   (320x180)
 455   fmt17 .. 3gp     (176x144)
 456
 457 Some of the videos available at last.fm are actually Youtube
 458 videos. clive can handle such video links.
 459
 460 =item B<video.google.com>
 461
 462 Format: (flv|mp4)
 463
 464 mp4 format is available for a limited number of videos.
 465
 466 =item B<dailymotion.com>
 467
 468 Format: (flv|spak-mini|vp6-hq|vp6-hd|vp6|h264)
 469
 470 The HD and HQ videos may not always be available.
 471
 472   ON2-1280x720 (vp6-hd)
 473   ON2-848x480  (vp6-hq)
 474   H264-512x384 (h264)
 475   ON2-320x240  (vp6)
 476   FLV-320x240  (flv/spark)
 477   FLV-80x60    (spak-mini)
 478
 479 =item B<vimeo.com>
 480
 481 Format: (flv|hd)
 482
 483 HD should be available for the vimeo.com/hd channel videos at least.
 484 Note that "flv" only means "default" here, as some of the hosted
 485 videos are encoded (as default) in other video formats such as
 486 "mp4" rather than "flv".
 487
 488 For further reading:
 489   http://vimeo.com/help/hd
 490
 491 =item B<evisor.tv>
 492
 493 =item B<liveleak.com>
 494
 495 =item B<tv.cctv.com>
 496
 497 =item B<sevenload.com>
 498
 499 =item B<break.com>
 500
 501 =item B<redtube.com>
 502
 503 Format: flv
 504
 505 =back
 506
 507 =head1 FILES
 508
 509 =over 4
 510
 511 =item ~/.cliverc
 512
 513 User configuration file. You can append the command line options to
 514 the config file, e.g.:
 515   % cat >> ~/.cliverc
 516   -f best
 517   --proxy=http://foo:1234
 518
 519 =item ~/.clivelast
 520
 521 File containing last user input.
 522
 523 =item ~/.clivecache
 524
 525 BerkeleyDB based cache containing the records of fetched and
 526 parsed video pages.
 527
 528 =item Notes
 529
 530 Note that the default path (~) for all non-config files can be altered
 531 with CLIVE_HOME environment variable and --home-dir option.
 532
 533 =back
 534
 535 =head1 CACHE
 536
 537
 538 The purpose of the cache is to allow clive to skip fetching
 539 and parsing the video page again. It does not contain any
 540 actual video data so one should not expect to recover a deleted
 541 video file from the cache. Only some of the parsed details
 542 are stored as records to the cache.
 543
 544 By now, it is should be a well known fact that the cache fails
 545 with some of the supported hosts. For example Youtube video links
 546 expire after some time, this causes the re-extraction to fail if
 547 the cached video link is used later again.
 548
 549 This was the main reason why in 2.2.0 reading from cache was
 550 disabled by default. Many users reported the reuse of expired
 551 video links as a bug previously even though it was well documented
 552 in the manual page explaining that most of the HTTP 403/404 errors
 553 were actually caused by expired video links.
 554
 555 It is, of course, still possible to read from cache. You can
 556 enable this by invoking the --cache-read option. This causes
 557 clive to look up a saved cache record and reuse the stored
 558 video details if they are found instead of fetching the video
 559 page.
 560
 561 The use of the cache can be disabled with the --no-cache option.
 562 This disables both read and write. Note that if the BerkeleyDB Perl
 563 module is not installed, clive will not use the cache.
 564
 565 See also the --cache-grep option.
 566
 567 =head1 BUGS
 568
 569 You are welcome to submit bug reports via the Issue tracker at:
 570   <http://code.google.com/p/clive/issues/>
 571
 572 =head1 EXIT STATUS
 573
 574 clive exits 0 on success, and E<gt>0 if an error occurs.
 575
 576  CLIVE_OK          = 0
 577  CLIVE_NOTHINGTODO = 1    # file already retrieved
 578  CLIVE_NOSUPPORT   = 2    # host not supported
 579  CLIVE_READ        = 3    # file open/read error
 580  CLIVE_GREP        = 4    # grep: nothing matched in cache
 581  CLIVE_OPTARG      = 5    # invalid option argument
 582  CLIVE_SYSTEM      = 6    # system call failed (e.g. fork)
 583  CLIVE_REGEXP      = 7    # regexp pattern matching failed
 584  CLIVE_FORMAT      = 8    # requested format unavailable
 585  CLIVE_NET         = 9    # network error
 586  CLIVE_STOP        = 10   # --stop-after
 587
 588 =head1 OTHER
 589
 590 Project page:
 591   <http://code.google.com/p/clive/>
 592
 593 Front-end (GUI):
 594   <http://code.google.com/p/abby/>
 595
 596 Development repository can be cloned with:
 597   % git clone git://repo.or.cz/clive.git
 598
 599 =head1 HISTORY
 600
 601 clive was originally written in Python but later (2.0.0)
 602 rewritten in Perl. The project started as a workaround
 603 to Youtube Adobe flash-plugin requirement which was
 604 poorly supported on Unix-like systems.
 605
 606 =head1 AUTHOR
 607
 608 Toni Gundogdu <legatvs@gmail.com>
 609
 610 Thanks to all those who have contributed to the project over the
 611 years by sending patches, reporting bugs and writing feedback.
 612 You know who you are.
 613
 614 =cut