bin/clive

   1 #!/usr/bin/perl
   2 # -*- coding: ascii -*-
   3 ###########################################################################
   4 # clive, command line video extraction utility.
   5 # Copyright 2007, 2008, 2009 Toni Gundogdu.
   6 #
   7 # This file is part of clive.
   8 #
   9 # clive is free software: you can redistribute it and/or modify it under
  10 # the terms of the GNU General Public License as published by the Free
  11 # Software Foundation, either version 3 of the License, or (at your option)
  12 # any later version.
  13 #
  14 # clive is distributed in the hope that it will be useful, but WITHOUT ANY
  15 # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
  16 # FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
  17 # details.
  18 #
  19 # You should have received a copy of the GNU General Public License along
  20 # with this program. If not, see <http://www.gnu.org/licenses/>.
  21 ###########################################################################
  22 use warnings;
  23 use strict;
  24
  25 binmode( STDOUT, ":utf8" );
  26 binmode( STDERR, ":utf8" );
  27
  28 use clive::App;
  29
  30 clive::App->main;
  31
  32 __END__
  33
  34 =head1 NAME
  35
  36 clive - command line video extraction utility
  37
  38 =head1 SYNOPSIS
  39
  40 clive [options]... [URL]...
  41
  42 =head1 DESCRIPTION
  43
  44 clive is a command line video extraction utility for Youtube and other similar
  45 video-sharing websites. It was written to work around the Adobe Flash plugin
  46 requirement as the technology is poorly supported on Unix-like systems.
  47
  48 clive is not an universal video extraction utility. In fact, it supports only
  49 a number of video websites. Each website typically exposes access to the video
  50 content in a very different way, meaning that clive has to be customized for
  51 each website in order to download any videos from them.
  52
  53 =head1 OPTIONS
  54
  55  -h, --help                     print help and exit
  56  -v, --version                  print version and exit
  57      --hosts                    print supported hosts and exit
  58      --upgrade-config           upgrade 2.0/2.1 config to 2.2+ format
  59  -l, --last                     recall last input
  60      --last-file=FILE           read/write FILE instead of default path
  61 Output Options:
  62      --emit-csv                 emit video details in CSV to stdout
  63      --debug                    print cURL debug messages
  64  -q, --quiet                    turn off all output
  65      --stderr                   redirect all output to stderr
  66      --print-fname              print filename before download starts
  67 HTTP Options:
  68      --agent=STRING             identify as STRING to http server
  69      --connect-timeout=SECS     max time allowed connection to take
  70      --connect-timeout-socks=S  same as above, tries to workaround SOCKS
  71      --proxy=ADDR               use ADDR for http proxy
  72      --no-proxy                 disable all use of http proxy
  73      --cookie-jar=FILE          enable cookies, write them to FILE
  74 Cache Options:
  75      --cache-file=FILE          read/write FILE instead of default path
  76  -r, --cache-read               enable reading from cache
  77  -d, --cache-dump               dump cache records to stdout
  78      --cache-dump-format=STRING print format for dumping cache records
  79      --cache-grep=PATTERN       grep cache records for PATTERN
  80  -i, --cache-ignore-case        ignore case while matching records
  81  -D, --cache-remove-record      remove matched records from cache
  82      --cache-clear              truncate cache records
  83      --no-cache                 disable cache all (read and write) use
  84 Download Options:
  85  -f, --format=FORMAT            extract FORMAT of video
  86  -O, --output-file=FILE         write video to file
  87  -c, --continue                 continue partially downloaded file
  88  -n, --no-extract               do not extract any videos
  89      --save-dir=DIR             save video files to DIR
  90      --cclass=CLASS             use character CLASS to filter titles
  91  -C, --no-cclass                do not apply character class
  92      --filename-format=STRING   use STRING to format output filename
  93      --exec=CMD                 command to run when transfer finishes
  94  -e, --exec-run                 invoke command defined with --exec
  95      --stream-exec=CMD          stream command to be invoked
  96      --stream=PERCENT           invoke --stream-exec when transfer reaches %
  97  -s, --stream-pass              pass video link to --stream-exec command
  98      --limit-rate=AMOUNT        limit transfer rate to AMOUNT (KB/s)
  99      --stop-after=SIZE|PERCENT  stop file transfer after SIZE or PERCENT
 100
 101 =head1 OPTION SYNTAX
 102
 103 You may freely mix different option styles and specify options after
 104 the command line arguments, e.g.:
 105   % clive -c URL --format=best
 106
 107 You may also put several options together that do not require arguments:
 108   % clive -cnrf best URL
 109
 110 Note that the "dashed" options have aliases. For example:
 111   % clive --no-extract --no_extract --noextract
 112   % clive --cache-read --cache_read --cacheread
 113
 114 =head1 OPTION DESCRIPTIONS
 115
 116 =over 4
 117
 118 =item B<-h, --help>
 119
 120 Print help and exit.
 121
 122 =item B<-v, --version>
 123
 124 Print version and exit.
 125
 126 =item B<--hosts>
 127
 128 Print supported hosts with available formats and exit.
 129
 130 =item B<--upgrade-config>
 131
 132 Upgrade clive 2.0/2.1 config to current 2.2+ format and exit.
 133
 134 =item B<-l, --last>
 135
 136 Re-feed the previously fed video page links from the last run time.
 137
 138 =item B<--last-file>=I<path>
 139
 140 Use I<path> instead of the default path. See also L</FILES>.
 141
 142 =back
 143
 144 B<Output options>
 145
 146 =over 4
 147
 148 =item B<--emit-csv>
 149
 150 Print (or emit) video details in CSV format to standard output.
 151 Implies --no-extract.
 152
 153 =item B<--print-fname>
 154
 155 Print filename on a separate line before download starts.
 156
 157 =item B<--debug>
 158
 159 Print cURL debug (or verbose) messages to standard error.
 160
 161 =item B<-q, --quiet>
 162
 163 Turn off all output to standard output and error.
 164
 165 =item B<--stderr>
 166
 167 Direct all output to standard error.
 168
 169 =back
 170
 171 B<HTTP Options>
 172
 173 =over 4
 174
 175 =item B<--agent>=I<string>
 176
 177 Identify clive as I<string> to HTTP servers. Defaults to "Mozilla/5.0".
 178
 179 =item B<--connect-timeout>=I<seconds>
 180
 181 Maximum time in I<seconds> allowed for connection to take. Defaults to 30.
 182
 183 =item B<--connect-timeout-socks>=I<seconds>
 184
 185 Same as above but tries to workaround the SOCKS proxy bug in cURL.
 186 Defaults to 30.
 187
 188 =item B<--proxy>=I<address>
 189
 190 Use I<address> for HTTP proxy. Example: "http://foo:1234".
 191
 192 =item B<--no-proxy>
 193
 194 Disable all use of HTTP proxy, even if http_proxy environment variable is set.
 195
 196 =item B<--cookie-jar>=I<file>
 197
 198 Enable cookies, which are otherwise rejected by default, and have libcurl
 199 to write them to I<file>. Specify "-" to instead to have the cookies written
 200 to stdout.
 201
 202 =back
 203
 204 B<Cache Options>
 205
 206 =over 4
 207
 208 =item B<--cache-file>=I<path>
 209
 210 Use I<path> instead of the default path. See L</FILES>.
 211
 212 =item B<-r, --cache-read>
 213
 214 Read video details from cache record if it exists. Allows clive to
 215 skip video page fetching and parsing again. See L</CACHE> section for more
 216 on this.
 217
 218 =item B<-d, --cache-dump>
 219
 220 Dump cache records to standard output.
 221
 222 =item B<--cache-dump-format>=I<format-string>
 223
 224 Used to format the output of the above. Defaults to "%n: %t [%f, %mMB]".
 225
 226 Example:
 227   % clive --cache-dump --cache-dump-format="%d: %t"
 228
 229 Supported format specifiers:
 230   %t .. video page title
 231   %i .. video id
 232   %h .. video host
 233   %l .. video file length (bytes)
 234   %m .. video file length (MB)
 235   %d .. date (last update)
 236   %T .. time (last update)
 237   %s .. time stamp (same as "%d %T")
 238   %f .. video file format
 239   %n .. index
 240
 241 =item B<--cache-grep>=I<pattern>
 242
 243 Grep stored cache records for I<pattern>. See also L</EXAMPLES - ADVANCED USE>.
 244
 245 =item B<-i, --cache-ignore-case>
 246
 247 Ignore case-differences while matching records.
 248
 249 =item B<-D, --cache-remove-record>
 250
 251 Remove matched records from cache.
 252
 253 =item B<--cache-clear>
 254
 255 Truncate cache records.
 256
 257 =item B<--no-cache>
 258
 259 Disable all (read and write) cache use.
 260
 261 =back
 262
 263 B<Download Options>
 264
 265 =over 4
 266
 267 =item B<-f, --format>=I<format>
 268
 269 Download the I<format> of the video. If I<format> is set to I<best>, clive
 270 will attempt to download the best quality of the video.
 271
 272 Note that the I<format> is strictly host specific. See the L</FORMATS>
 273 section for more on this.
 274
 275 =item B<-n, --no-extract>
 276
 277 Do not extract the video. In other words: simulate only to the point
 278 that clive verifies the video link after fetching and parsing the
 279 video page.
 280
 281 =item B<-O, --output-file>=I<file>
 282
 283 Write video to I<file>. Overwrites an already existing file.
 284
 285 Do not use this option when you are downloading more than one video
 286  on one go.
 287
 288 See also the note below.
 289
 290 =item B<-c, --continue>
 291
 292 Continue partially downloaded video file.
 293
 294 Note that, by default, clive appends a numeric suffix to the filename
 295 if the file exists already. That is unless:
 296
 297   * file is already completely retrieved, or:
 298   * -c or -O is used
 299
 300 =item B<--save-dir>=I<dir>
 301
 302 Save extracted videos to I<dir>. clive defaults to the current working
 303 directory.
 304
 305 =item B<--cclass>=I<class>
 306
 307 Use character-I<class> to filter video page titles. Defaults to "\w".
 308 This is a Perl regular expression character class. For example:
 309 "[A-Za-z0-9]".
 310
 311 =item B<-C, --no-cclass>
 312
 313 Disables the use of B<--cclass>. Causes clive to use the video page
 314 title as it is for output filename.
 315
 316 =item B<--filename-format>=I<format-string>
 317
 318 Use I<format-string> to format output video filenames. Default is "%t.%s".
 319
 320 Supported format specifiers:
 321   %t .. video page title (after applying character-class filter)
 322   %s .. video file suffix (e.g. "flv")
 323   %i .. video id
 324   %h .. video host
 325
 326 =item B<--exec>=I<command>;
 327
 328 Defines the I<command> to run when video file transfer completes.
 329 Note that B<--exec-run> must be used to actually cause clive
 330 to invoke the defined I<command>.
 331
 332 Optional arguments may be passed to the command. The expression must be
 333 terminated by a semicolon (";"). If the specifier "%i" appears anywhere
 334 in the I<command>, it is replaced by the pathname of the extracted
 335 video file.
 336
 337 =item B<--exec>=I<command>+
 338
 339 Same as above but "%i" is replaced with as many path names as
 340 possible for the invocation of I<command>.
 341
 342 =item B<-e, --exec-run>
 343
 344 Causes clive to invoke the command defined with B<--exec> when
 345 transfer finishes.
 346
 347 =item B<--stream-exec>=I<command>
 348
 349 Define the command to be invoked with B<--stream> and B<--stream-pass>.
 350 If a "%i" specifier is used in the I<command>, it will be replaced with
 351 either video file path name (B<--stream>) or parsed video link
 352 (B<--stream-pass>).
 353
 354 =item B<--stream>=I<percent>
 355
 356 Causes clive to fetch, parse, start download and eventually invoke
 357 the command defined with B<--stream-exec> when the transfer reaches
 358 the percentage defined with this option. See also L</EXAMPLES>.
 359
 360 Note that clive does nothing to check if there is enough data buffered
 361 before invoking the B<--stream-exec> defined I<command>. If the transfer
 362 rate drops significantly after starting the process and it runs out of data,
 363 clive does nothing to fix this.
 364
 365 Also note that clive will not continue to download another file before the
 366 child process exits.
 367
 368 This mode is supported for historical reasons. Consider using
 369 B<--stream-pass> instead.
 370
 371 =item B<-s, --stream-pass>
 372
 373 Otherwise the same as above but B<instead of> starting the download,
 374 clive passes the parsed video link to the command defined with
 375 B<--stream-exec>. See also L</EXAMPLES>.
 376
 377 This option was inspired by a C<clive(1)> wrapper script contributed
 378 by Bill Squire.
 379
 380 =item B<--limit-rate>=I<amount>
 381
 382 Limit transfer rate to I<amount> KB/s.
 383
 384 =item B<--stop-after>=I<size|percent>
 385
 386 Stop file transfer after I<size> or I<percent>. The value must
 387 be terminated by either '%' or 'M'.
 388
 389 =back
 390
 391 =head1 EXAMPLES - BASIC USE
 392
 393 =over 4
 394
 395 =item clive "http://youtube.com/watch?v=3HD220e0bx4"
 396
 397 Extracts video (flv) from the above video page link. You
 398 can then play the flv video file in a media player.
 399
 400 =item cat E<gt> url.lst
 401
 402   http://en.sevenload.com/videos/IUL3gda-Funny-Football-Clips
 403   http://youtube.com/watch?v=3HD220e0bx4
 404   http://break.com/index/beach-tackle-whip-lash.html
 405   http://www.liveleak.com/view?i=704_1228511265
 406
 407 =item cat url.lst | clive
 408
 409 You can feed clive multiple video page links like this
 410 or as command line arguments.
 411
 412 =item clive URL1 URL2 URL3 URL4
 413
 414 When you are using the pipes, be sure to separate each link with a newline.
 415
 416 =item xclip -o | clive
 417
 418 There are many X clipboard utilities. The above example uses C<xclip(1)>
 419 and a pipe to paste (or feed) the contents to clive.
 420
 421 =item clive -l
 422
 423 Recall last video page link input. Regardless the way they were fed
 424 to clive.
 425
 426 =back
 427
 428 =head1 EXAMPLES - ADVANCED USE
 429
 430 =over 4
 431
 432 =item clive -f best "http://youtube.com/watch?v=3HD220e0bx4"
 433
 434 Extract the best format of the video.
 435
 436 =item clive -r -f best "http://youtube.com/watch?v=3HD220e0bx4"
 437
 438 Same as above but read the cache record without fetching and
 439 parsing the video page again.
 440
 441 =item clive --cache-dump
 442
 443 Dump all cache records to stdout. You can use --cache-dump-format
 444 to format the output.
 445
 446 =item clive -ig 3hd2
 447
 448 Grep for "3hd2" pattern in cache records. If pattern matches, clive
 449 continues to extract the matched videos. Note the use of "-i"
 450 (--cache-ignore-case).
 451
 452 =item clive -ig 3hd2 -D
 453
 454 Same as above but removes the record from cache instead of extracting
 455 the video.
 456
 457 =item clive --exec="ffmpeg -i %i -acodec libvorbis %i.ogg;" -e URL
 458
 459 Copy audio from downloaded video to ogg with C<ffmpeg(1)>.
 460
 461 =item clive --stream-exec="mplayer -really-quiet %i" --stream=25 URL
 462
 463 Start playing the video being extracted when the transfer reaches
 464 25% complete.
 465
 466 =item echo '--stream-exec="mplayer -really-quiet %i"' >> ~/.cliverc
 467
 468 =item clive -s URL
 469
 470 Alternative to Adobe Flash. C<vlc(1)> and C<totem(1)> have been reported
 471 to work also.
 472
 473 =back
 474
 475 =head1 FORMATS
 476
 477 clive downloads "flv" by default from all of the supported websites.
 478
 479 =over 4
 480
 481 =item B<youtube.com>
 482
 483 =item B<last.fm>
 484
 485 Format: (flv|fmt17|fmt18|fmt22|fmt35)
 486
 487 flv (fmt34) and fmt18 (mp4) are usually available. Others may be
 488 available. At the time of writing this, the following formats are
 489 recognized by both clive and Youtube:
 490
 491   hd  .. fmt22 .. mp4  (1280x720)
 492   hq  .. fmt35 .. flv   (640x380)
 493   mp4 .. fmt18 .. mp4   (480x360)
 494   flv .. fmt34 .. flv   (320x180)
 495   3gp .. fmt17 .. 3gp   (176x144)
 496
 497 Note that you can use either format ID, e.g. fmt22 or hd.
 498
 499 Some of the last.fm videos are actually hosted by Youtube.
 500 clive can be used to download such videos.
 501
 502 =item B<video.google.com>
 503
 504 Format: (flv|mp4)
 505
 506 mp4 format is available for a limited number of videos.
 507
 508 =item B<dailymotion.com>
 509
 510 Format: (flv|spark-mini|vp6-hq|vp6-hd|vp6|h264)
 511
 512 The HD and HQ videos may not always be available.
 513
 514   vp6-hd    .. on2   (1280x720)
 515   vp6-hq    .. on2     (848x480)
 516   h264      .. h264    (512x384)
 517   vp6       .. on2     (320x240)
 518   flv       .. flv     (320x240)
 519   spark-mini .. flv       (80x60)
 520
 521 =item B<spiegel.de>
 522
 523 Format: (flv|vp6_(64|576|928)|h264_1400)
 524
 525   h264_1400 .. mp4 (996x560)
 526   vp6_928   .. flv (996x560)
 527   vp6_576   .. flv (560x315)
 528   flv       .. flv (180x100)
 529   vp6_64    .. flv (180x100)
 530
 531 Format: (3gp|small|iphone|podcast)
 532
 533 The data that clive parses indicates that these formats should be available
 534 although we are yet to find a video with these formats available.
 535 If you find one, let us know, too.
 536
 537   3gp       .. 3gp (?)
 538   small     .. 3gp (?)
 539   iphone    .. mp4 (?)
 540   podcast   .. mp4 (?)
 541
 542 =item B<golem.de>
 543
 544 Format: (flv|high|ipod)
 545
 546 =item B<vimeo.com>
 547
 548 Format: (flv|hd)
 549
 550 HD should be available for the vimeo.com/hd channel videos at least.
 551 Note that "flv" only means the "default flv". Some of the hosted
 552 "default" videos are actually "mp4", not "flv".
 553
 554 For further reading:
 555   http://vimeo.com/help/hd
 556
 557 =item B<evisor.tv>
 558
 559 =item B<liveleak.com>
 560
 561 =item B<tv.cctv.com>
 562
 563 =item B<sevenload.com>
 564
 565 =item B<break.com>
 566
 567 =item B<redtube.com>
 568
 569 =item B<ehrensenf.de>
 570
 571 =item B<clipfish.de>
 572
 573 Format: flv
 574
 575 =back
 576
 577 =head1 FILES
 578
 579 Should HOME environment variable be undefined for some reason, clive will
 580 use the current working directory instead.
 581
 582 =over 4
 583
 584 =item $HOME/.cliverc, $HOME/.clive/config, $HOME/.config/clive/config
 585
 586 User configuration file. For example:
 587   % cat >> ~/.cliverc
 588   -f best
 589   --proxy=http://foo:1234
 590
 591 =item $HOME/.cache/clive/last
 592
 593 File containing the last user input (video page links).
 594
 595 You can use --last-file to override the path, e.g.:
 596   --last-file=/path/to/last/file.
 597
 598 You can also define this option in the config file.
 599
 600 See also CLIVE_CACHE notes below.
 601
 602 =item $HOME/.cache/clive/cache
 603
 604 BerkeleyDB based cache file containing the records of fetched
 605 and parsed video pages.
 606
 607 You can use --cache-file to override the path., e.g.:
 608   --cache-file=/path/to/cache/file.
 609
 610 You can also define this option in the config file.
 611
 612 See also CLIVE_CACHE notes below.
 613
 614 =item Notes: CLIVE_CACHE
 615
 616 clive defaults to use $HOME/.cache/clive/ for "last" and
 617 "cache" files described above.
 618
 619 The use of the default path can be overridden by
 620 CLIVE_CACHE environment variable. Note that clive
 621 will attempt to create the specified path recursively.
 622
 623 Examples:
 624   setenv CLIVE_CACHE /home/user/cachedata (in csh terms)
 625   clive # will read/write /home/user/cachedata/(last|cache)
 626
 627   unsetenv CLIVE_CACHE
 628   clive # read/write $HOME/clive/(last|cache)
 629
 630   clive --last-file=mylast --cache-file=cachedata/mycache
 631     # read/write "mylast" file, read/write cachedata/mycache file
 632
 633 =back
 634
 635 =head1 CACHE
 636
 637 The purpose of the cache is to allow clive to skip fetching
 638 and parsing the video page again. It does not contain any
 639 actual video data so one should not expect to recover a deleted
 640 video file from the cache. Only some of the parsed details
 641 are stored as records to the cache.
 642
 643 By now, it is should be a well known fact that the cache fails
 644 with some of the supported hosts. For example Youtube video links
 645 expire after some time, this causes the re-extraction to fail if
 646 the cached video link is used later again.
 647
 648 This was the main reason why in 2.2.0 reading from cache was
 649 disabled by default. Many users reported the reuse of expired
 650 video links as a bug previously even though it was well documented
 651 in the manual page explaining that most of the HTTP 403/404 errors
 652 were actually caused by expired video links.
 653
 654 It is, of course, still possible to read from cache. You can
 655 enable this by invoking the --cache-read option. This causes
 656 clive to look up a saved cache record and reuse the stored
 657 video details if they are found instead of fetching the video
 658 page.
 659
 660 The use of the cache can be disabled with the --no-cache option.
 661 This disables both read and write. Note that if the BerkeleyDB Perl
 662 module is not installed, clive will not use the cache.
 663
 664 See also the --cache-grep option.
 665
 666 =head1 UNICODE
 667
 668 Q: Why am I seeing mangled video filenames?
 669
 670 A: Make sure you have set appropriate locale. For example (in csh/urxvt terms):
 671   % setenv LANG en_US.UTF-8
 672   % urxvt &
 673
 674 You can get a list of supported locales on your typical Unix-like system with:
 675   % locale -a
 676
 677 =head1 DEBUGGING
 678
 679 Some tips that we have found useful:
 680
 681   % clive --debug URL
 682
 683 Causes B<libcurl> to be verbose.
 684
 685   % clive -n URL
 686
 687 Simulates video extraction only.
 688
 689 =head1 BUGS
 690
 691 Sure to be some.
 692
 693 Please report them:
 694   <http://code.google.com/p/clive/issues/>
 695
 696 =head1 EXIT STATUS
 697
 698 clive exits 0 on success, and E<gt>0 if an error occurs.
 699
 700  CLIVE_OK          = 0
 701  CLIVE_NOTHINGTODO = 1    # file already retrieved
 702  CLIVE_NOSUPPORT   = 2    # host not supported
 703  CLIVE_READ        = 3    # file open/read error
 704  CLIVE_GREP        = 4    # grep: nothing matched in cache
 705  CLIVE_OPTARG      = 5    # invalid option argument
 706  CLIVE_SYSTEM      = 6    # system call failed (e.g. fork)
 707  CLIVE_REGEXP      = 7    # regexp pattern matching failed
 708  CLIVE_FORMAT      = 8    # requested format unavailable
 709  CLIVE_NET         = 9    # network error
 710  CLIVE_STOP        = 10   # --stop-after
 711
 712 =head1 OTHER
 713
 714 Project page:
 715   <http://clive.googlecode.com/>
 716
 717 Front-end (GUI):
 718   <http://abby.googlecode.com/>
 719
 720 Development code:
 721   % git clone git://repo.or.cz/clive.git
 722
 723 =head1 HISTORY
 724
 725   * Originally written in Python
 726   * Rewritten in Perl for 2.0.0
 727
 728 =head1 SEE ALSO
 729
 730 C<cclive(1)>
 731
 732 =head1 AUTHOR
 733
 734 Toni Gundogdu <legatvs@gmail.com>
 735
 736 Thanks to all those who have contributed to the project
 737 by sending patches, reporting bugs and writing feedback.
 738 You know who you are.
 739
 740 =cut