5 * Fix: dailymotion id parsing
8 # fmt34 is now treated as yet another (new?) format
9 # Rewrite youtube section of the FORMATS in the manual
10 * Fix: --format=best not working with youtube (closes issue #39)
11 # Thanks to Peter Baranyi for the fix
13 Developer-visible changes:
14 * Disable: t/11redtube.t (broken, see known issues)
15 * "make test": ADULT_OK env.var. now enables redtube test
18 * redtube: broken (verify -> http/404)
25 * Fix: liveleak: "error: Unsupported protocol (http/1)" (closes #36)
26 * Fix: vimeo: title parsing (Thanks to Peter Baranyi for the fix, closes #37)
27 - Fixes title issues with accented chars
28 * Fix: break: http/403 (Thanks to Werner Elsler for the fix, closes #38)
30 Developer-visible changes:
31 * t/11redtube.t: fix dead link
38 * Add: --print-fname option
39 * Fix: dailymotion ID parsing
40 * Fix: --hosts: dailymotion: spak-mini -> spark-mini
43 Developer-visible changes:
44 * Correct copyright years and notices in source code
45 * tlinks/youtube-utf8.url: fix dead link
53 # --stream-pass option (port from cclive)
59 * Thanks to Patrick Hoffmann <patrick@sobran.de> for data mining
61 # Misc. typos in manual
62 # Redtube support (videoid/title parsing issues)
63 # Dailymotion: spak-mini -> spark-mini
65 # replace dashes ("-") with underscores ("_") in video ids
67 Developer-visible changes:
79 # t/02youtube.t: test link (dead link)
80 # tlinks/ytube.url: ditto
81 # tlinks/multi.url: youtube: ditto
91 1. Support spiegel.de (closes issue #32) [thanks to anon for data]
92 2. Support golem.de (closes issue #33) [thanks to anon for data]
93 3. ~/.clive/config path support
94 4. --cookie-jar option
95 5. Support for ehrensenf.de (closes issue #34) [thanks to bjoern for data]
96 * Improve: unicode support (related to issue #29)
97 1. cctv: no longer requires -C and -R options (obsoletes -R)
98 2. --cclass: now works as expected with unicode as well
99 * Improve: backwards compatibility with clive 2.0, 2.1
100 1. Allow --format=(mp4|hd|hq|3gp) (closes Debian bug #535957)
101 2. Restore ~/.config/clive/config path support (closes Debian bug #535483)
102 3. Add --upgrade-config (2.0/2.1 config to 2.2+ format)
104 1. --hosts prints now supported formats
105 2. Pair --exec with --exec-run, the latter now triggers the former
106 2.1. Closes Debian bug #520520
108 3.1. ~/.clivelast -> ~/.cache/clive/last
109 3.2. ~/.clivecache -> ~/.cache/clive/cache
110 4. If HOME environment variable undefined, assume current workdir
111 5. --version: print locale etc.
113 1. --version: copyright years
114 2. File suffix for CCTV (.x-shockwave-flash -> .flv)
115 3. Remove "see examples" from pod markup (closes issue #30)
116 4. Prevent ".$suffix" output filenames
117 4.1. Use $id if !$title && %i was not defined in --filename-format
118 4.2. Related to issue #29
119 5. --no-proxy: really disable all proxy use (closes issue #25)
120 5.1. http_proxy setting was still used
121 6. Can't call method "filename" on unblessed reference at Exec.pm line 5x
122 6.1. e.g. clive URL --exec="ls;"
123 6.2. Closes Debian bug #535459
124 7. Dailymotion support (closes issue #35)
125 8. Redtube support (error: no match: `(?-xism:videoid=(.*?)')')
127 1. --recall, --recall-file -> --last, --last-file
128 2. CLIVE_HOME -> CLIVE_CACHE
130 1. -R, --raw (obsoleted by improved unicode support)
132 2.1. Missleading name
133 2.2. Obsoleted by --cache-file and --last-file
135 1. Add note about numeric suffix (closes issue #28)
136 2. Fix a number of typos, use better grammar
139 Developer-visible changes:
142 * clive/Host/Spiegel.pm
143 * clive/Host/Golem.pm
144 * clive/Host/Ehrensenf.pm
150 * tlinks/ehrensenf.url
151 * New prerequisite: Config::Tiny
152 1. Required by --upgrade-config
154 1. Extract VERSION from lib/clive/Config.pm
155 2. Add "use warnings/strct"
162 User-visible changes:
163 * Change: Google: --format=mp4/best
164 1. Default to flv if mp4 is not available
165 2. Fix --format=best support
167 Developer-visible changes:
168 * Add: t/$host.t: NO_INTERNET variable support (closes issue #24)
169 * Change: t/02youtube.t test link
171 * Remove: t/06lastfm.t
172 1. t/02youtube.t: includes the same test
178 User-visible changes:
179 * Add: support for Vimeo (closes issue #19)
180 * Add: return codes for each error case (closes issue #22)
181 * Fix: return code is always 0 (closes issue #20)
182 * Change: --hosts output now matches cclive output
184 Developer-visible changes:
185 * Change: Cctv, Dailymotion, Liveleak -> use clive::Utils::matchRegExps
195 This release is a major overhaul aimed to fix the previous design flaws
196 and clean up the codebase. Note that 2.2.0 breaks compatibility with
197 the earlier versions of clive.
199 Most users will not notice much differences after upgrading to 2.2.0
200 but those users who have used clive for anything more than "clive URL"
201 should read the changes carefully. The summary of changes section
202 includes more detailed changes that is recommended reading for all
203 users and maintainers.
205 2.2.0 changes the license from ISC to GPLv3. The license was last
206 changed in 2.1.0 to ISC but has now been reverted back to GPLv3 after
207 some further consideration.
209 Config::Tiny has been replaced with Getopt::ArgvFile. The latter had
210 some advantages over Config::Tiny that lead to the switch. For example,
211 instead of trying to memorize the (often confusing) config variable
212 names, users can now use command line options in the config file.
214 This also means that everytime a new feature is added to the program,
215 we are no longer required to modify the code responsible for parsing
216 the config file. Using Getopt::ArgvFile also required adding only one
217 line of code to the project whereas Config::Tiny required several.
219 cache no longer reads by default. This means that --cache-read option
220 must be invoked for clive to read previously stored video records
221 from the cache. The change was made after seeing how most users found
222 the reading from cache too confusing and frequently reported expired
223 link errors such as HTTP 403 as bugs even though the behaviour was
224 documented in the manual page. DAERTM?
226 --emit-csv now outputs very few details about the videos. Only those
227 fields that are known to be any use are printed out.
229 2.2.0 also removes a number of less used features that have lingered
230 in the project since the 1.x. Most of these features were never
231 requested and have only been burdening the program codebase since
234 Part time hackers and/or developers will notice that clive now follows
235 a new OO design -- as well as Perl5 allows it. While there are still
236 some considerations regarding the design and limitations that could not
237 quite be ported from C++ used in cclive, WYSIWYG.
239 Package/port maintainers will want to read the README file. The previously
240 used GNU Makefile is no longer used and clive installation now depends on
241 ExtUtils::MakeMaker instead.
246 User-visible changes:
250 2. Last changed in 2.1.0 (-> ISC)
252 * Slight improvements to program startup time
254 * Config file format changes (Config::Tiny -> Getopt::ArgvFile)
255 1. Command line options can now be used in config file
256 2. Breaks compatibility with previous versions
257 3. Cleaner and new format allows using cmdline options in configs
258 4. Format was last changed in 2.0.0
260 -----------------------------------------------
261 # Config::Tiny: clive 2.0 - 2.1
262 cat >> ~/.config/clive/config
264 proxy = "http://foo:1234"
266 savedir = "/home/user/videos"
268 # Getopt::ArgvFile: clive 2.2
270 --proxy="http://foo:1234"
271 --savedir="/home/user/videos"
272 -----------------------------------------------
274 * Fix: Redtube video title parsing
277 1. Cache is now passive (read:no, write:yes)
278 2. Add: --cache- option prefix
279 3. New option: --cache-read, --no-cache
280 5. Rename: misc. options (e.g. --show -> --cache-dump)
281 6. Record field changes
282 o Breaks compatibility with previous versions
283 7. New field delimiter '#'
287 1. ~/.config/clive/config -> ~/.cliverc
288 2. ~/.config/clive/recall -> ~/.clivelast
289 3. ~/.config/clive/cache -> ~/.clivecache
292 1. Add: --format=best support (closes issue #15 )
293 2. Rename: mp4 -> fmt18 (Youtube)
294 3. Rename: Dailymotion ID spark -> flv
297 1. cURL error handling
298 2. Google mp4 support
299 3. Do not strip link params (closes debian bug #530659)
300 4. --format=fmt6 (no longer supported by Youtube?)
301 5. Dailymotion ID parsing
309 6. --stop-after (closes issue #18)
311 * --emit-csv: changes to CSV fields (cleanup, print only the necessary fields)
313 page_link, video_link, filename, file_length_mb,
314 file_length_bytes, video_id, time_stamp, page_title,
315 initial_length, remaining_bytes
317 base_filename, file_length, video_link
321 2. --renew (now obsolete)
322 3. --youtube-user (broken since 2.1)
323 4. --youtube-pass (...)
335 1. Long options: aliases (e.g. --output_file | --output-file | --outputfile)
336 2. --filename-format: new specifiers, rename some of the old ones
337 3. --format: exit with an error if id is not recognized by clive
338 4. Rename: -r -> -l (--recall)
339 5. Many short options have been removed (or reused)
341 Developer-visible changes:
345 * Change: use a custom perltidyrc (closes issue #17)
346 - Thanks to Tong Sun <suntong001@gmail.com>
348 * Prerequisite changes
349 1. Written for Perl 5.10.0
350 o 5.8 probably OK although unverified
354 3. Removed prerequisites
359 4. Optional modules (no changes from 2.1.14)
363 * Change: use decode_utf8 returned html only if html is valid utf8
365 * Installation changes
366 1. Makefile.PL (ExtUtils::MakeMaker) [see README]
369 1. Makefile (obsoleted by Makefile.PL)
370 2. clive.pod (merged into bin/clive)
371 3. clive.1 (generated from bin/clive)
373 * Remove: binmode STDOUT, ":utf8"
375 * Remove: extra decode_utf8
381 User-visible changes:
382 * Add: support for youtube-nocookie.com (closes issue #12)
383 * Add: fmt35 format ID for Youtube [closes issue #10 (and #13)]
384 * Fix: liveleak id parsing
385 * Rename: --format IDs for Youtube
389 * Rename: --output-video -> --output-file
390 * Rename: output:file -> output:filename_format (config file)
391 * Remove: --overwrite option (use --output-file instead)
393 Developer-visible changes:
394 * Add: tests/ytube-nocookies.url
398 * 2.2.0 is under development, read more here:
399 http://code.google.com/p/clive/wiki/WhatsNew_2_2_0
405 Developer-visible changes:
406 * Change: use perltidy defaults for style
407 * Change: BerkeleyDB and File::Path modules are now optional
408 ** Cache features depend on these modules
409 * Moved opted_mods initilization to init_opted_mods (-> loop)
415 User-visible changes:
416 * Change: --connect-timeout, --connect-timeout-socks default -> 30s
417 * Fix: timeout after file transfer starts
418 * Fix: title parsing with custom character class (closes issue #8)
419 * Add: redtube support; thanks to anonprn2@gmail.com (closes issue #11)
420 * Fix: typo in --emit-xml --help string
426 User-visible changes:
427 * Change: --youtube-pass: prompt for password if left undefined
428 * Change: --filename-format: default -> "%t.%s"
429 * Fix: "Use of uninitialized value $opts{"ytpass"} ..."
432 ** --connect-timeout-socks
433 * Cleanup: manual page
435 Developer-visible changes:
436 * Rename: auth_youtube subroutine -> loginto_youtube
439 * Youtube login does not work (throw an error, patches welcome!)
445 NOTE: This release changes the clive status messages. This will likely
446 break compatibility with utilities that depend on these messages.
447 The changes were made to enable using abby (0.2.0+) as a front-end
450 User-visible changes:
451 * Add abby (front-end) compatibility
452 - Change output similar to cclive
453 - http://code.google.com/p/abby/
454 * Add Youtube HD support (--format=mp4_hd)
455 - Thanks to bf for the patch (via FreeBSD ports)
456 * Add tv.cctv.com support [issue #6]
457 - Thanks to suntong001 for data mining
459 - --limit-rate (config: [http]:limit_rate)
460 - --hosts (print supported hosts to stdout)
461 - --stderr (redirect all output to stderr instead of stdout)
462 * Improve file suffix parsing from content-type
463 - Rewrite the content-type handling block in query_video_length
464 - Use whatever is found after slash in the content-type string
465 - Use "flv" for exceptions (e.g. octet-*, x-flv, etc.)
466 * Replace dashes in video IDs with underscores before using them in fnames
467 * Change --version output
468 * Fix reading from stdin (closes Debian #522422)
469 - Assume reading from stdin only if queue and argv arrays are empty
470 * Fix exit with status code
471 - Exit 0 on success, and >0 if an error occurs
473 Developer-visible changes:
474 * Update examples/config with limit_rate
475 * Rename: configrc -> examples/config
476 * Replace the monstrous if-elsif-$host code block with a loop
482 User-visible changes:
483 * Fix GoogleVideo "id not found" handling
484 - Symptom: exit after "process page..." message
485 - Cause: no check for undefined id value
486 * Fix GoogleVideo support
487 - Symptom: see above fix
488 - Cause: HTML changes to the video pages
489 * Fix URI::Escape import for GoogleVideo redirects
490 - Symptom: crash with "Undefined subroutine &URI::Escape::uri_unescape"
491 - Cause: recent script start optimizations; a missing 'require'
493 - Symptom: "expected different content-type"
494 - Cause: website content-type changes
496 Developer-visible changes:
497 * Cleanup code (style-wise)
498 - perltidy -gnu -l=72
504 User-visible changes:
505 * Keep URL queue order
506 - Remove duplicate URL checking
507 * Remove --modver, change --version output
508 * Update manual page (clive.1, clive.pod)
509 - Rewrite EXAMPLES section
510 - Cleanup other sections
511 * Config file: add support for [output]:format (--format)
512 * Port dailymotion support from cclive
514 Developer-visible changes:
516 - Replace Perl module checking with for-loops (incl. optional)
517 * Optimize script start time
518 - Replace "use $module" with "require $module" where possible
519 * Update LICENSE, configrc
521 - Add [output]:format to configrc
522 * Add tests/dmotion.url
528 User-visible changes:
529 * Add --output-video option
530 * Add option groups to --help output
531 * Add --stream-exec and --stream options
533 * Config file: add support for
534 - [commands]:exec (--exec)
535 - [commands]:stream (--stream-exec)
536 - See configrc for an example
537 * Fix --exec; should no longer spew out "not found" shell errors
538 * Change default filename format to "%t_%d_%i.%s"
539 * Change to use "file is already fully retrieved" error message
540 * Port resizing progressbar from cclive
542 Developer-visible changes:
543 * Add tests/lastfm.url and tests/multi.lst
544 * Remove XML::Simple dependency
546 * Implement SIGWINCH handling
547 - Resize progressbar to match terminal width
548 - Progressbar width defaults to 80 (chars)
549 - OS must support SIGWINCH
550 - Requires Term::ReadKey module
551 - clive works without the module but will ignore any SIGWINCH events
552 * Touch cclive.1, cclive.pod, INSTALL and Makefile files
556 * Buried clive-list, grew weary of fighting spam with bare fists
557 - email the author directly or use the issue tracker, thanks
563 User-visible changes:
564 * Renamed the following options
565 - --noproxy => --no-proxy
566 - --noextract => --no-extract
567 - --nologin => --no-login
568 * Butchered --help output
569 - much less verbose now
570 - relocated manual page data to clive.pod
571 * Removed --manual option
572 * Obsoleted "play" and "rencode" config keys in "commands" section
573 * Added --exec option which resembles cclive's --exec command
574 * Removed --play, --noplay, --rencode and --norencode options
576 * Changed --paste short option from '-x' to '-p'
577 * Removed wmv format (breakcom)
578 - persistant HTTP/404 (not found) error
580 Developer-visible changes:
582 - additional INSTALL variables
584 - can be used to disable module checks
585 - removed "man" from "all"
586 - removed "clean" (useless)
592 User-visible changes:
593 * Fixed sevenload support
599 User-visible changes:
600 * Fixed Youtube login which sometimes reported failed attempts as OK
601 - Youtube defaults web interface language based on users ip
602 - clive looks for English words in the post-login page to confirm login
603 * Tweaked --version output
605 Developer-visible changes:
606 * Removed metacafe remnants
607 * Fixed bug in time2str subroutine which caused incorrect calculations
609 * removed genman script (obsoleted by "make man")
610 * Added a rudimentary (GNU) Makefile
611 * Cleaned up translate_embed subroutine
614 * Removed the clive-announce and clive-users mailing lists
615 - New release announcements will be posted via Freshmeat alone from now on
616 - Subscribe to the project at http://freshmeat.net/projects/clive
617 * Created the clive-list@googlegroups.com mailing-list
618 - For support questions, patches and bug reports
619 - Issue tracker will remain for tracking bugs
620 - Send an email to clive-list-subscribe@googlegroups.com to subscribe
626 User-visible changes:
627 * Documented missing "%t" (--filename-format) in manual page
628 * Tweaked "localfile_length == remotefile_length" message
629 * Renamed CLIVE_CONFIGDIR environment variable to CLIVE_HOME
631 Developer-visible changes:
632 * Cleaned up manual page
634 * Cleaned up FreeBSD details (INSTALL)
640 User-visible changes:
641 * Removed "See --manual" from --version output
642 * Added support for evisortv
643 - http://code.google.com/p/clive/issues/detail?id=4
644 * Fixed: reused incorrectly last http error code for remaining urls in queue
645 * Fixed: progressbar reported 100% even if error had occurred (e.g. http/403)
647 Developer-visible changes:
648 * Cleaned up subroutines
654 User-visible changes:
655 * Fixed "Requested range was not delivered by the server (http/33)" w/ -c
656 - this would occur with a batch of URLs while using the --continue option,
657 e.g. if the first file was continued, the following page fetches exited
659 * Added --savebatch option
660 * Added --overwrite option
661 * Replaced the buggy progressbar with a "lazy man's progressbar"
662 * Fixed "print() on closed filehandle STDOUT" with --grep+delete
663 * Fixed progressbar display with --continue
665 Developer-visible changes:
667 * Started to bundle src tarball with clive.1 manual page
668 * Added genman script which wraps pod2man with common options
669 * Fixed a potential div by zero bug in calc_progress
675 NOTE: Changed license GPL3 => ISC/OpenBSD.
677 User-visible changes:
678 * Added --modversion option
679 * Replaced Term::Progressbar with a built-in thermometer progressbar
680 * Removed --progress=bar:nominor support due to above changes
681 * Added liveleakcom support
682 - http://code.google.com/p/clive/issues/detail?id=3
684 Developer-visible changes:
685 * Made IO::Pager module optional
687 * Cleaned up --progress option description
688 * Changed to require WWW::Curl 4.05+
689 * Removed Term::Progress module dependency
690 * Fixed a potential bug in calc_progress (improper $rate init)
692 * Made Expect module optional
698 User-visible changes:
699 * Fixed gvideo/mp4 support (http/404 even if the extraction link existed)
700 * Changed "found redirect" message
701 * Changed --play, --rencode status messages to use file basename
703 Developer-visible changes:
704 * decode_utf8($page): removes extra steps, works better with HTML::TokeParser
705 * Added missing Expect module version to INSTALL
712 This aims to be the last beta before the release of 2.0 in December 2008.
713 Please report any remaining bugs to the issue tracker at:
714 http://code.google.com/p/clive/issues/list
716 User-visible changes:
717 * Removed metacafe support until fixed (see known issues)
718 * Changed Youtube login to report if there were "too many login failures"
719 * Fixed Youtube login support: reported failures incorrectly as OK
720 * Added support clivepass(1) utility (http://code.google.com/p/clive-utils)
721 * Added support for metacafe and sevenload embed URL translation
723 Developer-visible changes:
725 * New prerequisite module: Expect
726 * Updated FreeBSD + Ubuntu details (INSTALL)
727 * Cleaned up some code
728 * Rewrote embed URL translation
731 * Metacafe support broken (http/404)
737 User-visible changes:
738 * Fixed pasting from clipboard containing multiple URLs
739 * Fixed page fetch progress glitch that occured with 1+ batches
740 * Added --agent option
741 * Added --proxy option
742 * Added --savedir option
743 * Added --cclass option
744 * Added --filename-format option
745 * Added --show-format option
746 * Added --youtube-user and --youtube-pass options
747 * Added --emit-xml option
748 * Added --progress=[none|bar|dot]
749 * Replaced --noprogress with --progress=none
750 * Added avg. transfer rate notification
751 * Fixed filename printing for existing output files (e.g. output.flv.1)
752 * Added --play option
753 * Fixed metacafe support (error: failed to extract videoCDNURL)
754 * Added --rencode option
756 Developer-visible changes:
757 * Ported --emit-csv option from 1.x with minor changes
758 - Backwards compatible with 1.x
759 - Prints additional video details
760 - $length, $video_id, $time_stamp, $page_title, $cont_from, $remaining
761 * Changed to prefer 'dot' bar if the output is not a TTY
762 * Replaced progress bar with Term::ProgressBar (new module prerequisite)
763 * Cleaned up queue processing subroutine
769 User-visible changes:
770 * clive now ignores input lines that start with the '#' character
771 * Added support for CLIVE_CONFIGDIR environment variable
772 * Ported lastfm video support from 1.x
773 * --version now displays XML::Simple
774 * Changed "= Play:" to "=> Playing ..."
775 * Playing occurs now subsequently after going over the URL batch
777 Developer-visible changes:
778 * Removed a google video redirect (commented out, see code)
779 * Added the AUTHORS file
780 * XML::Simple was missing as a prerequisite in INSTALL
781 * Made minor syntactic tweaks to the source code
782 * Fixed missing --quiet check with the "=> Play" message
788 A complete overhaul and rewrite of 1.x.
790 User-visible changes:
791 * Users are expected to use the --continue/-c option to resume transfers
792 * Configuration file format (=> INI), path was changed (=> ~/.config/clive)
793 * All extractions now assume flv as the default download format
794 * Users are expected to understand when/how use the --format/-f option
795 * Reading input from multiple sources is possible (e.g. % clive -x URL URL)
796 * Added: --grep: used to grep and recall cached URL entries
797 * Added: --background and --output=logfile, --append=logfile options
798 * Many old cmdline and config options were removed due to new design
799 * Improved GoogleVideo support: handles redirects to other hosts
800 * Pager is used where needed (e.g. --show/-s)
801 * Changed default output file naming format to "%n-(%i)-[%d].%s"
802 * Removed: guba (deadweight), myvideo (borked), dmotion (bitches @ anon users)
804 Developer-visible changes:
806 - See the INSTALL file for prerequisites
807 * No installation: all-in-one script (excl. the 3rd party modules)
808 * Cache is now an integral part of clive, see CACHE in the manual page
809 * Embedded the --manual page (Pod)
810 * Improved Youtube login: removed is_adult HTTP get (done manually now)
811 * Grab the file suffix from the Content-Type header
812 * Improved GoogleVideo support: mp4 detection no more relies on English words
813 * Improved Metacafe support: removed unnecessary "familyfilter" HTTP get
817 - 2.x+ at http://googlecode.com/p/clive/
818 - 1.x remains at http://gna.org/projects/clive
820 - 2.x+ at http://googlecode.com/p/clive/
821 - 1.x remain at http://dl.gna.org/clive
823 - http://clive.sf.net/
824 - http://home.gna.org/clive redirects to the above
826 - 2.x+ at git://repo.or.cz/clive.git
827 - 1.x remains at svn://svn.gna.org/svn/clive
828 * Issue tracking: Moved
829 - 2.x+ at http://googlecode.com/p/clive/issues/
830 - 1.x bug tracker remains at http://gna.org/projects/clive
831 * Created http://googlegroups.com/group/clive-announce/
833 NOTE: See http://dl.gna.org/clive/CHANGES for 1.x.