Fix: timeout after file transfer starts.

[clive.git] / clive.pod

=head1 NAME

clive - the non-interactive video extraction utility

=head1 SYNOPSIS

clive [option]... [URL]...

=head1 DESCRIPTION

clive is a command line utility for extracting videos from Youtube and other
video sharing Web sites. It was originally written to bypass the Adobe Flash
requirement needed to view the hosted videos.

clive can be non-interactive. This means it can work in the background
while the user is not logged on, allowing the user to start an extraction
and disconnect from the system and letting clive finish the work.

clive was influenced by the GNU L<wget(1)> utility. Kudos to the wget team
for their original work.

=head1 OPTION SYNTAX

You may freely mix different option styles, or specify options after the
command-line arguments. For example:
  % clive -c --format=mp4 URL -n

You may also put several options together that do not require arguments.
For example:
  % clive -pcn URL

Which is equivalent to:
  % clive -p -c -n URL

=head1 OPTIONS

=over 4

=item B<-h --help>

Print this help and exit.

=item B<-v --version>

Print version and exit.

=item B<--hosts>

Print supported hosts and exit.

=item B<-b --background>

Go to background immediately after startup. If no output file is specified
using the B<--output> or B<--append>, the output is redirected to I<clive-log>
file. Implies B<--progress=>I<dot>.

=item B<-e --emit-csv>

Do not extract any videos, instead print the video details as CSV to stdout.
Ignores the B<--quiet> option for CSV.

=item B<-E --emit-xml>

Equivalent to B<--emit-csv> in every way but the output is XML. Negates
B<--emit-csv>.

=item B<-V --clivepass=>I<path>

I<path> to the L<clivepass(1)> utility. If CLIVEPASS_PATH environment
variable is set, it will will used.

=back

B<HTTP options>

=over 4

=item B<-U --agent=>I<string>

Identify as I<string> to the HTTP server. Defaults is "Mozilla/5.0".

=item B<-y --proxy=>I<address>

Use I<address> for HTTP proxy, e.g. http://foo:1234. If undefined,
uses the http_proxy environment setting if present.

=item B<-X --no-proxy>

Do not use HTTP proxy.

=item B<--connect-timeout=>I<seconds>

Maximum time in I<seconds> allowed the connection to the server to take.
This limits only the connection phase. B<clive> will only timeout
on the system's internal timeouts by default.

http://curl.haxx.se/docs/knownbugs.html:

  "34. The SOCKS4 connection codes don't properly acknowledge
       (connect) timeouts. Also see #12. According to bug #1556528,
       even the SOCKS5 connect code does not do it right:
       http://curl.haxx.se/bug/view.cgi?id=1556528"

  "12. When connecting to a SOCKS proxy, the (connect) timeout
       is not properly acknowledged after the actual TCP connect
       (during the SOCKS 'negotiate' phase)."

See also B<--connect-timeout-socks>.

=item B<--connect-timeout-socks=>I<seconds>

Same as B<--connect-timeout> but works around the SOCKS proxy bug by using
CURLOPT_TIMEOUT instead of CURLOPT_CONNECTIMEOUT.

Ignored for file transfers, otherwise curl could interrupt started transfers.

More about these curl options:
  http://curl.haxx.se/libcurl/c/curl_easy_setopt.html

=back

B<Cache Options>

=over 4

=item B<-R --renew>

Renew cache entries for the re-visited URLs. See L</CACHE> in the manual page.

=item B<-s --show>

Print all cache entries to standard output.

=item B<-H --show-format=>I<string>

Format printed entries using the I<string>. Default is '%D: "%t" | %mMB'.
The following specifiers are supported:
  %t = video page title
  %i = video id
  %l = video file length (bytes)
  %u = video page url
  %x = video extraction url
  %D = video extraction date
  %T = video extraction time
  %S = video extraction timestamp (same as: %D %T)

=item B<-g --grep=>I<pattern>

Grep cache entries for I<pattern>. Matching is B<not> restricted to any
specific cache entry field (e.g. title or id).
  % clive --grep=git
  % clive --grep=^git --ignore-case

The matched entries are then extracted. To only show the matches, use the
B<--show> option. For example:
  % clive -sg ^git

=item B<-i --ignore-case>

Causes clive to ignore case-differences with B<--grep>.

=item B<-D --delete>

Delete grep-matched entries from the cache. If used with the B<--show>
option, clive will prompt before deleting, e.g.:
  % clive -siDg ^git

=item B<-C --clear>

Clear cache and exit.

=back

B<Logging and Input Options>

=over 4

=item B<-o --output=>I<logfile>

Log all messages to I<logfile>. The messages are normally reported to
stdout and stderr.

=item B<-a --append=>I<logfile>

Append to I<logfile>. This is the same as B<--output> but appends to
I<logfile> instead of overwriting it. If the I<logfile> does not exist,
it is created.

=item B<-d --debug>

Causes the program to print Curl debug messages.

=item B<-q --quiet>

Turn off all output.

=item B<-r --recall>

Recall the last URL batch from the previous runtime. Note that clive overwrites
this information at every runtime. See also L</CACHE>.

=item B<-T --savebatch=>I<file>

Save current URL batch to I<file>. 

=item B<-p --paste>

Paste input from clipboard. The pasted URLs are expected to be separated with
newlines.

=item B<--stderr>

Redirect all input to stderr, even if no TTY is available.

=back

B<Download Options>

=over 4

=item B<-O --output-video=>I<file>

Write video to I<file>.

=item B<-n --no-extract>

Do not actually extract any videos.

=item B<-c --continue>

Continue extraction of a partially downloaded file. Note that this works only
with HTTP servers that support the "Range" header. Ignored unless I<localfile>
E<lt> I<remotefile>.

The "requested range was not delivered" error typically implies that the
host does not allow continuing partially extracted video files. You will
see this error if you attempt to continue a partially downloaded flv video
from Youtube, for example.

=item B<-W --overwrite>

Causes clive to overwrite already existing video files. By default clive
appends a numeric suffix to the output filename.

=item B<-G --progress=>I<type>

Select the I<type> of the progress indicator. Accepted types are "bar",
"dot" and "none".

The "bar" indicator is used by default. It spews out some details indicating
the status of retrieval. If the output is not a TTY, the "dot" bar will be
used instead.

Use --progress=dot to switch to the "dot" display. It traces the retrieval
by printing dots on the screen, each dot representing a fixed amount of
downloaded data.

When using the dotted retrieval, you may also set the style by specifying
the I<type> as dot:style.
  --progress=dot:default
    each dot 1K, 10-dots clusters, 50-dots lines
  --progress=dot:binary
    each dot  8K, 16-dots clusters, 48-dots lines
  --progress=dot:mega
    each dot 64K,  8-dots clusters, 48-dots lines

Progress indication can be disabled completely by setting I<type> to "none".

=item B<-u --youtube-user=>I<username>

Causes B<clive> to log into Youtube as I<username>. This is done I<before>
extraction, even if the batch does not contain any Youtube video links.

=item B<-t --youtube-pass=>I<password>

Use I<password> for Youtube login.  If undefined, B<clive> prompts for it.

=item B<-L --no-login>

Do not log into Youtube.

=item B<-S --savedir=>I<dir>

Save extracted videos to I<dir>.

=item B<-f --format=>I<format>

Extract I<format> of the video. All supported websites have at least the flv
format available. See L</FORMATS> for more.

=item B<-l --cclass=>I<character-class>

Use I<character-class> to filter the video page titles. Default is B<\w>.
The filtered video page titles are used to name the extracted videos.

=item B<-N --filename-format=>I<string>

Use I<string> to construct the name of the extracted video. Default is
"%t.%s". The following specifiers are supported:
  %t = video title (after applying --cclass)
  %i = video id
  %d = video domain
  %s = video suffix
  %D = current date
  %T = current time
  %S = timestamp (same as %D %T)

=item B<-x --exec=>I<command>B<;>

Execute I<command>. Optional arguments may be passed to the command.
The expression must be terminated by a semicolon (";"). If the specifier
"%i" appears anywhere in I<command> name or the arguments it is replaced
by the pathname of the extracted video file.

Example:
  % clive --exec="mplayer %i;" URL

=item B<-x, --exec=>I<command>B<+>

Same as B<--exec>, except that "%i" is replaced with as many pathnames
as possible for the invocation of I<command>.

=item B<--stream-exec=>I<command>

Execute I<command> when file transfer progress reaches I<percent> as defined
with the B<--stream> option. If a "%i" specifier is used in the I<command>
expression, it will be replaced with video pathname.

B<clive> forks the I<command> and continues with the transfer. Note that
B<clive> will not continue to download another file before the child process
terminates.

Some players and formats may not work properly with this feature.
L<mplayer(1)> is known to work properly with most flv videos.

=item B<--stream=>I<percent>

Execute the I<command> defined with the B<--stream-exec> option when the
file transfer progress reaches I<percent>.

Example:
  % clive --stream-exec="mplayer -really-quiet %i" --stream=20 URL

Note that B<clive> does nothing to check if there is enough data buffered
before forking the stream process. If the transfer rate drops significantly
after starting the process and it runs out of data, B<clive> does nothing
to fix this.

=item B<--limit-rate=>I<amount>

Limit video download rate to I<amount> KB/s.

=back

=head1 EXAMPLES

=over 4

=item Typical use:

% clive "http://youtube.com/watch?v=3HD220e0bx4"

=item Multiple video URLs:

% cat E<gt>E<gt> url.lst
  http://en.sevenload.com/videos/IUL3gda-Funny-Football-Clips
  http://youtube.com/watch?v=3HD220e0bx4
  http://break.com/index/beach-tackle-whip-lash.html
  http://www.liveleak.com/view?i=704_1228511265

% cat url.lst | clive

=item Combine input from multiple sources:

% clive -p URL URL2

=item Recall the last URL batch, extract mp4 video format:

% clive -rf mp4

=item Grep for the pattern in cache, extract matched videos:

% clive -g 3HD220e0bx4

=item Same as above but I<delete> the matched entries from cache:

% clive -iDg ^3hd2

Note the use of -i (--ignore-case).

=item Print cache contents to stdout:

% clive -s

=item Combine matching and printing cache entries:

% clive -sig ^3hd2

Note that -s negates video extraction.

=item Match, redirect output to I<my.log>, go to background:

% clive -big ^3hd2 -o my.log

=item Extract video from URL, re-encode audio to mp3:

% clive --exec="ffmpeg -y -i %i %i.mpg;" URL

=item Login to Youtube to extract a video:

% clive -u myusername -t mypassword URL

This isn't typically needed unless you are trying to extract videos flagged
as mature content.

=item Use protected login password:

Otherwise same as above but the login info is saved to a clivepass password
repository file. The saved passwords are encrypted.

Note that you will need to install the clive-utils for this. See the L</OTHER>
section for details.

Create new password repository:
  % clivepass --create

Add new login name "myusername":
  % clivepass --add myusername

Use the created login info from clive:
  % clive -u myusername -t - -V /usr/bin/clivepass URL

This causes B<clive> to request for the login password for "myusername" from
L<clivepass(1)>. The password is then used to login to Youtube before extracting
the video from the URL.

=back

=head1 FORMATS

clive defaults to extract the flv format unless the B<--format> option is
used. The requested format may not always be available. The server typically
returns one of the HTTP/40x errors if the requested format is not available.

The quality of the video depends on the uploaded video quality. Each website
typically recompresses the uploaded videos to 320x240 resolution (sometimes
higher). Because this varies per video and website, you should treat the
list below only as a rough reference of what formats should be available.

=over 4

=item B<www.youtube.com>

=item B<www.last.fm>

Formats: (flv|mp4|mp4_hd|3gpp|xflv)  Continue: Yes (except flv)

mp4 and 3gpp formats are typically, or will soon become, available.
xflv appears to be very rare. Videos dating back to 2006 may only be
available as flv. The high-definition mp4_hd format is currently available
for a limited number of videos, which are usually explicitly marked as "HD"
or "720p".

Some of the videos displayed on last.fm site are actually Youtube videos.
You can pass the lastfm'ed video URLs to B<clive>.

=back

=over 4

=item B<video.google.com>

Formats: (flv|mp4)  Continue: Only mp4

The mp4 may not always be available. GoogleVideo mp4 are encoded using a
different codec than Youtube mp4 videos. You can typically B<--stream-exec>
Youtube mp4 videos but this is not case with GoogleVideo mp4 videos. If you
find a way, drop an email.

=back

=over 4

=item B<www.sevenload.com>

Formats: flv  Continue: Yes

=back

=over 4

=item B<www.break.com>

Formats: flv  Continue: Yes

=back

=over 4

=item B<www.liveleak.com>

Formats: flv  Continue: Yes

=item B<www.evisor.tv>

Formats: flv  Continue: Yes

=item B<www.dailymotion.com>

Formats: (spark|spak-mini|vp6-hq|vp6-hd|vp6|h264)  Continue: Yes

spark is the "regular" 320x240 flv. B<clive> defaults to spark
if requested format is not available.

=item B<tv.cctv.com>

Formats: flv  Continue: Yes

=back

=head1 CACHE

The cache has two purposes:

=over 4

=item 1.

Keep reusable video data available for fast re-extraction.

=item 2.

Keep a record of extracted videos.

=back

Each cache entry contains information about a video, including, but not limited
to, page title, file length and extraction URL. You can use the B<--grep> option
to match these details when you are searching for videos from the cache. See
the L</EXAMPLES> section for examples of how to do this.

Some entries may need to be renewed from time to time as certain websites have
their extraction URLs expire after awhile. Youtube is an example of this.
Youtube servers usually return the HTTP/410 error if the extraction URL has
expired. You can use the B<--renew> option to fix this.

Note that B<clive> renews the cache entry automatically if the video format
does not match the cached video format string. This is done for two reasons:

=over 4

=item 1.

Cached video extraction URL points to a different location

=item 2.

Cached video file length is likely to be incorrect

=back

=head1 FILES

clive searches the ~/.config/clive directory for the config file. You can
override this by setting the B<CLIVE_HOME> environment variable.

=over 4

=item ~/.config/clive/config

Configuration file for clive.

=item ~/.config/clive/cache

Contains the cache entries of the visited URLs. A Berkeley DB (Hash) file.

=item ~/.config/clive/recall

Contains the last URL batch. You can B<--recall> this file.

=back

=head1 EXIT STATUS

clive exits 0 on success, and E<gt>0 if an error occurs.

=head1 SEE ALSO

L<clivefeed(1)> L<clivescan(1)> L<clivepass(1)>

=head1 OTHER

Homepage:
  <http://clive.sourceforge.net/>

Project page:
  <http://code.google.com/p/clive/>

Development repository can be cloned with:
  % git clone git://repo.or.cz/clive.git

For release announcements, subscribe to the project at:
  <http://freshmeat.net/projects/clive/>

Additional utilities (clive-utils):
  <http://code.google.com/p/clive-utils/>

=head1 AUTHOR

Written by Toni Gundogdu <legatvs@gmail.com>.

=cut