bin/rsync-over-recentfile-3.pl

   1 #!/usr/bin/perl
   2
   3 =pod
   4
   5 Q1: sorted by epoch? Who has to sort when how often?
   6
   7 A11 (wrong): nobody! The mechanism is "event" based and the array is only
   8 running push and shift (and grep). It is a journal (that throws memory
   9 away based on an interval) and is itself NOT rsynced but in fact (?)
  10 rewritten by every slave.
  11
  12 A12: usually nobody because the normal flow of things just sort
  13 everything in the right slots. But in case somebody breaks the rules,
  14 she has to sort accordingly and set a dirtymark flag.
  15
  16 Q2: or we do not ever promise that the timestamps are sorted or mirror
  17 the sequence of events or we make them floating point numbers so they
  18 become uniq and can be treated as hash keys [this I like!]). Please
  19 keep in mind that we must be able to help customers who have
  20 6000000000 files.
  21
  22 A21: yes, we promise the timestamps are sorted. They are floats and
  23 keeping them sorted and uniq should be doable without too complicated
  24 algorithms. But doing all this with a database instead of in-memory is
  25 currently not our plan.
  26
  27 Q3: Interesting/Funny is the idea that dependents fetch files from
  28 each other.
  29
  30 A31: yes
  31
  32 Q4: So the loop below says it wants to last at least 20 seconds but it
  33 may take longer when there are many files to transfer. What needs to
  34 be changed to make sure we refresh our copy of the recentfile when
  35 some time has gone by?
  36
  37 A41: The rf object needs to return after every chunk, it must know when
  38 it has covered the recentfile, if must know which intervals it has
  39 covered and it must be able to merge intervals based on the
  40 recent_events array. And the caller must get some info about these
  41 states.
  42
  43 Q5: the refetching of the recentfile can either be done within the
  44 mirror command or by the caller. There seems to be a difference
  45 between the principal and the others. Only when the principal changes,
  46 others can also change, so others should not waste time with
  47 refreshing.
  48
  49 =cut
  50
  51 use strict;
  52 use warnings;
  53 use File::Basename qw(dirname);
  54 use File::Spec;
  55 use Getopt::Long qw(GetOptions);
  56 use List::Util qw(min);
  57 use Time::HiRes qw(sleep);
  58
  59 use lib "/home/k/sources/rersyncrecent/lib/";
  60 require File::Rsync::Mirror::Recent;
  61
  62 our %Opt;
  63 GetOptions(\%Opt,
  64            "verbose!",
  65           ) or die;
  66
  67 my %reached;
  68 ITERATION: while () {
  69   my $iteration_start = time;
  70
  71   for my $tuple ([authors => "6h"],[modules => "1h"]) {
  72     my($rmodule,$interval) = @$tuple;
  73     my $rf = File::Rsync::Mirror::Recentfile->new
  74         (
  75          canonize => "naive_path_normalize",
  76          filenameroot => "RECENT",
  77          ignore_link_stat_errors => 1,
  78          interval => $interval,
  79          localroot => "/home/ftp/pub/PAUSE/$rmodule",
  80          remote_dir => "",
  81          remote_host => "pause.perl.org",
  82          remote_module => $rmodule,
  83          rsync_options => {
  84                            # intenionally not using archive=>1 because it contains "r"
  85                            compress => 1,
  86                            'rsync-path' => '/usr/bin/rsync',
  87                            links => 1,
  88                            times => 1,
  89                            'omit-dir-times' => 1,
  90                            checksum => 1,
  91                           },
  92          verbose => $Opt{verbose},
  93         );
  94
  95     $rf->mirror(after => $reached{$rmodule}||0);
  96     my $re = $rf->recent_events;
  97     $reached{$rmodule} = $re->[0]{epoch};
  98   }
  99   $reached{now} = time;
 100   for my $k (keys %reached) {
 101     next if $k =~ /T/;
 102     $reached{$k . "T"} = scalar localtime $reached{$k};
 103   }
 104   require YAML::Syck;
 105   print STDERR "Line " . __LINE__
 106       . ", File: " . __FILE__
 107           . "\n"
 108               . YAML::Syck::Dump(\%reached);
 109
 110   my $minimum_time_per_loop = 20;
 111   my $sleep = $iteration_start + $minimum_time_per_loop - time;
 112   if ($sleep > 0.01) {
 113     sleep $sleep;
 114   } else {
 115     # negative time not invented yet
 116   }
 117 }
 118
 119 print "\n";
 120
 121 __END__
 122
 123 # Local Variables:
 124 # mode: cperl
 125 # cperl-indent-level: 2
 126 # End: