bump version of Net-Async-Redis

[andk-cpan-tools.git] / CPAN-Blame / root / about.tt

[% WRAPPER page.tt title = "about cpan testers reports statistical analysis" %]
<h3><a href=".">&#x21d1;</a> About Mixed Results: Samples of Regression Analyses On CPAN Testers Reports</h3>

<table><tr><td></td><td>

<h3>What this is about</h3>

<p>The <i>perl programming language</i> has a huge repository of
reusable code known as the CPAN. CPAN has a strong tradition of
providing self-testing facilities with every software package it
contains. There are dedicated serverfarms running around the clock to
produce fresh test results of many different platforms in many
different environments. These get delivered to the <a
href="http://www.cpantesters.org/">cpantesters</a>.</p>

<p>On this page we are watching the results of all these tests and
calculate <a
href="http://en.wikipedia.org/wiki/Regression_analysis">statistical
regressions</a> to help the users of CPAN spot patterns where things
are starting to fail.</p>

</td></tr><tr><td></td><td>
<h3>Howto use</h3>

</td></tr><tr><td valign="top">
<p>
<table border="1" width="100%">
<tr>
<th>#</th>
<th>distro</th>
<th>links</th>
<th>pass</th>
<th>fail</th>
<th>uploaded</th>
<th>third fail</th>
<th>RT/comment</th>
<th>high correlations</th>

</tr>


<tr>
<td align="right">
...
</td>
<td>
...
</td>
<td valign="top">
...
</td>
<td align="center" valign="top">
...
</td>
<td valign="top">
...
</td>
<td valign="top">
...
</td>
<td valign="top">
...
</td>
<td valign="top">
...
</td>
<td valign="top">
...
</td>
</tr>

<tr>
<td align="right">
16
</td>
<td>
TONYC/&#x200b;Imager-&#x200b;0.71_02

</td>
<td valign="top">
matrix
</td>
<td align="right" valign="top">
64
</td>
<td align="right" valign="top">
7
</td>
<td valign="top">
2009-12-01 09:23
</td>
<td valign="top">
2009-12-02 18:06
</td>
<td valign="top">
 

</td>

<td valign="top">
qr:(Can't locate \S+pm)
</td>
</tr>

<tr>
<td align="right">
...
</td>
<td>
...
</td>
<td valign="top">
...
</td>
<td align="center" valign="top">
...
</td>
<td valign="top">
...
</td>
<td valign="top">
...
</td>
<td valign="top">
...
</td>
<td valign="top">
...
</td>
<td valign="top">
...
</td>
</tr>

</table> </p> </td><td> <p>The <a href=".">main page</a> consists of a
large table of CPAN distributions that had at least 3 PASSes
<i>and</i> at least 3 FAILs and has been uploaded within the last 16
years. This limitation is an entirely arbitrary compromise between
incoming tests, number of distros with mixed results and computing
power needed to calculate all the regressions. It may change any time
when some of these parameters change.</p>

<p>The list is sorted by upload date, descending. Of any distribution
only the latest upload can be included in the list. If Javascript is
enabled in your browser you can sort this table by clicking on the
table header in the column you want to sort.</p>

<p>
These are the columns on the main page:
<dl>

<dt><b>#</b></dt><dd>just a counter</dd>

<dt><b>distro</b></dt><dd>Path to the distro consisting of author's
id, maybe some path below their directory, and filename of the distro.
This is usually a link to a page with details about the calculated
results. If it isn't a link, then the statistical regression has not
yet been calculated but will be calculated in the next hours.</dd>

<dt><b>links</b></dt><dd>Links to the CPAN testers matrix (which
provides a comprehensive descriptive table of passes and fails by perl
version and OS)</dd>

<dt><b>pass</b> and <b>fail</b></dt><dd>The number of pass reports and fail
reports respectively. Other report types are ignored for the sake of this
investigation. Note that only distributions are listed that have at
least three passes and three fails.</dd>

<dt><b>uploaded</b></dt><dd>Upload date and time in UTC</dd>

<dt><b>third fail</b></dt><dd>Date and time in UTC when the third
tester reported a fail. Sorting by this column is one of the most
interesting views on the table because it gathers all recently
introduced problems on the top.</dd>

<dt><b>RT/comment</b></dt><dd>This column is maintained manually and
is for this reason often outdated: it contains a comment or a link to
an RT or other bugtracker ticket that has something to do with the
diagnosed problem.</dd>

<dt><b>high correlations</b></dt><dd>This is an interesting part: a
comma separated list of parameters whose values have a correlation to
the test result. The CPAN testers reporting tools provide a lot of
independent parameters that might influence the result. Whenever there
is a correlation between one of these parameters and the result, this
parameter is listed in this column. Of course, as always with
statistical findings, such correlations may be trivial artefacts (e.g.
<a
href="https://en.wikipedia.org/wiki/Heteroskedascity">Heteroskedascity</a>)
that provide little to no information, so be careful not to draw wrong
conclusions. <a href="#ioc">See below</a> for hints about which
insights have been gained so far.</dd> </ul> </p>

<p>The distro column contains links to separate pages per distribution
providing results of the regressions analyses. These are sorted by
R-squared in descending order. R-squared is a measurement for the
quality of the statistical correlation. It's always between
<i>zero</i> and <i>one</i>. The higher the R-squared the stronger the
impact of the referenced variable on the outcome.</p>

</td></tr><tr><td></td><td>
<a name="cor"><h3>Collection of Regressions</h3></a>

</td></tr><tr><td>
<p>
<h3>Imager-0.71_02 (matrix, searchtools, parsed reports, metacpan.org)</h3>

<p>
<table>
<tr><td><b>Uploaded:</b></td><td>2009-12-01T09:23</td></tr>
<tr><td><b>Calculated:</b></td><td>2009-12-04T06:32</td></tr>
<tr><td><b>Distro-ID:</b></td><td>TONYC/Imager-0.71_02.tar.gz</td></tr>
<tr><td><b>Pass/Fail:</b></td><td>64/7</td></tr>
<tr><td><b>Pass/Fail at calctime:</b></td><td>64/7</td></tr>
<tr><td><b>Top regressions:</b></td><td>1.000 <a href="#">qr:(Can't locate \S+pm)</a></td></tr>
</table>
</p>
 <table border="1" width="100%">
 <tr><th colspan="4">Regression#1 <a href="#cor">qr:(Can't locate \S+pm)</a></th></tr>
 <tr><td>Name</td><td align="right">Theta</td><td align="right">StdErr</td><td align="right">T-stat</td></tr>
  <tr>
  </tr>
  <tr>
   <td>
   <code>...</code>
   </td>
   <td align="right">
   <code>...</code>
   </td>
   <td align="right">
   <code>...</code>
   </td>
   <td align="right">
   <code>...</code>
   </td>
  </tr>
 </table>

 <table border="1" width="100%">
 <tr><th colspan="4">Regression#2 <a href="#cor">conf:bincompat5005</a></th></tr>
 <tr><td>Name</td><td align="right">Theta</td><td align="right">StdErr</td><td align="right">T-stat</td></tr>
  <tr>
  </tr>
  <tr>
   <td>
   <code>...</code>
   </td>
   <td align="right">
   <code>...</code>
   </td>
   <td align="right">
   <code>...</code>
   </td>
   <td align="right">
   <code>...</code>
   </td>
  </tr>
 </table>

 <table border="1" width="100%">
 <tr><th colspan="4">Regression#3 <a href="#cor">conf:ccdlflags</a></th></tr>
 <tr><td>Name</td><td align="right">Theta</td><td align="right">StdErr</td><td align="right">T-stat</td></tr>
  <tr>
  </tr>
  <tr>
   <td>
   <code>...</code>
   </td>
   <td align="right">
   <code>...</code>
   </td>
   <td align="right">
   <code>...</code>
   </td>
   <td align="right">
   <code>...</code>
   </td>
  </tr>
 </table>
</p>
</td>
<td valign="top">

<p>The individual CPAN distribution's page of results of the many
individual regressions starts with a small listing of metadata like
who the author and what the exact filename is and links to other sites
dealing with CPAN.
</p>

<p>Below that a rather long list of individual results of regression
calculations follows. These are sorted by their goodness of fit, i.e.
the first few results are likely to be the only interesting ones. But
since statistics needs to be interpreted to be useful we present them
all.</p>

<p>Every regression has a header denoting the name of the independent
variable. These names are documented in the
<code>CPAN::Testers::ParseReport</code> module and should in most
cases be self explaining. So the variable with the name
<i>conf:bincompat5005</i> denotes the config variable
<i>bincompat5005</i> of the perl interpreters that were running these
tests. And <i>qr:(Can't locate \S+pm)</i> denotes a string found in
the test output when matching it agaist the regular expression <i>(Can't
locate \S+pm)</i>. </p>

<p>When you click of any such header you are taken to the page
described below under <a href="#ltid">Links to input data</a>.

</td></tr><tr><td></td><td>
<a name="ioc"><h3>Interpretation of correlations</h3></a>
</td></tr><tr><td valign="top">

<p>
<table border="1" width="100%"> <tr><th colspan="4">Regression#1 <a href="#ioc">qr:(Can't locate
\S+pm)</a></th></tr>
<tr><td>Name</td><td align="right">Theta</td><td align="right">StdErr</td><td align="right">T-stat</td></tr>
<tr> </tr> <tr> <td> <code>[0='const']</code> </td>
 <td align="right"> <code>1.0000</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right"> <code>37707732907022898844.00</code> </td> </tr>
 <tr> <td> <code>[1='eq_Can't
locate Kwiki/Plugin.pm']</code> </td>
 <td style="background: #ffaaaa" align="right"> <code>-1.0000</code> </td> <td> <code>0.0000</code> </td>
 <td align="right"> <code>-32758385024668659982.00</code> </td> </tr>
 <tr> <td> <code>[2='eq_Can't
locate Spoon/Plugin.pm']</code> </td>
 <td style="background: #ffaaaa" align="right"> <code>-1.0000</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right"> <code>-7256856581466726773.00</code> </td> </tr>
 <tr><td colspan="4">R^2=
1.000, N= 107, K= 3</td></tr> </table>
</p>

</td><td>

<p>You can read more about statistical <a
href="https://en.wikipedia.org/wiki/Regression_analysis">regression
analysis</a> but you are allowed to bypass the theory and focus on two
things: the term <a
href="https://en.wikipedia.org/wiki/R-squared">R<sup>2</sup></a>
(pronounced R squared) is the overall measure of the <a
href="https://en.wikipedia.org/wiki/Goodness_of_fit">goodness of
fit</a> of the estimates. It is between 0 and 1 where 1 is a good fit.
The T-stat values measure each estimation if its slope differs
significantly from 0. It is between −infinity and +infinity.</p>

<p>In the detailed page of all regression tables for a particular
distribution you may find a table like the one on the left.</p>

<p>First a few notes about what you see here: The title of the table
denotes the name of the variable that is inspected by this regression
test, in this case <i>qr:(Can't locate \S+pm)</i>. At the bottom you
find the <b>R²</b> (R-squared; R^2) described above, the <b>N</b>
denotes the number of observations we had and the <b>K</b> denotes the
number of lines (actual values) we see in the middle part. The middle
part lists the influencing factors in this test: in the first line a
constant part <i>[0='const']</i>, below it the actually observed
strings. Each line has the <b>Theta</b> value that can be interpreted
as the direction and slope of the influence, the <b>StdErr</b> that
gives us a measurement of the variance, and the <b>T-stat</b> which
provides the significance of this theta (being different from
zero).</p>

<p><span style="background:#afa">We</span>
<span style="background:#bfb">have</span>
<span style="background:#cfc">chosen</span>
<span style="background:#dfd">to</span>
<span style="background:#faa">color-code</span>
<span style="background:#fbb">the</span>
<span style="background:#fcc">Theta</span>
<span style="background:#fee">column</span>:

negative values are reddish and positive values are greenish,
signifying that positive values indicate an influence towards a PASS
and negative ones to a FAIL. Both green and red are getting paler the
lower the R-squared is to indicate that interpreting values with low
R-squared should be avoided. A weak influence is only then a weak
influence when the goodness of fit confirms it. Otherwise you just
can't tell. Some reason makes this influence insignificant.</p>

<p>The example regression to the left is the most trivial case:
whenever the message <i>Can't locate Kwiki/Plugin.pm</i> has been seen
in the test output then the test result was a FAIL. Same thing for
<i>Spoon/Plugin.pm</i>. The author has most probably forgotten to
declare a dependency on the two modules <code>Kwiki::Plugin</code> and
<code>Spoon::Plugin</code>. Not necessarily, of course. It may be that
the author has forgotten the dependency on <code>Spoon::Plugin</code>
and that the dependency on <code>Kwiki::Plugin</code> is an indirect
dependency. Or it may be some completely remote connection between
some other relevant variable and this kind of observed error
message.</p>

<p>Usually the <i>Can't locate \S+\.pm</i> messages are the most reliable
indicators of the real reason of a fail. Another good candidate are
<i>mod:*</i> variables like the following:</p>


</td></tr><tr><td valign="top">
<p>
<table border="1" width="100%"> <tr><th colspan="4">Regression#2
<a href="#ioc">mod:Test::More</a></th></tr>
<tr><td>Name</td><td align="right">Theta</td><td align="right">StdErr</td><td align="right">T-stat</td></tr>
<tr> </tr>
 <tr> <td> <code>[0='const']</code> </td>
 <td align="right"> <code>1.0000</code> </td>
 <td align="right"> <code>0.0000</code>
</td>
 <td align="right"> <code>67664785050021653200.00</code> </td> </tr>
 <tr> <td> <code>[1='eq_0.72']</code> </td>
 <td style="background: #ffaaaa" align="right"> <code>-1.0000</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right">
<code>-14765666636887643371.00</code> </td> </tr>
 <tr> <td> <code>[2='eq_0.80']</code> </td>
 <td style="background: #ffaaaa" align="right"> <code>-1.0000</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right"> <code>-10567464028645793135.00</code>
</td> </tr>
 <tr> <td> <code>[3='eq_0.86']</code> </td>
 <td style="background: #ffaaaa" align="right"> <code>-1.0000</code> </td>
 <td align="right">
<code>0.0000</code> </td>
 <td align="right"> <code>-47228831801881750440.00</code> </td> </tr>
 <tr> <td>
<code>[4='eq_0.88']</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right"> <code>6.01</code>
</td> </tr>
 <tr> <td> <code>[5='eq_0.89_01']</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right">
<code>0.0000</code> </td>
 <td align="right"> <code>2.54</code> </td> </tr>
 <tr> <td> <code>[6='eq_0.92']</code> </td>
 <td align="right">
<code>-0.0000</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right"> <code>-5.29</code> </td> </tr>
 <tr> <td>
<code>[7='eq_0.94']</code> </td>
 <td align="right"> <code>-0.0000</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right"> <code>-4.58</code>
</td> </tr>
 <tr><td colspan="4">R^2= 1.000, N= 178, K= 8</td></tr>
</table>
</p>
</td><td valign="top">
<p>Here we have a case where the author seems to be using a feature of
<code>Test::More</code> that was introduced in version 0.88. Older
versions have only FAILs, 0.88 and up have only PASSes.<p>

<p>But such <i>mod:*</i> correlations can be misleading and at times
completely bogus when the number of test results is low and the number
of releases of a represented module is relatively high. Then (and not
only then) it may happen that only one tester farm has this version
installed and this testers farm is broken for some reason and the real
reason for the fail has nothing to do with this version of that
module. This is why you're required to reproduce FAILs and PASSes on
your own hardware and compare with the other correlations and draw
your conclusions carefully before writing a ticket on RT.</p>

</td></tr><tr><td valign="top">
<p>
<table border="1" width="100%"> <tr><th colspan="4">Regression#3
<a href="#ioc">mod:XML::ExtOn</a></th></tr>
<tr><td>Name</td><td align="right">Theta</td><td align="right">StdErr</td><td align="right">T-stat</td></tr>
<tr> </tr>
 <tr> <td> <code>[0='const']</code> </td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right"> <code>0.0000</code>
</td>
 <td align="right"> <code>17.27</code> </td> </tr>
 <tr> <td> <code>[1='eq_0.11']</code> </td>
 <td style="background: #aaffaa" align="right"> <code>1.0000</code>
</td>
 <td align="right"> <code>0.0000</code> </td>
 <td align="right"> <code>143147779654612836944.00</code> </td> </tr>
<tr><td colspan="4">R^2= 1.000, N= 26, K= 2</td></tr> </table>
</p>
</td><td valign="top">

<p>One annoyance of regression analysis is the fact that every
calculation presented is always relative to a <i>reference
group</i>. See for example this result:</p>


<p>We can recognize that having installed version 0.11 of
<code>XML::ExtOn</code> helps to get a PASS. But we cannot see,
compared to <i>which other version</i>. This is something you have to
test yourself before drawing conclusions. This is why you sometimes
will want to click on the header of the table to zoom in into the
actual test inputs as described in the next section.</p>

</td></tr><tr><td></td><td>
<a name="ltid"><h3>Links to input data</h3></a>

</td></tr><tr><td valign="top">
<p>
<h3>Pod-ToDocBook-0.7</h3>
</p>
<p>
<form action="#ltid" method="get" enctype="application/x-www-form-urlencoded">
<input type="submit" name="SUBMIT_xxx" value="Submit"/><br/>
<select name="field" size="8" multiple="multiple">
  <option selected="selected">mod:XML::ExtOn</option>
  <option>qr:(You planned.*)</option>
  <option>meta:date</option>
  <option>id</option>
<option>----below this line sorted alphabetically----</option>
  <option>conf:Off_t</option>
  <option>conf:alignbytes</option>
  <option>conf:archname</option>
  <option>conf:archname+osvers</option>
  <option>conf:bincompat5005</option>
  <option>conf:byteorder</option>
  <option>conf:cc</option>
  <option>conf:cccdlflags</option>

  <option>conf:ccdlflags</option>
</select>
</form>
</p>
<p>
<table border="1">
<tr><th>id</th><th>state</th>

<th>mod:XML::ExtOn</th>

</tr>

<tr>
<td>
<a href="http://nntp.x.perl.org/group/perl.cpan.testers/6231877">6231877</a>

</td>
<td>



<b style="color: brown">UNKNOWN</b>
</td>

<td>
0.11
</td>

</tr>

<tr>
<td>

<a href="http://nntp.x.perl.org/group/perl.cpan.testers/6172961">6172961</a>
</td>
<td>



<b style="color: brown">UNKNOWN</b>
</td>

<td>
0.11
</td>

</tr>

<tr>
<td>
<a href="http://nntp.x.perl.org/group/perl.cpan.testers/6161798">6161798</a>
</td>
<td>



<b style="color: green">PASS</b>
</td>

<td>
0.11
</td>

</tr>

<tr>
<td>
<a href="http://nntp.x.perl.org/group/perl.cpan.testers/6155043">6155043</a>
</td>
<td>



<b style="color: green">PASS</b>
</td>

<td>
0.11

</td>

</tr>

<tr>
<td>
<a href="http://nntp.x.perl.org/group/perl.cpan.testers/6095573">6095573</a>
</td>
<td>



<b style="color: green">PASS</b>
</td>

<td>
0.11
</td>

</tr>

<tr>
<td>
...
</td>
<td>
...
</td>

<td>
...
</td>

</tr>

<tr>
<td>

<a href="http://nntp.x.perl.org/group/perl.cpan.testers/4955988">4955988</a>
</td>
<td>



<b style="color: red">FAIL</b>
</td>

<td>
0.09
</td>

</tr>

<tr>
<td>
<a href="http://nntp.x.perl.org/group/perl.cpan.testers/4942282">4942282</a>
</td>
<td>



<b style="color: red">FAIL</b>
</td>

<td>
0.09
</td>

</tr>

<tr>
<td>
<a href="http://nntp.x.perl.org/group/perl.cpan.testers/4931627">4931627</a>
</td>
<td>



<b style="color: red">FAIL</b>
</td>

<td>
0.09

</td>

</tr>

<tr>
<td>
...
</td>
<td>
...
</td>

<td>
...
</td>

</tr>


</table> </p> </td><td valign="top"> <p>If you click on the header of
an individual regression result you follow a link to this table. It
shows a summary of the input data collected and each line in turn has
a link to the original cpantester report containing the referenced
value. You can choose from the drop-down menu which columns you want
to have summarized. Note that you can select multiple columns in this
drop down simultaneously. If Javascript is enabled in your browser you
can sort this table by clicking on the table header in the column you
want to sort.</p>

<p>If you want to collect statistics you're missing here, you may want
to check out <code>CPAN::Testers::ParseReport</code>.</p>

</td></tr><tr><td></td><td>
<h3>How can I help?</h3>

<p>Write me your bugreports at the <a
href="https://rt.cpan.org/Dist/Display.html?Name=CPAN-Testers-ParseReport">CPAN-Testers-ParseReport
Tracker</a>. Become a <a
href="http://wiki.cpantesters.org/wiki/QuickStart">CPAN tester</a>.
Provide more and diverse data. When you find a pass to fail ratio of
9:5 you should not expect too much and rather try to improve the data
than draw conclusions. Be aware that the regression analysis often is
exposing <a
href="https://en.wikipedia.org/wiki/Covariance">covariances</a> that
explain nothing. Healthy results typically appear with higher numbers
of PASS and FAIL. When the column <i>high correlations</i> shows many
different candidate
fields then it is also very likely that many testers had identical
configurations and the results are not really telling much.</p>

<p>Watch your own perl installations and see where you can try to
diversify the data and provide tests with them. If the data suggest
that perl 5.8.4 is failing, try to find a 5.8.4 on your disk and see
if you can get it working. If you find your own results in the sample
try to reproduce them. Too often we have seen random test results. And
try to produce the other result by variation of parameters you can
influence. Anything that makes the data less uniform automatically
helps the statistical methods to bring the most interesting factors to
the top.</p>

<p> And when you identify the culprit don't hesitate to report the bug
to RT (or whichever bug tracking has been chosen for the distro) so
that others do not waste their time with the same distro.</p>

</td></tr><tr><td></td><td>
<h3>Source code</h3>

<p>The sources of all the programs involved here are in my git repo of
cpan and related tools git://repo.or.cz/andk-cpan-tools.git . The main
script is cnntp-solver.pl. It is maintaining the databases and
producing regressions. The catalyst app with the name
<i>CPAN-Blame</i> produces the pages.</p>


<p>Andreas König, 2015-03-31</p>
</td></tr></table>
[% END %]