From 1c1b2c7d581c45b650ac20809bb37a28e41f3c09 Mon Sep 17 00:00:00 2001 From: Joe Moudrik Date: Tue, 24 Jul 2012 15:54:42 +0200 Subject: [PATCH] tex: again and again, another round of review --- tex/POSUDEK3 | 26 ++++++++--- tex/POSUDEK4 | 146 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 166 insertions(+), 6 deletions(-) create mode 100644 tex/POSUDEK4 diff --git a/tex/POSUDEK3 b/tex/POSUDEK3 index 001a2b2..0684823 100644 --- a/tex/POSUDEK3 +++ b/tex/POSUDEK3 @@ -1,10 +1,15 @@ Comments to the Reviewer 1 -Foremostly, I would like to thank for pointing out the problems in our paper. +Foremostly, I would like to thank for pointing out the problems in +our paper. -The main problem was the excessive use of footnotes and lot of one sentence paragraphs. +The main problem was the excessive use of footnotes and lot of one +sentence paragraphs. -We have cut down the number of footnotes to maximum of 2 footnotes per page. Furthermore, we have reformated/merged the one-sentence paragraphs and cut down the use of colons & semicolons as well, hoping to make our paper more fluent. +We have cut down the number of footnotes to maximum of 2 footnotes per +page. Furthermore, we have reformated/merged the one-sentence paragraphs +and cut down the use of colons & semicolons as well, hoping to make our +paper more fluent. > REFERENCES > The References contain many errors: @@ -13,9 +18,18 @@ We have cut down the number of footnotes to maximum of 2 footnotes per page. Fur > - Some years bracketed, some not bracketed. > - Other formatting errors, etc. -I fully aggree, that the References formating is not optimal in every regard, this is however caused by the official IEEE Transactions bibliography style. Different documents (Book, Article, Electronic, Tech report, ..) have different position (and bracketing) of the year field. I took special care to make sure that all the fields in all the references are present (if possible), and I am afraid I cannot make it any better than the current version. - -The use of et.al. in the paper refers mostly to online project with at least partial public contribution, where gathering list of all the authors (likely very long) is very complicated (if possible). We use full author list in other cases. +I fully aggree, that the References formating is not optimal in every +regard, this is however caused by the official IEEE Transactions +bibliography style. Different documents (Book, Article, Electronic, +Tech report, ..) have different position (and bracketing) of the year +field. I took special care to make sure that all the fields in all the +references are present (if possible), and I am afraid I cannot make it +any better than the current version. + +The use of et.al. in the paper refers mostly to online project with at +least partial public contribution, where gathering list of all the authors +(likely very long) is very complicated (if possible). We use full author +list in other cases. -------------------- Comments to the Reviewer 2 diff --git a/tex/POSUDEK4 b/tex/POSUDEK4 new file mode 100644 index 0000000..4d27b58 --- /dev/null +++ b/tex/POSUDEK4 @@ -0,0 +1,146 @@ +>Reviewer: 2 +> Comments to the Author The authors failed to properly address my +> previous comments. They ignored some of the references I provided which +> demonstrated a lack of novelty in their approach. Instead of discussing +> these references properly to differentiate their work from previous +> work in the literature, they chose to ignore that literature. + +We explained our decision to ignore the 2 of 3 papers suggested in +previous-previous-Reply To Rewievers, which I include below this reply +for the sake of clarity. + +We found the 2 other papers irrelevant and feebly useful. The papers deal +with general go-unspecific Skill and Competence theory, which - judging +by the results in papers provided - present tools that do not perform +nearly as good as our solution, as we mention in the paper. + +> Moreover, the paper is written as a simple undergraduate +> project. Describing the different basic classifiers which are all +> available in Weka (noting that the authors did not acknowledge which +> software they used) is not appropriate for a journal on the level +> of CIAIG. + +I shall leave the judging of whether our paper is or is not written as +a simple undergraduate project to some other authority. Even though, +a project does not have to be complex and/or postgraduate to be novel and +interesting. + +We consider a gentle introduction to data-mining methods used +as a positive thing regarding the fact that the paper is targeted for +people who might not be familiar with data-mining. + +The note about not acknowledging software we used is simply false. We +acknowledged and referenced the software used rigorously and thoroughly +in the section III. G. (Data Mining > Implementation). + +> Putting all of that aside, the approach does not scale and the fact +> that they only focused on clusters with a specific group of ranking +> does not prove the aim of the paper. Some statements demonstrate a lack +> of understanding of classification. For example, how can you say that +> neural networks do not work when you increase sample size? This does +> not make any sense. + +I do not understand how does the reviewer mean that our approach does not +scale. The more data (the bigger the scale) the more precisely we can +infer various characteristics of players. Moreover our feature extraction +procedure can be used to infer different things one might be interested +in. The examples of Style, Strength, Year of player presented in paper +are just proofs of concept. + +Our paper focused on devising Go-specific, good performing, informed +approaches to feature extraction to describe a player (or generaly any +group of players/games with some common relationship), a requirement +we believe has been met. + +The reviewer does not seem to have read our paper very deeply and did +not understand our methodology of preparing the training vectors by +sampling of the game set. We said quite the opposite (to cite our paper, +Section IV-C2): + However, DUE TO THE DECREASING NUMBER OF TRAINING + VECTORS with increasing game sample sizes, the neural network + gets unusable for large sample sizes. + +Understandably, given a fixed number of games in the database, the +biggger the number of games we use to sample one training vector, the +smaller the number of vectors. + +We have tried to make this even clearer in this revision of the paper. + +> There is no justification given as to why did you choose these +> classifiers. There is no discussion whatsoever on generalization. The +> results do not differentiate between training and testing. Overall, the +> classification results are flawed from a machine learning perspective. + +As presented in the paper, the classifiers were used to prove that the +pattern trend analysis and feature extraction suggested in the paper +work. As these machine learning tools simply learn a mapping, some other +tools might be freely chosen to learn from our pattern vectors. We have +shown that even with some basic classifiers we used, the pattern analysis +and feature extraction approach suggested is viable, and it is perfectly +possible to reliably infer non-trivial information even from a small +sample of games. + +From a machine learning perspective, our results are not flawed at all. +As we write in the paper, we have strictly divided the data samples to +Training and Testing data and performed a many-fold cross validation to +maximize the reliability of our approach. + +As clearly stated in the paper, the results presented in the +tables are always the results for Testing data. Though we could add +performance-tables of the methods for the Training data, these are not +very interesting from our point of view and would practically doubled +the current number of tables. + +====================================================================== +====== Below is the aforementioned last reply to the reviewer ======== +====================================================================== + + +> In this paper, the authors collected a large corpus of GO games and +> analysed it using data mining techniques. The idea is not novel, as it +> seems that the authors are not aware of the large literature that exists +> on this topic. For example, you should discuss in details the following +> work in your paper +> +> Ghoneim A., Essam D., and Abbass H.A. (2011) On Computations and +> Strategies for Real and Artificial Systems, European Conference on +> Artificial Life, Paris, August 8-12. +> +> Ghoneim A., Essam D., and Abbass H.A. (2011) Competency Awareness in +> Strategic Decision Making, IEEE Conference on Cognitive Methods in +> Situation Awareness and Decision Support, Florida, USA, February, 2011. +> +> Michael Harré, Terry Bossomaier, Allan Snyder: The Development of +> Human Expertise in a Complex Environment. Minds and Machines 21(3): +> 449-464 (2011) + +We were not aware of this research, so we have altered our paper to +mention the second conference paper, which seems to be relevant. However, +we still consider our work novel, since the solution in the paper 1) +uses GnuGo as a blackbox, making it hard to do any pattern analysis, +diminishing the Go-theoretical and study usage. 2) does not develop any +specialized Go-concerned techniques 3) performs poorly. + +> The analysis of style is interesting. However, the whole paper is taking +> a pure data analysis approach. The paper has a table after another, +> with minimum discussion of the theory or the implications of the results. +> On the theory side, the authors need to discuss the skill and competency +> literature, which represent the foundation for this work. From the +> implications side, what does this all mean? Putting together the list of +> discoveries from this work, how will it translate into something useful +> for playing go? What is the use of this research? + +The tables and precise data are very important to back up the statements +about the performance of our framework. Additionaly, precise numerical +results allow for replication of our results and their comparison with +other methods possibly developed in the future. In this manner our paper +tries to honour the rigorous scientific methodology. + +Although our work might be viewed through the skill and competency theory, +we do not find these very useful or relevant for Go-analysis. + +We believe that we have made the possible implications and uses for +playing Go clear in Section VI (Proposed Applications), and Section VII +(Future Work). We are currently preparing online application based on +the research, which will help to pinpoint patterns to avoid, games to +replay and possible study directions. -- 2.11.4.GIT