3 # BioPerl module for Bio::PullParserI
5 # Please direct questions and support issues to <bioperl-l@bioperl.org>
7 # Cared for by Sendu Bala <bix@sendu.me.uk>
11 # You may distribute this module under the same terms as perl itself
13 # POD documentation - main docs before the code
17 Bio::PullParserI - A base module for fast 'pull' parsing
21 # do not use this class, it is intended for parser module
26 If you are writing a module to parse some new format, you may wish to use
27 a 'pull' approach whereby you only do work (reading file data, parsing it,
28 turning the parsed data in an object) when absolutely necessary.
30 PullParserI provides a system for doing exactly that. As a PullParser you
31 need a chunk. A chunk is just a Bio::Root::IO that contains all the raw data
32 you would want to parse. You can use the chunk() method to create a chunk from
33 a filename, existing filehandle or even a string. If you make a chunk from a
34 large file, but actually only want your chunk to be some portion of the whole
35 file, supply start and end amounts in bytes to chunk() at the same time.
36 The methods _chunk_seek() and _chunk_tell() provide seeks and tells that are
37 relative to the start and end of your chunk, not the whole file.
39 The other thing you will need to decide when making a chunk is how to handle
40 piped input. A PullParser typically needs seekable data to parse, so if your
41 data is piped in and unseekable, you must decide between creating a temp file
42 or reading the input into memory, which will be done before the chunk becomes
43 usable and you can begin any parsing. Alternatively you can choose to force
44 a sequential read, in which case you can make use of _dependencies() to define
45 the linear order of methods that would result in the file being read
46 sequentially. The return value of _sequential() is also useful here, if you
47 would need to cache some data or otherwise behave differently during a
50 The main method in the system is get_field(). This method relies on the
51 existance of a private hash reference accessible to it with the method
52 _fields(). That hash ref should have as keys all the sorts of data you will want
53 to parse (eg. 'score'), and prior to parsing the values would be undefined. A
54 user of your module can then call either $module-E<gt>get_field('score') or
55 $module-E<gt>score and get_field will either return the answer from
56 $self-E<gt>_fields-E<gt>{score} if it is defined, or call a method _discover_score()
57 first if not. So for the system to work you need to define a _discover_*()
58 method for every field in the fields hash, and ensure that the method stores an
59 answer in the fields hash.
61 How you implement your _discover_* methods is up to you, though you should never
62 call a _discover_* method directly yourself; always use get_field(), since
63 get_field() will deal with calling dependant methods for you if a forced
64 sequenctial read is in progress due to piped input. You will almost certainly
65 want to make use of the various chunk-related methods of this class (that are
66 denoted private by the leading '_'; this means you can use them as the author of
67 a parser class, but users of your parser should not).
69 Primary amongst them is _*_chunk_by_end() to which you provide text that
70 represents the end of your desired chunk and it does a readline with your
71 argument as $/. The chunk knows about its line-endings, so if you want your
72 end definition to include a new line, just always use "\n" and PullParserI will
73 do any necessary conversion for you.
75 If your input data is hierarchical (eg. report-E<gt>many results-E<gt>many hits-E<gt>many
76 hsps), and you want an object at the leaf of the hierarchy to have access to
77 information that is shared amongst all of them (is parsed in the root), you
78 don't have to copy the data to each leaf object; simply by defining parent(),
79 when you call get_field() and the requested field isn't in your leaf's fields
80 hash, the leaf's parent will be asked for the field instead, and so on till
83 See Bio::SearchIO::hmmer_pull for an example of implementing a parser using
90 User feedback is an integral part of the evolution of this and other
91 Bioperl modules. Send your comments and suggestions preferably to
92 the Bioperl mailing list. Your participation is much appreciated.
94 bioperl-l@bioperl.org - General discussion
95 http://bioperl.org/wiki/Mailing_lists - About the mailing lists
99 Please direct usage questions or support issues to the mailing list:
101 I<bioperl-l@bioperl.org>
103 rather than to the module maintainer directly. Many experienced and
104 reponsive experts will be able look at the problem and quickly
105 address it. Please include a thorough description of the problem
106 with code and data examples if at all possible.
108 =head2 Reporting Bugs
110 Report bugs to the Bioperl bug tracking system to help us keep track
111 of the bugs and their resolution. Bug reports can be submitted via the
114 http://bugzilla.open-bio.org/
116 =head1 AUTHOR - Sendu Bala
118 Email bix@sendu.me.uk
122 Inspired by a posting by Aaron J. Mackey
126 The rest of the documentation details each of the object methods.
127 Internal methods are usually preceded with a _
131 # Let the code begin...
133 package Bio
::PullParserI
;
135 use vars
qw($AUTOLOAD $FORCE_TEMP_FILE);
140 use base qw(Bio::Root::RootI);
143 # chunk() needs perl 5.8 feature for modes other than temp_file, so will
144 # workaround by forcing temp_file mode in <5.8. Could also rewrite using
145 # IO::String, but don't want to.
147 $FORCE_TEMP_FILE = 1;
154 Usage : $obj->_fields( { field1 => undef } );
155 my $fields_ref = $obj->_fields;
156 Function: Get/set the hash reference containing all the fields for this parser
158 Args : none to get, OR hash ref to set
165 $self->{_fields
} = shift;
167 unless (defined $self->{_fields
}) {
168 $self->{_fields
} = { };
170 return $self->{_fields
};
176 Usage : if ($obj->has_field('field_name') {...}
177 Function: Ask if a particular object has a given field (doesn't ask ancestors)
179 Args : string (the field name to test)
184 my ($self, $desired) = @_;
186 return exists $self->_fields->{$desired};
192 Usage : my $field_value = $obj->get_field('field_name');
193 Function: Get the value of a given field. If this $obj doesn't have the field,
194 it's parent() will be asked, and so on until there are no more
196 Returns : scalar, warns if a value for the field couldn't be found and returns
198 Args : string (the field to get)
204 my $desired = shift || return keys %{$self->_fields};
205 if (exists $self->_fields->{$desired}) {
206 unless (defined $self->_fields->{$desired}) {
207 my $method = '_discover_'.$desired;
209 my $dependency = $self->_dependencies($desired);
210 if ($dependency && ! defined $self->_fields->{$dependency}) {
211 $self->get_field($dependency);
215 $self->$method unless defined $self->_fields->{$desired};
217 return $self->_fields->{$desired};
220 # is it a field of our parent? (checks all ancestors)
221 if (my $parent = $self->parent) {
222 return $parent->get_field($desired);
225 $desired =~ s/_discover_//;
226 $self->warn("This report does not hold information about '$desired'");
233 Usage : $obj->parent($parent_obj);
234 my $parent_obj = $obj->parent;
235 Function: Get/set the parent object of this one.
236 Returns : Bio::PullParserI
237 Args : none to get, OR Bio::PullParserI to set
243 if (@_) { $self->{parent
} = shift }
244 return $self->{parent
} || return;
250 Usage : $obj->chunk($filename);
251 my $chunk = $obj->chunk;
252 Function: Get/set the chunk of this parser.
253 Returns : Bio:Root::IO
254 Args : none to get, OR
255 First argument of a GLOB reference, filename string, string data to
256 treat as the chunk, or Bio::Root::IO.
257 Optionally, also provide:
258 -start => int : the byte position within the thing described by the
259 first arguement to consider as the start of this
261 -end => int : the byte position to consider as the end (default
263 -piped_behaviour => 'memory'|'temp_file'|'sequential_read'
265 The last option comes into effect when the first argument is
266 something that cannot be seeked (eg. piped input filehandle).
267 'memory' means read all the piped input into a string
268 first, then set the chunk to that string.
269 'temp_file' means read all the piped input and output it to
270 a temp file, then set the chunk to that temp file.
271 'sequential_read' means that the piped input should be read
272 sequentially and your parsing code must cope with
273 not being able to seek.
274 'memory' is the fastest but uses the most memory. 'temp_file' and
275 'sequential_read' can be slow, with 'temp_file' being the most memory
276 efficient but requiring disc space. The default is 'sequential_read'.
277 Note that in versions of perl earlier than 5.8 only temp_file works
278 and will be used regardless of what value is supplied here.
286 my $thing = shift || $self->throw("Trying to set chunk() to an undefined value");
287 if (ref($thing) eq 'GLOB') {
288 $self->{_chunk
} = Bio
::Root
::IO
->new(-fh
=> $thing);
290 elsif (ref(\
$thing) eq 'SCALAR') {
291 if ($thing !~ /\n/ && -e
$thing) {
292 $self->{_chunk
} = Bio
::Root
::IO
->new(-file
=> $thing);
295 unless ($FORCE_TEMP_FILE) {
296 # treat a string as a filehandle
297 open(my $fake_fh, "+<", \
$thing); # requires perl 5.8
298 $self->{_chunk
} = Bio
::Root
::IO
->new(-fh
=> $fake_fh);
301 my ($handle) = $self->{_chunk
}->tempfile();
302 print $handle $thing;
303 $self->{_chunk
} = Bio
::Root
::IO
->new(-fh
=> $handle);
307 elsif ($thing->isa('Bio::Root::IO')) {
308 $self->{_chunk
} = $thing;
311 $self->throw("Unknown input into chunk()");
314 my ($piped_behaviour, $start, $end);
316 ($piped_behaviour, $start, $end) =
317 $self->_rearrange([qw(PIPED_BEHAVIOUR START END)], @_);
319 $piped_behaviour ||= 'sequential_read';
320 $FORCE_TEMP_FILE && ($piped_behaviour = 'temp_file');
322 $self->_chunk_true_start($start);
323 $self->_chunk_true_end($end);
325 # determine if the chunk is seekable
326 my $fh = $self->{_chunk
}->_fh;
328 my $first_line = <$fh>;
330 my $seekable = tell($fh) == 0;
332 if ($piped_behaviour eq 'memory') {
333 my $string = $first_line;
337 $self->chunk($string);
339 elsif ($piped_behaviour eq 'temp_file') {
340 my ($handle) = $self->{_chunk
}->tempfile();
341 print $handle $first_line;
346 $self->chunk($handle);
348 elsif ($piped_behaviour eq 'sequential_read') {
349 $self->{_chunk
}->_pushback($first_line);
350 $self->_sequential(1);
353 $self->throw("Unknown piped behaviour type '$piped_behaviour'");
357 # determine our line ending
358 if ($first_line =~ /\r\n/) {
359 $self->_line_ending("\r\n");
361 elsif ($first_line =~ /\r/) {
362 $self->_line_ending("\r");
365 $self->_line_ending("\n");
369 return $self->{_chunk
} || return;
375 Usage : if ($obj->_sequential) {...}
376 Function: Ask if we have to do operations such that the input is read
379 Args : none to get, OR boolean to set (typically, you should never set this
387 $self->{_sequential
} = shift;
389 return $self->{_sequential
} || 0;
394 Title : _dependencies
395 Usage : $obj->_dependencies( { field1 => field2 } );
396 my $dependancy = $obj->_dependencies('field_name');
397 Function: Set the fields that are dependent on each other, or get the field
398 than another is dependent upon.
399 Returns : string (a field name)
400 Args : string (a field name) to get, OR hash ref to initially set, with
401 field names as keys and values, key field being dependent upon value
407 my ($self, $thing) = @_;
409 if (ref($thing) eq 'HASH') {
410 $self->{_dependencies
} = $thing;
413 return $self->{_dependencies
}->{$thing};
417 =head2 _chunk_true_start
419 Title : _chunk_true_start
420 Usage : my $true_start = $obj->_chunk_true_start;
421 Function: Get/set the true start position of the chunk within the filehandle
424 Args : none to get, OR int to set (typically, you won't set this yourself)
428 sub _chunk_true_start
{
431 $self->{_chunk_start
} = shift;
433 return $self->{_chunk_start
} || 0;
436 =head2 _chunk_true_end
438 Title : _chunk_true_end
439 Usage : my $true_end = $obj->_chunk_true_end;
440 Function: Get/set for the true end position of the chunk within the filehandle
443 Args : none to get, OR int to set (typically, you won't set this yourself)
447 sub _chunk_true_end
{
450 $self->{_chunk_end
} = shift;
452 return $self->{_chunk_end
};
458 Usage : my $line_ending = $obj->_line_ending;
459 Function: Get/set for the line ending for the chunk.
461 Args : none to get, OR string to set (typically, you won't set this
469 $self->{_chunk_line_ending
} = shift;
471 return $self->{_chunk_line_ending
};
477 Usage : $obj->_chunk_seek($pos);
478 Function: seek() the chunk to the provided position in bytes, relative to the
479 defined start of the chunk within its filehandle.
481 In _sequential() mode, this function does nothing.
489 my ($self, $pos) = @_;
490 $self->throw("Undefined position passed") unless defined $pos;
491 return if $self->_sequential;
493 my $fh = $self->chunk->_fh;
495 # seek to the defined start
496 seek($fh, $self->_chunk_true_start, 0);
498 # now seek to desired position relative to defined start
505 Usage : my $pos = $obj->_chunk_tell;
506 Function: Get the current tell() position within the chunk, relative to the
507 defined start of the chunk within its filehandle.
509 In _sequential() mode, this function does nothing.
518 return if $self->_sequential;
520 my $fh = $self->chunk->_fh;
521 return tell($fh) - $self->_chunk_true_start;
524 =head2 _get_chunk_by_nol
527 Usage : my $string = $obj->_get_chunk_by_nol;
528 Function: Get a chunk of chunk() from the current position onward for the given
531 Args : int (number of lines you want)
535 sub _get_chunk_by_nol
{
536 my ($self, $nol) = @_;
537 $nol > 0 || $self->throw("Can't request a chunk of fewer than 1 lines");
542 while (defined($_ = $self->chunk->_readline)) {
545 last if $count == $nol;
548 my $current = $self->_chunk_tell;
549 my $end = ($current || 0) + $self->_chunk_true_start;
550 if (! $current || ($self->_chunk_true_end ?
$end <= $self->_chunk_true_end : 1)) {
556 =head2 _get_chunk_by_end
558 Title : _get_chunk_by_end
559 Usage : my $string = $obj->_get_chunk_by_end;
560 Function: Get a chunk of chunk() from the current position onward till the end
561 of the line, as defined by the supplied argument.
563 Args : string (line ending - if you want the line ending to include a new
568 sub _get_chunk_by_end
{
569 my ($self, $chunk_ending) = @_;
571 my $start = $self->_chunk_tell;
573 my $line_ending = $self->_line_ending;
574 $chunk_ending =~ s/\n/$line_ending/g;
575 local $/ = $chunk_ending || '';
576 my $line = $self->chunk->_readline;
578 my $current = $self->_chunk_tell;
579 my $end = ($current || 0) + $self->_chunk_true_start;
580 if (! $current || ($self->_chunk_true_end ?
$end <= $self->_chunk_true_end : 1)) {
584 $self->_chunk_seek($start);
588 =head2 _find_chunk_by_end
590 Title : _find_chunk_by_end
591 Usage : my $string = $obj->_find_chunk_by_end;
592 Function: Get the start and end of what would be a chunk of chunk() from the
593 current position onward till the end of the line, as defined by the
596 In _sequential() mode, this function does nothing.
598 Returns : _chunk_tell values for start and end in 2 element list
599 Args : string (line ending - if you want the line ending to include a new
604 sub _find_chunk_by_end
{
605 my ($self, $chunk_ending) = @_;
606 return if $self->_sequential;
608 my $line_ending = $self->_line_ending;
609 $chunk_ending =~ s/\n/$line_ending/g;
610 local $/ = $chunk_ending || '';
612 my $start = $self->_chunk_tell;
613 $self->chunk->_readline;
614 my $end = $self->_chunk_tell;
616 my $comp_end = $end + $self->_chunk_true_start;
617 if ($self->_chunk_true_end ?
$comp_end <= $self->_chunk_true_end : 1) {
618 return ($start, $end);
621 $self->_chunk_seek($start);
629 Function: Assumes that any unknown method called should be treated as
630 get_field($method_name).
638 ref($self) || return;
640 my $name = $AUTOLOAD;
641 $name =~ s/.*://; # strip fully-qualified portion
643 # is it one of our fields?
644 return $self->get_field($name);