File Coverage

blib/lib/Iterator/Records.pm
Criterion Covered Total %
statement 244 346 70.5
branch 49 78 62.8
condition 8 11 72.7
subroutine 52 71 73.2
pod 20 22 90.9
total 373 528 70.6


line stmt bran cond sub pod time code
1             package Iterator::Records;
2              
3 5     5   330199 use 5.006;
  5         56  
4 5     5   30 use strict;
  5         7  
  5         101  
5 5     5   22 use warnings;
  5         10  
  5         134  
6 5     5   25 use Carp;
  5         8  
  5         314  
7 5     5   2561 use Iterator::Simple;
  5         24709  
  5         377  
8             #use Data::Dumper;
9              
10             =head1 NAME
11              
12             Iterator::Records - a simple iterator for arrayref record sources
13              
14             =head1 VERSION
15              
16             Version 0.02
17              
18             =cut
19              
20             our $VERSION = '0.02';
21              
22              
23             =head1 SYNOPSIS
24              
25             Iterator::Records uses L to work with iterators whose values are arrayrefs of named fields. These can be called I.
26             A record stream can be seen as the same thing as a DBI retrieval, but without most of the machinery for DBI - and of course, a DBI query is one of the ways you
27             can build a record stream.
28              
29             The actual API of Iterator::Records isn't as simple or elegant as L, simply because there's more to keep track of. But the basic
30             approach is similar: an Iterator::Records object defines how to iterate something, then you use the iter() method to create an iterator from it.
31             The result is an Iterator::Simple iterator known to return records, i.e. arrayrefs of fields that match the field list specified.
32              
33             Note that the Iterator::Records object is an iterator *factory*, and the actual iterator itself is returned by the call to iter().
34              
35             use Iterator::Records;
36            
37             my $spec = Iterator::Records->new (, ['field 1', 'field 2']);
38            
39             my $iterator = $spec->iter();
40             while (my $row = $iterator->()) {
41             my ($field1, $field2) = @$row;
42             }
43            
44             $iterator = $spec->iter_hash();
45             while (my $row = $iterator->()) {
46             print $row->{field 1};
47             }
48            
49             my ($f1, $f2);
50             $iterator = $spec->iter_bind(\$f1, \$f2);
51             while ($iterator->()) {
52             print "$f1 - $f2\n";
53             }
54              
55             Note that the iterator itself is just an L iterator. Now hold on, though, because here's where things get interesting.
56              
57             my $recsource = Iterator::Records->new (sub { ... }, ['field 1', 'field 2']);
58             my $iterator = $recsource->select ("field 1")->iter;
59             while (my $row = $iterator->()) {
60             my ($field1) = @$row;
61             }
62            
63             my @fields = $recsource->fields();
64             my $fields = $recsource->fields(); # Returns an arrayref in scalar context.
65            
66             $rs = $recsource->where (sub { ... }, "field 1", "field 2");
67             $rs = $recsource->fixup ("field 1", sub { ... } );
68             $rs = $recsource->calc ("field 3", sub { ... } );
69             $rs = $rs->select ("field 2", "field 3", "field 1");
70             $rs = $rs->select (["field 2", "field 3", "field 1"]);
71            
72             $rs = $recsource->transform (["where", ["field 1", "=", "x"]],
73             ["fixup", ["field 1, sub { ... }]]);
74              
75              
76             Since Iterator::Records is essentially a more generalized way of iterating DBI results, there are a few wrappers to make things easy.
77              
78             my $dbh = Iterator::Records::db->connect(--DBI syntax--);
79             my $dbh = Iterator::Records::db->open('sqlite file');
80             my $dbh = Iterator::Records::db->open(); # Defaults to an in-memory SQLite database
81            
82             This is not the direct DBI handle; it's got simplified syntax as follows:
83              
84             my $value = $dbh->get ('select value from table where id=?', $id); # Single value retrieval in one whack.
85             $dbh->do ("insert ..."); # Regular insertion, just like in DBI, except simpler.
86             my $record = $dbh->insert ("insert ..."); # Calls last_insert_id ('', '', '', ''), which will likely fail except with SQLite.
87              
88             And then you have the actual iterator machinery.
89              
90             my $iter = $dbh->iterator ('select * from table')->iter();
91             my $sth = $dbh->prepare (--DBI syntax--);
92             my $iter = $sth->iter ($value1, $value2);
93             while ($iter->()) {
94             my ($field1, $field2) = @$_;
95             }
96            
97             We can load an iterator into a table. If you have Data::Tab installed, it will make a Data::Tab with the column names from this iterator.
98             Otherwise, it will simply return an arrayref of arrayrefs by calling Iterator::Simple's "list" method.
99              
100             my $data = $recsource->table;
101            
102             The "report" method returns an Iterator::Simple that applies an sprintf to each value in the record source. If you supply a list of fields to dedupe it will replace them with ""
103             if their value is the same as the previous row. This is useful for tabulated data where, for instance, the date may be the same from line to line and if so should only be
104             displayed once.
105              
106             my $report = $recsource->report ("%-20s %s", ["field 1"]); # Here, field 2 would not be deduped.
107             my $report = join ('\n', $recsource->report (...)->list);
108              
109             =cut
110              
111 5     5   96 use Iterator::Records;
  5         18  
  5         14307  
112              
113             =head1 BASIC ITERATION
114              
115             =head2 new (iterable, arrayref of fields)
116              
117             To specify an Iterator::Records from scratch, just take whatever iterable thing you have, and specify a list of fields in the resulting records.
118             If the iterable is anything but a coderef, Iterator::Records->iter will simply pass it straight to Iterator::Simple for iteration. If it's a coderef,
119             it will be called, and its return value will be passed to Iterator::Simple. This allows record streams to be reused.
120              
121             As an added bonus of this extra level of indirection, you can call "iter" with parameters that will be passed on to the coderef. This turns the
122             Iterator::Records object into a parameterizable iterator factory.
123              
124             =cut
125              
126             sub new {
127 24     24 1 1872 my ($class, $iterable, $fields) = @_;
128 24         54 my $self = bless ({}, $class);
129 24 50       73 croak "Iterator spec not iterable" unless Iterator::Simple::is_iterable($iterable);
130 24         254 $self->{gen} = $iterable;
131 24         48 $self->{f} = $fields;
132 24         50 $self->{id} = '*';
133 24         120 $self;
134             }
135              
136 21     21 0 83 sub fields { $_[0]->{f}; }
137             sub id {
138 0     0 0 0 my $self = shift;
139 0 0       0 if (scalar @_) {
140 0         0 $self->{id} = shift;
141             }
142 0         0 $self->{id};
143             }
144              
145             =head2 iter, iter_hash, iter_bind
146              
147             Basic iteration of the record source returns an arrayref for each record. Alternatively, an iterator can be created which returns a hashref for each
148             record, with the field names keying the return values in each record. This is less efficient, but it's often handy. The third option is to bind a list
149             of scalar references that will be written automagically on each retrieval. The return value in this case is still the original arrayref record.
150              
151             =cut
152              
153             sub iter {
154 45     45 1 4426 my $self = shift;
155 45 100       131 if (ref $self->{gen} eq 'CODE') {
156 20         52 return Iterator::Simple::iter($self->{gen}->(@_));
157             } else {
158 25         72 return Iterator::Simple::iter($self->{gen});
159             }
160             }
161             sub iter_hash {
162 2     2 1 1708 my $self = shift;
163 2         6 my $iter = $self->iter();
164             Iterator::Simple::Iterator->new(sub {
165 6 100   6   35 if (my $rec = $iter->()) {
166 4         26 my $ret = {};
167 4         6 my $i = 0;
168 4         8 foreach my $f (@{$self->{f}}) {
  4         9  
169 8         20 $ret->{$f} = $rec->[$i++];
170             }
171 4         28 return $ret;
172             } else {
173 2         12 return undef;
174             }
175 2         58 });
176             }
177             sub iter_bind {
178 2     2 1 580 my $self = shift;
179 2         6 my $iter = $self->iter();
180 2         53 my @fields = (@_);
181             Iterator::Simple::Iterator->new(sub {
182 6 100   6   4413 if (my $rec = $iter->()) {
183 4         25 my $i = 0;
184 4         9 foreach my $f (@fields) {
185 8         16 $$f = $rec->[$i++];
186             }
187 4         11 return $rec;
188             } else {
189 2         11 return undef;
190             }
191 2         10 });
192             }
193              
194             =head1 TRANSMOGRIFIERS
195              
196             Since our record stream sources are very often provided by fairly simple drivers (like the filesystem walker in File::Org), it's not at all unusual to find ourselves
197             in a position where we want to modify them on the fly, either filtering out some of the records or modifying the records as they go through. There are four different
198             "transmogrifiers" for record streams: where, select, calc, and fixup. The "where" transmogrifier discards records that don't match a particular pattern; "select"
199             removes columns; "calc" adds a column that is calculated by an arbitrary coderef provided; and "fixup" applies a coderef to the record to modify individual field
200             values.
201              
202             Each transmogrifier takes an iterator I, not an iterator - and returns a new specification that can be iterated. The source stream will then be iterated
203             internally.
204              
205             =head2 where (sub { ... }, 'field 1', 'field 2')
206              
207             Filtration of records is not really any different from igrep - given a record stream, we provide a coderef that tells us to include or not to include. If fields
208             are specified, their values for the record to be examined will be passed to the coderef as its parameters; otherwise the entire record is provided as an arrayref
209             and the coderef can extract values on its own. The list of fields is not affected.
210              
211             =head2 select ('field 1', 'field 3')
212              
213             Returns a spec for a new iterator that includes only the fields listed, in the order listed.
214              
215             =head2 calc ('new field', sub { ... }, 'field 1', 'field 2')
216              
217             Returns a spec for a new iterator that includes a new field calculated by the coderef provided; as for "where", if fields are listed they will be passed into the
218             coderef as parameters, but otherwise the entire record will be passed in. The new field will appear at the end of the current field list.
219              
220             =head2 fixup (sub { ... })
221              
222             Returns a spec for a new iterator in which each record is first visited by the coderef provided. This is just an imap in more record-based form. The field
223             list is unchanged.
224              
225             =head2 dedupe ('field 1', 'field 2')
226              
227             Keeps track of the last values for field 1 and field 2; if the new value is a duplicate, passes an empty string through instead. Useful for reporting.
228             The field list is unchanged.
229              
230             =head2 rename ('field 1', 'new name', [more pairs])
231              
232             To rename a field (or more than one), use 'rename'. The record is not changed.
233              
234             =head2 transmogrify (['where', ...], ['calc', ...])
235              
236             Any sequence of transmogrifiers can be chained together in a single step using the L method.
237              
238             =cut
239              
240             sub _find_offsets {
241 33     33   51 my $field_list = shift;
242 33         41 my $size = scalar @{$field_list}-1;
  33         60  
243 33         72 my @output;
244 33         61 foreach my $f (@_) {
245 49         93 my ($index) = grep { $field_list->[$_] eq $f } (0 .. $size);
  110         247  
246 49 100       187 croak "Unknown field '$f' used in transmogrifier" unless defined $index;
247 48         95 push @output, $index;
248             }
249 32         71 @output;
250             }
251              
252             sub where {
253 1     1 1 9 my $self = shift;
254 1         5 $self->transmogrify (['where', @_]);
255             }
256              
257             sub _where {
258 6     6   10 my $fields = shift;
259 6         9 my $tester = shift;
260 6         14 my @field_offsets = _find_offsets ($fields, @_);
261 6         12 my $parms = "";
262 6 50       24 $parms = '$rec->[' . join ('], $rec->[', @field_offsets) . ']' if scalar @field_offsets;
263            
264 6         13 my $sub = <<"EOF";
265             sub {
266             my \$rec = shift;
267             return \$rec if \$tester->($parms);
268             return undef;
269             }
270             EOF
271             #print STDERR $sub;
272 6         534 eval $sub;
273             }
274              
275             sub select {
276 1     1 1 3 my $self = shift;
277 1         5 $self->transmogrify (['select', @_]);
278             }
279             sub _select_fields {
280 3     3   4 shift;
281 3         11 \@_;
282             }
283             sub _select {
284 7     7   13 my $fields = shift;
285 7         16 my @field_offsets = _find_offsets ($fields, @_);
286 7         12 my $parms = "";
287 7 50       32 $parms = '$rec->[' . join ('], $rec->[', @field_offsets) . ']' if scalar @field_offsets;
288            
289 7         34 my $sub = <<"EOF";
290             sub {
291             my \$rec = shift;
292             return [$parms];
293             }
294             EOF
295             #print STDERR $sub;
296 7         527 eval $sub;
297             }
298              
299             sub calc {
300 1     1 1 4 my $self = shift;
301 1         5 $self->transmogrify (['calc', @_]);
302             }
303             sub _calc_fields {
304 4     4   6 my @fields = @{$_[0]};
  4         12  
305 4         9 push @fields, $_[2];
306 4         13 \@fields;
307             }
308             sub _calc {
309 8     8   15 my $fields = shift;
310 8         10 my $calcer = shift;
311 8         11 shift; # The name of our output variable
312 8         23 my @field_offsets = _find_offsets ($fields, @_);
313 7         12 my $parms = "";
314 7 100       29 $parms = '$rec->[' . join ('], $rec->[', @field_offsets) . ']' if scalar @field_offsets;
315            
316 7         17 my $sub = <<"EOF";
317             sub {
318             my \$rec = shift;
319             return [@\$rec, \$calcer->($parms)];
320             }
321             EOF
322             #print STDERR $sub;
323 7         607 eval $sub;
324             }
325              
326             sub fixup {
327 1     1 1 30 my $self = shift;
328 1         9 $self->transmogrify (['fixup', @_]);
329             }
330             sub _fixup {
331 2     2   4 my $fields = shift;
332 2         3 my $calcer = shift;
333 2         7 my @field_offsets = _find_offsets ($fields, @_);
334 2         7 my $output = '$rec->[' . shift(@field_offsets) . ']';
335 2         4 my $parms = "";
336 2 50       10 $parms = '$rec->[' . join ('], $rec->[', @field_offsets) . ']' if scalar @field_offsets;
337            
338 2         7 my $sub = <<"EOF";
339             sub {
340             my \$rec = shift;
341             \$rec = [@\$rec];
342             $output = \$calcer->($parms);
343             return \$rec;
344             }
345             EOF
346             #print STDERR $sub;
347 2         211 eval $sub;
348             }
349              
350             sub dedupe {
351 2     2 1 7 my $self = shift;
352 2         7 $self->transmogrify (['dedupe', @_]);
353             }
354             sub _dedupe {
355 4     4   6 my $fields = shift;
356 4         11 my ($target) = _find_offsets ($fields, $_[0]);
357              
358 4         13 my $value = '$rec->[' . $target . ']';
359            
360 4         5 my $last_value = ''; # Closures are magic.
361              
362 4         11 my $sub = <<"EOF";
363             sub {
364             my \$rec = shift;
365             my \$val = $value;
366             if (\$val eq \$last_value) {
367             \$rec = [@\$rec];
368             $value = '';
369             } else {
370             \$last_value = \$val;
371             }
372             return \$rec;
373             }
374             EOF
375             #print STDERR $sub;
376 4         451 eval $sub;
377             }
378              
379             sub _gethashval_fields {
380 1     1   3 my $fields = shift;
381 1         2 shift; # skip the name of the value bag.
382 1         3 my @fields = (@$fields, @_);
383 1         4 \@fields;
384             }
385              
386             sub _gethashval {
387 2     2   3 my $fields = shift;
388 2         6 my ($fieldno) = Iterator::Records::_find_offsets ($fields, shift);
389 2         11 my $vals = "\$rec->[$fieldno]->{" . join ("}, \$rec->[$fieldno]->{", @_) . '}';
390            
391 2         6 my $sub = <<"EOF";
392             sub {
393             my \$rec = shift;
394             return [@\$rec, $vals ];
395             }
396             EOF
397             #print STDERR $sub;
398 2         140 eval $sub;
399             }
400              
401             sub _count {
402 2     2   4 my $fields = shift;
403 2         3 shift; # The name of our output variable
404 2         3 my $count = shift;
405 2 50       7 $count = 0 unless defined $count;
406 2         3 $count -= 1;
407             sub {
408 4     4   6 my $rec = shift;
409 4         5 $count += 1;
410 4         10 [@$rec, $count];
411             }
412 2         44 }
413              
414             sub _limit {
415 3     3   6 my $fields = shift;
416 3         5 my $limit = shift;
417 3         6 my $count = 0;
418             sub {
419 8 100   8   18 return undef unless $count < $limit;
420 4         5 $count += 1;
421 4         9 shift();
422             }
423 3         12 }
424              
425             # The walker framework is *really* minimalistic. It basically looks exactly like _calc, but instead of returning just the parent record (to which, unlike calc, it can add fields),
426             # it can return a list of records and/or iterators to take the place of the parent record - which can include the parent record, so it's effectively an add-or-replace.
427             sub _walk_fields {
428 2     2   3 my $fields = shift;
429 2         4 shift; # skip the walker coderef
430 2         4 my $newfields = shift;
431 2 50       5 if (defined $newfields) {
432 2         9 return [@$newfields, @$fields];
433             }
434 0         0 $fields;
435             }
436             sub _walk {
437 4     4   6 my $fields = shift;
438 4         6 my $walker_factory = shift;
439 4         5 my $newfields = shift; # The (optional) arrayref of fields we're going to add
440 4         10 my @field_offsets = _find_offsets ($fields, @_);
441              
442             # The walker framework is unusual in that, instead of building a closure here, we are actually given a closure *factory* that will make it for us.
443             # This factory is given the fields for the input record, the new fields expected for the output record, the list of fields it's supposed to use,
444             # and the offsets of those fields in the input record.
445             # It returns a closure that takes the input record and returns a list of records and/or new iterators that ->transmogrify will buffer and return as appropriate.
446 4         15 $walker_factory->($fields, $newfields, \@_, \@field_offsets);
447             }
448              
449             sub rename {
450 0     0 1 0 my $self = shift;
451 0         0 $self->transmogrify (['rename', @_]);
452             }
453             sub _rename_fields {
454 1     1   3 my $fields = shift;
455 1         3 my @fields = @$fields;
456 1         6 while (scalar @_) {
457 2         4 my $from = shift;
458 2         4 my $to = shift;
459 2 50       5 last unless defined $to;
460 2 100       4 @fields = map { $_ eq $from ? $to : $_ } @fields;
  6         19  
461             }
462 1         4 \@fields;
463             }
464             sub _no_change {
465 4     4   6 sub { $_[0] }; # passes the record through as efficiently as possible
  3     3   10  
466             }
467              
468             our $transmogrifiers = {
469             'where' => [undef, \&_where],
470             'select' => [\&_select_fields, \&_select],
471             'calc' => [\&_calc_fields, \&_calc],
472             'fixup' => [undef, \&_fixup],
473             'dedupe' => [undef, \&_dedupe],
474             'rename' => [\&_rename_fields, \&_no_change],
475             'gethashval' => [\&_gethashval_fields, \&_gethashval],
476             'count' => [\&_calc_fields, \&_count],
477             'limit' => [undef, \&_limit],
478             'walk' => [\&_walk_fields, \&_walk],
479             };
480 19     19   40 sub _find_transmogrifier { $_[0]->_find_core_transmogrifier ($_[1]); } # This is where you want to look up specialty transmogrifiers in subclasses.
481             sub _find_core_transmogrifier {
482 19 100   19   205 croak "Unknown transmogrifier '" . $_[1] . "'" unless exists $transmogrifiers->{$_[1]};
483 18         28 @{$transmogrifiers->{$_[1]}};
  18         53  
484             }
485              
486             sub transmogrify {
487 14     14 1 1086 my $self = shift;
488            
489 14         32 my $fieldlist = $self->fields();
490            
491             # Convert the list of transmogrifier specs into a list of coderefs, and calculate the field list at each step.
492 14         24 my @tlist = ();
493 14         30 foreach my $t (@_) {
494 19         52 my ($transmog, @parms) = @$t;
495 19         38 my ($fielder, $coder) = $self->_find_transmogrifier ($transmog);
496 18         55 $coder->($fieldlist, @parms); # Run through one build just to check our input fields for correctness
497 17         160 push @tlist, [$coder, $fieldlist, [@parms]]; # Then save our builders so we can call them afresh on each iteration.
498 17 100       69 $fieldlist = $fielder->($fieldlist, @parms) if defined $fielder;
499             #print STDERR "field list is now " . Dumper($fieldlist);
500             }
501              
502             my $sub = sub {
503 15     15   33 my $in = $self->iter(@_); # Parameters are passed through to source iterator.
504              
505 15         371 my @buffer = ();
506 15         24 my $buffer_which;
507             my $subiterator;
508            
509 15         25 my @reified_tlist = ();
510 15         26 foreach my $t (@tlist) {
511 23         54 my ($coder, $fieldlist, $parms) = @$t;
512 23         53 push @reified_tlist, $coder->($fieldlist, @$parms);
513             }
514             #my @reified_tlist = map { $_->() } @tlist;
515            
516             sub {
517 61         87 SKIP:
518             my $rec = undef;
519 61         84 my $walked = 0; # Is this record one that came from a walker?
520 61 100       124 if (defined $subiterator) {
521 3         7 $rec = $subiterator->();
522 3 100       13 if (not defined $rec) {
523 1         4 $subiterator = undef;
524             } else {
525 2         4 $walked = 1;
526             }
527             }
528            
529 61 100       110 if (not defined $rec) {
530 59 100       110 if (scalar @buffer) {
531 2         3 $rec = shift @buffer;
532 2 100       7 if (ref $rec ne 'ARRAY') {
533 1         3 $subiterator = $rec->iter();
534 1         25 goto SKIP;
535             }
536 1         3 $walked = 1;
537             } else {
538 57         101 $rec = $in->();
539             }
540             }
541 60 100       311 return undef unless defined $rec;
542            
543 46         60 my $which_t = 0;
544 46         78 foreach my $t (@reified_tlist) {
545 70         93 $which_t += 1;
546 70 100 66     145 next if ($walked and $which_t <= $buffer_which); # Skip the transmogrifiers that ran before the stage of this walk-originated record
547 67         861 my @things = $t->($rec);
548 67 100       337 goto SKIP unless defined $things[0]; # This is the shortcut used for "where" functionality.
549 60 50       117 if (ref ($things[0]) eq 'ARRAY') {
550 60         92 $rec = shift @things;
551 60 100       148 if (@things) {
552 2         5 push @buffer, @things;
553 2         4 $buffer_which = $which_t; # Tacit requirement: only one walker per transmogrifier list, with bad error otherwise
554             }
555             } else {
556 0         0 push @buffer, @things;
557 0         0 $buffer_which = $which_t;
558 0         0 goto SKIP;
559             }
560             }
561 39         132 $rec;
562             }
563 12         47 };
  15         98  
564            
565 12         26 my $class = ref($self);
566 12         35 $class->new ($sub, $fieldlist); # 2019-02-23 - use class of source, not "Iterator::Records"
567             }
568              
569             =head1 LOADING AND REPORTING
570              
571             These are some handy utilities for dealing with record streams.
572              
573             =head2 load ([limit]), load_parms(parms...), load_lparms(limit, parms...), load_iter(iterator, [limit])
574              
575             The I function simply loads the stream into an arrayref of arrayrefs. If I is specified, at most that many rows will be loaded; otherwise,
576             the iterator runs as long as it has data.
577              
578             Note that this is called directly on the definition of the stream, not on the resulting iterator. Consequently, I can't be used to "page" through
579             an existing record stream - if you want to do that, you should look at L, which was written specifically to support the buffered reading of
580             record streams and manipulation of the resulting buffers.
581              
582             This form of C can't be used on iterator factories that take parameters. If you have a factory that requires parameters, use C. Finally,
583             to use both a limit and parameters, use C.
584              
585             All of these are just sugar for the core method C, which, given a started iterator and an optional limit, loads it.
586              
587             =cut
588              
589             sub load_iter {
590 19     19 1 388 my ($self, $i, $limit) = @_;
591            
592 19         27 my @returns = ();
593 19         31 my $row;
594 19   100     146 while (((not defined $limit) or (defined $limit and $limit > 0)) and $row = $i->()) {
      100        
595 49 100       145 $limit = $limit - 1 if defined $limit;
596 49         189 push @returns, [@$row];
597             }
598 19         135 \@returns;
599             }
600              
601             sub load {
602 16     16 1 1149 my ($self, $limit) = @_;
603 16         41 $self->load_iter($self->iter(), $limit);
604             }
605             sub load_parms {
606 1     1 1 6 my $self = shift;
607 1         4 $self->load_iter($self->iter(@_));
608             }
609             sub load_lparms {
610 1     1 1 2 my $self = shift;
611 1         3 my $limit = shift;
612 1         3 $self->load_iter($self->iter(@_), $limit);
613             }
614              
615             =head2 report (format, [dedupe list])
616              
617             The I method is another retrieval method; that is, it returns an iterator when called. However, this iterator is not a record stream; instead,
618             it is a string iterator. Each record in the defined stream is passed through sprintf with the format provided. For convenience, if a list of columns
619             is provided, performs a dedupe transmogrification on the incoming records before formatting them.
620              
621             =cut
622              
623             sub report {
624 1     1 1 870 my $self = shift;
625 1         3 my $format = shift;
626 1 50       5 if (scalar @_) {
627 0         0 my $self = $self->dedupe (@_);
628             }
629 1         4 my $iter = $self->iter();
630             Iterator::Simple::Iterator->new(sub {
631 3 100   3   1192 if (my $rec = $iter->()) {
632 2         31 return sprintf ($format, @$rec);
633             } else {
634 1         7 return undef;
635             }
636 1         29 });
637             }
638              
639             =head2 table ([limit]), table_parms(parms...), table_lparms(limit, parms...), table_iter(iterator, [limit])
640              
641             The I functions work just like the I functions, but load the iterator into a L, if that module is installed.
642              
643             =cut
644              
645             sub table_iter {
646 0     0 1   my $self = shift;
647            
648 0           eval "use Data::Org::Table";
649 0 0         croak 'Data::Org::Table is not installed' if (@!);
650              
651 0           Data::Org::Table->new ($self->load_iter(@_), $self->fields, 0);
652             }
653              
654             sub table {
655 0     0 1   my ($self, $limit) = @_;
656 0           $self->table_iter($self->iter(), $limit);
657             }
658             sub table_parms {
659 0     0 1   my $self = shift;
660 0           $self->table_iter($self->iter(@_));
661             }
662             sub table_lparms {
663 0     0 1   my $self = shift;
664 0           my $limit = shift;
665 0           $self->table_iter($self->iter(@_), $limit);
666             }
667              
668             package Iterator::Records::db;
669 5     5   49 use Iterator::Simple;
  5         9  
  5         265  
670 5     5   35 use Carp;
  5         10  
  5         496  
671              
672             our $dbi_ok = 1;
673             our $sqlite_ok = 0;
674 5     5   910 eval "use DBI;";
  0         0  
  0         0  
675             $dbi_ok = 0 if $@;
676             if ($dbi_ok) {
677 5     5   37 use vars qw(@ISA);
  5         26  
  5         4747  
678             @ISA = qw(DBI::db);
679            
680             eval "use DBD::SQLite;";
681             $sqlite_ok = 1 unless $@;
682             }
683              
684             =head2 open ([filename])
685              
686             The C method opens an SQLite database file. Opens an in-memory file if no filename is provided.
687              
688             =cut
689              
690             sub open {
691 0 0   0     croak "DBI is not installed" unless $dbi_ok;
692 0 0         croak "SQLite is not installed" unless $sqlite_ok;
693 0           my $class = shift;
694 0   0       my $file = shift || ':memory:';
695 0           my $dbh = DBI->connect('dbi:SQLite:dbname=' . $file);
696 0           bless($dbh, $class);
697 0           $dbh;
698             }
699              
700             sub open_dbh {
701 0 0   0     croak "DBI is not installed" unless $dbi_ok;
702 0           my $class = shift;
703 0           my $dbh = shift;
704 0           bless ($dbh, $class);
705 0           $dbh;
706             }
707              
708             =head2 connect(...)
709              
710             The C method is just the DBI connect method; we get it via inheritance.
711              
712             =head2 get (query, [parms])
713              
714             The C method takes some SQL, executes it with the parameters passed in (if any), retrieves the first row, and returns
715             the value of the first field of that row.
716              
717             =cut
718              
719             sub get {
720 0     0     my $self = shift;
721 0           my $query = shift;
722 0           my $sth = $self->prepare($query);
723 0           $sth->execute(@_);
724 0           my $row = $sth->fetchrow_arrayref;
725 0           $row->[0];
726             }
727              
728             =head2 select
729              
730             The C
731             In scalar mode, returns the arrayref from C.
732              
733             =cut
734              
735             sub select {
736 0     0     my $self = shift;
737 0           my $query = shift;
738 0 0         return unless defined wantarray;
739 0           my $sth = $self->prepare($query);
740 0           $sth->execute(@_);
741 0           my $ret = $sth->fetchall_arrayref;
742 0 0         return wantarray ? @$ret : $ret;
743             }
744              
745             =head2 select_one
746              
747             The C method runs a query and returns the first row as an arrayref.
748              
749             =cut
750              
751             sub select_one {
752 0     0     my $self = shift;
753 0           my $query = shift;
754 0           my $sth = $self->prepare($query);
755 0           $sth->execute(@_);
756 0           $sth->fetchrow_arrayref;
757             }
758              
759              
760             =head2 iterator (query, [parms), itparms (query, fields)
761              
762             This is the actual reason for putting this into the Iterator::Records namespace - given a query against the database, we return
763             an iterator factory for iterators over the rows of the query. Like C
764             and execute it. It will then ask DBI for the names of the fields in the query and use that information to build an C object
765             that, when iterated, will return the query results. If iterated again, it will run a new query.
766              
767             If you want to have parameterized queries instead, use C, then pass parameters to the factory it creates. In this case, since
768             the query can't be run in advance, you have to provide the field names you expect. (They don't have to match the ones the database will give
769             you, though, in this case.)
770              
771             =cut
772              
773             # Here's the subtle part. We have to execute the query once to get the field names from the DBI driver. So the first time the iterator factory
774             # is called, it should return an iterator over *that instance*. But the next time, it has to create a new one.
775             # 2019-04-23 - turns out SQLite is perfectly capable of returning NAMES after the prepare but before execute - not all drivers can, but SQLite can. So this is largely unnecessary.
776             sub iterator {
777 0     0     my $self = shift;
778 0           my $query = shift;
779 0           my $sth = $self->prepare($query);
780 0           $sth->execute(@_);
781 0           my $names = $sth->{NAME};
782 0           my $first_time = 1;
783             my $factory = sub {
784 0 0   0     if ($first_time) {
785 0           $first_time = 0;
786             } else {
787 0           $sth = $self->prepare($query);
788 0           $sth->execute(@_);
789             }
790             sub {
791 0           $sth->fetchrow_arrayref;
792             }
793 0           };
  0            
794 0           Iterator::Records->new ($factory, $names);
795             }
796              
797             sub itparms {
798 0     0     my $self = shift;
799 0           my $query = shift;
800 0           my $fields = shift;
801 0           my $sth = $self->prepare($query);
802 0 0         $fields = $sth->{NAME} unless defined $fields;
803            
804             my $factory = sub {
805 0     0     $sth->execute(@_);
806             sub {
807 0           $sth->fetchrow_arrayref;
808             }
809 0           };
  0            
810 0           Iterator::Records->new ($factory, $fields);
811             }
812              
813             =head2 insert
814              
815             The C command calls C after the insertion, and returns that value. Just a little shorthand. Since retrieval of the ID for the last
816             row inserted is very database-specific, it may not work for your particular configuration.
817              
818             =cut
819              
820             sub insert {
821 0     0     my $self = shift;
822 0           my $query = shift;
823 0           my $sth = $self->prepare($query);
824 0           $sth->execute(@_);
825 0           $self->last_insert_id('', '', '', '');
826             }
827              
828             =head2 load_table (table, iterator), load_sql (insert query, iterator)
829              
830             For bulk loading, we have single-call methods C and C. The former will build an appropriate insert query for the table in question using the iterator's field list.
831             The second takes an arbitrary insert query, then executes it on each record coming from the iterator. This method can take either an L object, or any coderef or activated
832             iterator that returns arrayrefs; if given the latter it will simply pass them to the execute call.
833              
834             Each returns the number of rows inserted.
835              
836             =cut
837              
838             sub load_table {
839 0     0     my ($self, $table, $iterator) = @_;
840 0           my $fields = $iterator->fields();
841 0           my @inserts = map { '?' } @$fields;
  0            
842 0           my $sql = "insert into $table values (" . join (', ', @inserts) . ')';
843 0           $self->load_sql ($sql, $iterator->iter()); # TODO: an error in our SQL will show this line. Do better.
844             }
845              
846             sub load_sql {
847 0     0     my ($self, $query, $iterator) = @_;
848 0 0         croak "Source for bulk load not iterable" unless Iterator::Simple::is_iterable($iterator);
849 0           my $iter = Iterator::Simple::iter($iterator);
850 0           my $sth = $self->prepare($query);
851 0           my $count = 0;
852 0           while (my $rec = $iter->()) {
853 0           $count += 1;
854 0           $sth->execute(@$rec);
855             }
856 0           $count;
857             }
858              
859             =head2 do
860              
861             The C command works a little differently from the standard API; DBI's version wants a hashref of attributes that I never use
862             and regularly screw up.
863              
864             =cut
865              
866             sub do {
867 0     0     my $self = shift;
868 0           my $query = shift;
869 0           my $sth = $self->prepare($query);
870 0           $sth->execute(@_);
871             }
872              
873             package Iterator::Records::st;
874 5     5   66 use vars qw(@ISA);
  5         12  
  5         425  
875             @ISA = qw(DBI::st);
876              
877             # We don't actually have anything to override in the statement, but it has to be defined or the DBI machinery won't work.
878              
879              
880             =head1 AUTHOR
881              
882             Michael Roberts, C<< >>
883              
884             =head1 BUGS
885              
886             Please report any bugs or feature requests to C, or through
887             the web interface at L. I will be notified, and then you'll
888             automatically be notified of progress on your bug as I make changes.
889              
890              
891              
892              
893             =head1 SUPPORT
894              
895             You can find documentation for this module with the perldoc command.
896              
897             perldoc Iterator::Records
898              
899              
900             You can also look for information at:
901              
902             =over 4
903              
904             =item * RT: CPAN's request tracker (report bugs here)
905              
906             L
907              
908             =item * AnnoCPAN: Annotated CPAN documentation
909              
910             L
911              
912             =item * CPAN Ratings
913              
914             L
915              
916             =item * Search CPAN
917              
918             L
919              
920             =back
921              
922              
923             =head1 ACKNOWLEDGEMENTS
924              
925              
926             =head1 LICENSE AND COPYRIGHT
927              
928             Copyright 2015 Michael Roberts.
929              
930             This program is free software; you can redistribute it and/or modify it
931             under the terms of the the Artistic License (2.0). You may obtain a
932             copy of the full license at:
933              
934             L
935              
936             Any use, modification, and distribution of the Standard or Modified
937             Versions is governed by this Artistic License. By using, modifying or
938             distributing the Package, you accept this license. Do not use, modify,
939             or distribute the Package, if you do not accept this license.
940              
941             If your Modified Version has been derived from a Modified Version made
942             by someone other than you, you are nevertheless required to ensure that
943             your Modified Version complies with the requirements of this license.
944              
945             This license does not grant you the right to use any trademark, service
946             mark, tradename, or logo of the Copyright Holder.
947              
948             This license includes the non-exclusive, worldwide, free-of-charge
949             patent license to make, have made, use, offer to sell, sell, import and
950             otherwise transfer the Package with respect to any patent claims
951             licensable by the Copyright Holder that are necessarily infringed by the
952             Package. If you institute patent litigation (including a cross-claim or
953             counterclaim) against any party alleging that the Package constitutes
954             direct or contributory patent infringement, then this Artistic License
955             to you shall terminate on the date that such litigation is filed.
956              
957             Disclaimer of Warranty: THE PACKAGE IS PROVIDED BY THE COPYRIGHT HOLDER
958             AND CONTRIBUTORS "AS IS' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES.
959             THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
960             PURPOSE, OR NON-INFRINGEMENT ARE DISCLAIMED TO THE EXTENT PERMITTED BY
961             YOUR LOCAL LAW. UNLESS REQUIRED BY LAW, NO COPYRIGHT HOLDER OR
962             CONTRIBUTOR WILL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, OR
963             CONSEQUENTIAL DAMAGES ARISING IN ANY WAY OUT OF THE USE OF THE PACKAGE,
964             EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
965              
966              
967             =cut
968              
969             1; # End of Iterator::Records