line
stmt
bran
cond
sub
pod
time
code
1
package Iterator::Records;
2
3
5
5
330199
use 5.006;
5
56
4
5
5
30
use strict;
5
7
5
101
5
5
5
22
use warnings;
5
10
5
134
6
5
5
25
use Carp;
5
8
5
314
7
5
5
2561
use Iterator::Simple;
5
24709
5
377
8
#use Data::Dumper;
9
10
=head1 NAME
11
12
Iterator::Records - a simple iterator for arrayref record sources
13
14
=head1 VERSION
15
16
Version 0.02
17
18
=cut
19
20
our $VERSION = '0.02';
21
22
23
=head1 SYNOPSIS
24
25
Iterator::Records uses L to work with iterators whose values are arrayrefs of named fields. These can be called I.
26
A record stream can be seen as the same thing as a DBI retrieval, but without most of the machinery for DBI - and of course, a DBI query is one of the ways you
27
can build a record stream.
28
29
The actual API of Iterator::Records isn't as simple or elegant as L, simply because there's more to keep track of. But the basic
30
approach is similar: an Iterator::Records object defines how to iterate something, then you use the iter() method to create an iterator from it.
31
The result is an Iterator::Simple iterator known to return records, i.e. arrayrefs of fields that match the field list specified.
32
33
Note that the Iterator::Records object is an iterator *factory*, and the actual iterator itself is returned by the call to iter().
34
35
use Iterator::Records;
36
37
my $spec = Iterator::Records->new (, ['field 1', 'field 2']);
38
39
my $iterator = $spec->iter();
40
while (my $row = $iterator->()) {
41
my ($field1, $field2) = @$row;
42
}
43
44
$iterator = $spec->iter_hash();
45
while (my $row = $iterator->()) {
46
print $row->{field 1};
47
}
48
49
my ($f1, $f2);
50
$iterator = $spec->iter_bind(\$f1, \$f2);
51
while ($iterator->()) {
52
print "$f1 - $f2\n";
53
}
54
55
Note that the iterator itself is just an L iterator. Now hold on, though, because here's where things get interesting.
56
57
my $recsource = Iterator::Records->new (sub { ... }, ['field 1', 'field 2']);
58
my $iterator = $recsource->select ("field 1")->iter;
59
while (my $row = $iterator->()) {
60
my ($field1) = @$row;
61
}
62
63
my @fields = $recsource->fields();
64
my $fields = $recsource->fields(); # Returns an arrayref in scalar context.
65
66
$rs = $recsource->where (sub { ... }, "field 1", "field 2");
67
$rs = $recsource->fixup ("field 1", sub { ... } );
68
$rs = $recsource->calc ("field 3", sub { ... } );
69
$rs = $rs->select ("field 2", "field 3", "field 1");
70
$rs = $rs->select (["field 2", "field 3", "field 1"]);
71
72
$rs = $recsource->transform (["where", ["field 1", "=", "x"]],
73
["fixup", ["field 1, sub { ... }]]);
74
75
76
Since Iterator::Records is essentially a more generalized way of iterating DBI results, there are a few wrappers to make things easy.
77
78
my $dbh = Iterator::Records::db->connect(--DBI syntax--);
79
my $dbh = Iterator::Records::db->open('sqlite file');
80
my $dbh = Iterator::Records::db->open(); # Defaults to an in-memory SQLite database
81
82
This is not the direct DBI handle; it's got simplified syntax as follows:
83
84
my $value = $dbh->get ('select value from table where id=?', $id); # Single value retrieval in one whack.
85
$dbh->do ("insert ..."); # Regular insertion, just like in DBI, except simpler.
86
my $record = $dbh->insert ("insert ..."); # Calls last_insert_id ('', '', '', ''), which will likely fail except with SQLite.
87
88
And then you have the actual iterator machinery.
89
90
my $iter = $dbh->iterator ('select * from table')->iter();
91
my $sth = $dbh->prepare (--DBI syntax--);
92
my $iter = $sth->iter ($value1, $value2);
93
while ($iter->()) {
94
my ($field1, $field2) = @$_;
95
}
96
97
We can load an iterator into a table. If you have Data::Tab installed, it will make a Data::Tab with the column names from this iterator.
98
Otherwise, it will simply return an arrayref of arrayrefs by calling Iterator::Simple's "list" method.
99
100
my $data = $recsource->table;
101
102
The "report" method returns an Iterator::Simple that applies an sprintf to each value in the record source. If you supply a list of fields to dedupe it will replace them with ""
103
if their value is the same as the previous row. This is useful for tabulated data where, for instance, the date may be the same from line to line and if so should only be
104
displayed once.
105
106
my $report = $recsource->report ("%-20s %s", ["field 1"]); # Here, field 2 would not be deduped.
107
my $report = join ('\n', $recsource->report (...)->list);
108
109
=cut
110
111
5
5
96
use Iterator::Records;
5
18
5
14307
112
113
=head1 BASIC ITERATION
114
115
=head2 new (iterable, arrayref of fields)
116
117
To specify an Iterator::Records from scratch, just take whatever iterable thing you have, and specify a list of fields in the resulting records.
118
If the iterable is anything but a coderef, Iterator::Records->iter will simply pass it straight to Iterator::Simple for iteration. If it's a coderef,
119
it will be called, and its return value will be passed to Iterator::Simple. This allows record streams to be reused.
120
121
As an added bonus of this extra level of indirection, you can call "iter" with parameters that will be passed on to the coderef. This turns the
122
Iterator::Records object into a parameterizable iterator factory.
123
124
=cut
125
126
sub new {
127
24
24
1
1872
my ($class, $iterable, $fields) = @_;
128
24
54
my $self = bless ({}, $class);
129
24
50
73
croak "Iterator spec not iterable" unless Iterator::Simple::is_iterable($iterable);
130
24
254
$self->{gen} = $iterable;
131
24
48
$self->{f} = $fields;
132
24
50
$self->{id} = '*';
133
24
120
$self;
134
}
135
136
21
21
0
83
sub fields { $_[0]->{f}; }
137
sub id {
138
0
0
0
0
my $self = shift;
139
0
0
0
if (scalar @_) {
140
0
0
$self->{id} = shift;
141
}
142
0
0
$self->{id};
143
}
144
145
=head2 iter, iter_hash, iter_bind
146
147
Basic iteration of the record source returns an arrayref for each record. Alternatively, an iterator can be created which returns a hashref for each
148
record, with the field names keying the return values in each record. This is less efficient, but it's often handy. The third option is to bind a list
149
of scalar references that will be written automagically on each retrieval. The return value in this case is still the original arrayref record.
150
151
=cut
152
153
sub iter {
154
45
45
1
4426
my $self = shift;
155
45
100
131
if (ref $self->{gen} eq 'CODE') {
156
20
52
return Iterator::Simple::iter($self->{gen}->(@_));
157
} else {
158
25
72
return Iterator::Simple::iter($self->{gen});
159
}
160
}
161
sub iter_hash {
162
2
2
1
1708
my $self = shift;
163
2
6
my $iter = $self->iter();
164
Iterator::Simple::Iterator->new(sub {
165
6
100
6
35
if (my $rec = $iter->()) {
166
4
26
my $ret = {};
167
4
6
my $i = 0;
168
4
8
foreach my $f (@{$self->{f}}) {
4
9
169
8
20
$ret->{$f} = $rec->[$i++];
170
}
171
4
28
return $ret;
172
} else {
173
2
12
return undef;
174
}
175
2
58
});
176
}
177
sub iter_bind {
178
2
2
1
580
my $self = shift;
179
2
6
my $iter = $self->iter();
180
2
53
my @fields = (@_);
181
Iterator::Simple::Iterator->new(sub {
182
6
100
6
4413
if (my $rec = $iter->()) {
183
4
25
my $i = 0;
184
4
9
foreach my $f (@fields) {
185
8
16
$$f = $rec->[$i++];
186
}
187
4
11
return $rec;
188
} else {
189
2
11
return undef;
190
}
191
2
10
});
192
}
193
194
=head1 TRANSMOGRIFIERS
195
196
Since our record stream sources are very often provided by fairly simple drivers (like the filesystem walker in File::Org), it's not at all unusual to find ourselves
197
in a position where we want to modify them on the fly, either filtering out some of the records or modifying the records as they go through. There are four different
198
"transmogrifiers" for record streams: where, select, calc, and fixup. The "where" transmogrifier discards records that don't match a particular pattern; "select"
199
removes columns; "calc" adds a column that is calculated by an arbitrary coderef provided; and "fixup" applies a coderef to the record to modify individual field
200
values.
201
202
Each transmogrifier takes an iterator I, not an iterator - and returns a new specification that can be iterated. The source stream will then be iterated
203
internally.
204
205
=head2 where (sub { ... }, 'field 1', 'field 2')
206
207
Filtration of records is not really any different from igrep - given a record stream, we provide a coderef that tells us to include or not to include. If fields
208
are specified, their values for the record to be examined will be passed to the coderef as its parameters; otherwise the entire record is provided as an arrayref
209
and the coderef can extract values on its own. The list of fields is not affected.
210
211
=head2 select ('field 1', 'field 3')
212
213
Returns a spec for a new iterator that includes only the fields listed, in the order listed.
214
215
=head2 calc ('new field', sub { ... }, 'field 1', 'field 2')
216
217
Returns a spec for a new iterator that includes a new field calculated by the coderef provided; as for "where", if fields are listed they will be passed into the
218
coderef as parameters, but otherwise the entire record will be passed in. The new field will appear at the end of the current field list.
219
220
=head2 fixup (sub { ... })
221
222
Returns a spec for a new iterator in which each record is first visited by the coderef provided. This is just an imap in more record-based form. The field
223
list is unchanged.
224
225
=head2 dedupe ('field 1', 'field 2')
226
227
Keeps track of the last values for field 1 and field 2; if the new value is a duplicate, passes an empty string through instead. Useful for reporting.
228
The field list is unchanged.
229
230
=head2 rename ('field 1', 'new name', [more pairs])
231
232
To rename a field (or more than one), use 'rename'. The record is not changed.
233
234
=head2 transmogrify (['where', ...], ['calc', ...])
235
236
Any sequence of transmogrifiers can be chained together in a single step using the L method.
237
238
=cut
239
240
sub _find_offsets {
241
33
33
51
my $field_list = shift;
242
33
41
my $size = scalar @{$field_list}-1;
33
60
243
33
72
my @output;
244
33
61
foreach my $f (@_) {
245
49
93
my ($index) = grep { $field_list->[$_] eq $f } (0 .. $size);
110
247
246
49
100
187
croak "Unknown field '$f' used in transmogrifier" unless defined $index;
247
48
95
push @output, $index;
248
}
249
32
71
@output;
250
}
251
252
sub where {
253
1
1
1
9
my $self = shift;
254
1
5
$self->transmogrify (['where', @_]);
255
}
256
257
sub _where {
258
6
6
10
my $fields = shift;
259
6
9
my $tester = shift;
260
6
14
my @field_offsets = _find_offsets ($fields, @_);
261
6
12
my $parms = "";
262
6
50
24
$parms = '$rec->[' . join ('], $rec->[', @field_offsets) . ']' if scalar @field_offsets;
263
264
6
13
my $sub = <<"EOF";
265
sub {
266
my \$rec = shift;
267
return \$rec if \$tester->($parms);
268
return undef;
269
}
270
EOF
271
#print STDERR $sub;
272
6
534
eval $sub;
273
}
274
275
sub select {
276
1
1
1
3
my $self = shift;
277
1
5
$self->transmogrify (['select', @_]);
278
}
279
sub _select_fields {
280
3
3
4
shift;
281
3
11
\@_;
282
}
283
sub _select {
284
7
7
13
my $fields = shift;
285
7
16
my @field_offsets = _find_offsets ($fields, @_);
286
7
12
my $parms = "";
287
7
50
32
$parms = '$rec->[' . join ('], $rec->[', @field_offsets) . ']' if scalar @field_offsets;
288
289
7
34
my $sub = <<"EOF";
290
sub {
291
my \$rec = shift;
292
return [$parms];
293
}
294
EOF
295
#print STDERR $sub;
296
7
527
eval $sub;
297
}
298
299
sub calc {
300
1
1
1
4
my $self = shift;
301
1
5
$self->transmogrify (['calc', @_]);
302
}
303
sub _calc_fields {
304
4
4
6
my @fields = @{$_[0]};
4
12
305
4
9
push @fields, $_[2];
306
4
13
\@fields;
307
}
308
sub _calc {
309
8
8
15
my $fields = shift;
310
8
10
my $calcer = shift;
311
8
11
shift; # The name of our output variable
312
8
23
my @field_offsets = _find_offsets ($fields, @_);
313
7
12
my $parms = "";
314
7
100
29
$parms = '$rec->[' . join ('], $rec->[', @field_offsets) . ']' if scalar @field_offsets;
315
316
7
17
my $sub = <<"EOF";
317
sub {
318
my \$rec = shift;
319
return [@\$rec, \$calcer->($parms)];
320
}
321
EOF
322
#print STDERR $sub;
323
7
607
eval $sub;
324
}
325
326
sub fixup {
327
1
1
1
30
my $self = shift;
328
1
9
$self->transmogrify (['fixup', @_]);
329
}
330
sub _fixup {
331
2
2
4
my $fields = shift;
332
2
3
my $calcer = shift;
333
2
7
my @field_offsets = _find_offsets ($fields, @_);
334
2
7
my $output = '$rec->[' . shift(@field_offsets) . ']';
335
2
4
my $parms = "";
336
2
50
10
$parms = '$rec->[' . join ('], $rec->[', @field_offsets) . ']' if scalar @field_offsets;
337
338
2
7
my $sub = <<"EOF";
339
sub {
340
my \$rec = shift;
341
\$rec = [@\$rec];
342
$output = \$calcer->($parms);
343
return \$rec;
344
}
345
EOF
346
#print STDERR $sub;
347
2
211
eval $sub;
348
}
349
350
sub dedupe {
351
2
2
1
7
my $self = shift;
352
2
7
$self->transmogrify (['dedupe', @_]);
353
}
354
sub _dedupe {
355
4
4
6
my $fields = shift;
356
4
11
my ($target) = _find_offsets ($fields, $_[0]);
357
358
4
13
my $value = '$rec->[' . $target . ']';
359
360
4
5
my $last_value = ''; # Closures are magic.
361
362
4
11
my $sub = <<"EOF";
363
sub {
364
my \$rec = shift;
365
my \$val = $value;
366
if (\$val eq \$last_value) {
367
\$rec = [@\$rec];
368
$value = '';
369
} else {
370
\$last_value = \$val;
371
}
372
return \$rec;
373
}
374
EOF
375
#print STDERR $sub;
376
4
451
eval $sub;
377
}
378
379
sub _gethashval_fields {
380
1
1
3
my $fields = shift;
381
1
2
shift; # skip the name of the value bag.
382
1
3
my @fields = (@$fields, @_);
383
1
4
\@fields;
384
}
385
386
sub _gethashval {
387
2
2
3
my $fields = shift;
388
2
6
my ($fieldno) = Iterator::Records::_find_offsets ($fields, shift);
389
2
11
my $vals = "\$rec->[$fieldno]->{" . join ("}, \$rec->[$fieldno]->{", @_) . '}';
390
391
2
6
my $sub = <<"EOF";
392
sub {
393
my \$rec = shift;
394
return [@\$rec, $vals ];
395
}
396
EOF
397
#print STDERR $sub;
398
2
140
eval $sub;
399
}
400
401
sub _count {
402
2
2
4
my $fields = shift;
403
2
3
shift; # The name of our output variable
404
2
3
my $count = shift;
405
2
50
7
$count = 0 unless defined $count;
406
2
3
$count -= 1;
407
sub {
408
4
4
6
my $rec = shift;
409
4
5
$count += 1;
410
4
10
[@$rec, $count];
411
}
412
2
44
}
413
414
sub _limit {
415
3
3
6
my $fields = shift;
416
3
5
my $limit = shift;
417
3
6
my $count = 0;
418
sub {
419
8
100
8
18
return undef unless $count < $limit;
420
4
5
$count += 1;
421
4
9
shift();
422
}
423
3
12
}
424
425
# The walker framework is *really* minimalistic. It basically looks exactly like _calc, but instead of returning just the parent record (to which, unlike calc, it can add fields),
426
# it can return a list of records and/or iterators to take the place of the parent record - which can include the parent record, so it's effectively an add-or-replace.
427
sub _walk_fields {
428
2
2
3
my $fields = shift;
429
2
4
shift; # skip the walker coderef
430
2
4
my $newfields = shift;
431
2
50
5
if (defined $newfields) {
432
2
9
return [@$newfields, @$fields];
433
}
434
0
0
$fields;
435
}
436
sub _walk {
437
4
4
6
my $fields = shift;
438
4
6
my $walker_factory = shift;
439
4
5
my $newfields = shift; # The (optional) arrayref of fields we're going to add
440
4
10
my @field_offsets = _find_offsets ($fields, @_);
441
442
# The walker framework is unusual in that, instead of building a closure here, we are actually given a closure *factory* that will make it for us.
443
# This factory is given the fields for the input record, the new fields expected for the output record, the list of fields it's supposed to use,
444
# and the offsets of those fields in the input record.
445
# It returns a closure that takes the input record and returns a list of records and/or new iterators that ->transmogrify will buffer and return as appropriate.
446
4
15
$walker_factory->($fields, $newfields, \@_, \@field_offsets);
447
}
448
449
sub rename {
450
0
0
1
0
my $self = shift;
451
0
0
$self->transmogrify (['rename', @_]);
452
}
453
sub _rename_fields {
454
1
1
3
my $fields = shift;
455
1
3
my @fields = @$fields;
456
1
6
while (scalar @_) {
457
2
4
my $from = shift;
458
2
4
my $to = shift;
459
2
50
5
last unless defined $to;
460
2
100
4
@fields = map { $_ eq $from ? $to : $_ } @fields;
6
19
461
}
462
1
4
\@fields;
463
}
464
sub _no_change {
465
4
4
6
sub { $_[0] }; # passes the record through as efficiently as possible
3
3
10
466
}
467
468
our $transmogrifiers = {
469
'where' => [undef, \&_where],
470
'select' => [\&_select_fields, \&_select],
471
'calc' => [\&_calc_fields, \&_calc],
472
'fixup' => [undef, \&_fixup],
473
'dedupe' => [undef, \&_dedupe],
474
'rename' => [\&_rename_fields, \&_no_change],
475
'gethashval' => [\&_gethashval_fields, \&_gethashval],
476
'count' => [\&_calc_fields, \&_count],
477
'limit' => [undef, \&_limit],
478
'walk' => [\&_walk_fields, \&_walk],
479
};
480
19
19
40
sub _find_transmogrifier { $_[0]->_find_core_transmogrifier ($_[1]); } # This is where you want to look up specialty transmogrifiers in subclasses.
481
sub _find_core_transmogrifier {
482
19
100
19
205
croak "Unknown transmogrifier '" . $_[1] . "'" unless exists $transmogrifiers->{$_[1]};
483
18
28
@{$transmogrifiers->{$_[1]}};
18
53
484
}
485
486
sub transmogrify {
487
14
14
1
1086
my $self = shift;
488
489
14
32
my $fieldlist = $self->fields();
490
491
# Convert the list of transmogrifier specs into a list of coderefs, and calculate the field list at each step.
492
14
24
my @tlist = ();
493
14
30
foreach my $t (@_) {
494
19
52
my ($transmog, @parms) = @$t;
495
19
38
my ($fielder, $coder) = $self->_find_transmogrifier ($transmog);
496
18
55
$coder->($fieldlist, @parms); # Run through one build just to check our input fields for correctness
497
17
160
push @tlist, [$coder, $fieldlist, [@parms]]; # Then save our builders so we can call them afresh on each iteration.
498
17
100
69
$fieldlist = $fielder->($fieldlist, @parms) if defined $fielder;
499
#print STDERR "field list is now " . Dumper($fieldlist);
500
}
501
502
my $sub = sub {
503
15
15
33
my $in = $self->iter(@_); # Parameters are passed through to source iterator.
504
505
15
371
my @buffer = ();
506
15
24
my $buffer_which;
507
my $subiterator;
508
509
15
25
my @reified_tlist = ();
510
15
26
foreach my $t (@tlist) {
511
23
54
my ($coder, $fieldlist, $parms) = @$t;
512
23
53
push @reified_tlist, $coder->($fieldlist, @$parms);
513
}
514
#my @reified_tlist = map { $_->() } @tlist;
515
516
sub {
517
61
87
SKIP:
518
my $rec = undef;
519
61
84
my $walked = 0; # Is this record one that came from a walker?
520
61
100
124
if (defined $subiterator) {
521
3
7
$rec = $subiterator->();
522
3
100
13
if (not defined $rec) {
523
1
4
$subiterator = undef;
524
} else {
525
2
4
$walked = 1;
526
}
527
}
528
529
61
100
110
if (not defined $rec) {
530
59
100
110
if (scalar @buffer) {
531
2
3
$rec = shift @buffer;
532
2
100
7
if (ref $rec ne 'ARRAY') {
533
1
3
$subiterator = $rec->iter();
534
1
25
goto SKIP;
535
}
536
1
3
$walked = 1;
537
} else {
538
57
101
$rec = $in->();
539
}
540
}
541
60
100
311
return undef unless defined $rec;
542
543
46
60
my $which_t = 0;
544
46
78
foreach my $t (@reified_tlist) {
545
70
93
$which_t += 1;
546
70
100
66
145
next if ($walked and $which_t <= $buffer_which); # Skip the transmogrifiers that ran before the stage of this walk-originated record
547
67
861
my @things = $t->($rec);
548
67
100
337
goto SKIP unless defined $things[0]; # This is the shortcut used for "where" functionality.
549
60
50
117
if (ref ($things[0]) eq 'ARRAY') {
550
60
92
$rec = shift @things;
551
60
100
148
if (@things) {
552
2
5
push @buffer, @things;
553
2
4
$buffer_which = $which_t; # Tacit requirement: only one walker per transmogrifier list, with bad error otherwise
554
}
555
} else {
556
0
0
push @buffer, @things;
557
0
0
$buffer_which = $which_t;
558
0
0
goto SKIP;
559
}
560
}
561
39
132
$rec;
562
}
563
12
47
};
15
98
564
565
12
26
my $class = ref($self);
566
12
35
$class->new ($sub, $fieldlist); # 2019-02-23 - use class of source, not "Iterator::Records"
567
}
568
569
=head1 LOADING AND REPORTING
570
571
These are some handy utilities for dealing with record streams.
572
573
=head2 load ([limit]), load_parms(parms...), load_lparms(limit, parms...), load_iter(iterator, [limit])
574
575
The I function simply loads the stream into an arrayref of arrayrefs. If I is specified, at most that many rows will be loaded; otherwise,
576
the iterator runs as long as it has data.
577
578
Note that this is called directly on the definition of the stream, not on the resulting iterator. Consequently, I can't be used to "page" through
579
an existing record stream - if you want to do that, you should look at L, which was written specifically to support the buffered reading of
580
record streams and manipulation of the resulting buffers.
581
582
This form of C can't be used on iterator factories that take parameters. If you have a factory that requires parameters, use C. Finally,
583
to use both a limit and parameters, use C.
584
585
All of these are just sugar for the core method C, which, given a started iterator and an optional limit, loads it.
586
587
=cut
588
589
sub load_iter {
590
19
19
1
388
my ($self, $i, $limit) = @_;
591
592
19
27
my @returns = ();
593
19
31
my $row;
594
19
100
146
while (((not defined $limit) or (defined $limit and $limit > 0)) and $row = $i->()) {
100
595
49
100
145
$limit = $limit - 1 if defined $limit;
596
49
189
push @returns, [@$row];
597
}
598
19
135
\@returns;
599
}
600
601
sub load {
602
16
16
1
1149
my ($self, $limit) = @_;
603
16
41
$self->load_iter($self->iter(), $limit);
604
}
605
sub load_parms {
606
1
1
1
6
my $self = shift;
607
1
4
$self->load_iter($self->iter(@_));
608
}
609
sub load_lparms {
610
1
1
1
2
my $self = shift;
611
1
3
my $limit = shift;
612
1
3
$self->load_iter($self->iter(@_), $limit);
613
}
614
615
=head2 report (format, [dedupe list])
616
617
The I method is another retrieval method; that is, it returns an iterator when called. However, this iterator is not a record stream; instead,
618
it is a string iterator. Each record in the defined stream is passed through sprintf with the format provided. For convenience, if a list of columns
619
is provided, performs a dedupe transmogrification on the incoming records before formatting them.
620
621
=cut
622
623
sub report {
624
1
1
1
870
my $self = shift;
625
1
3
my $format = shift;
626
1
50
5
if (scalar @_) {
627
0
0
my $self = $self->dedupe (@_);
628
}
629
1
4
my $iter = $self->iter();
630
Iterator::Simple::Iterator->new(sub {
631
3
100
3
1192
if (my $rec = $iter->()) {
632
2
31
return sprintf ($format, @$rec);
633
} else {
634
1
7
return undef;
635
}
636
1
29
});
637
}
638
639
=head2 table ([limit]), table_parms(parms...), table_lparms(limit, parms...), table_iter(iterator, [limit])
640
641
The I functions work just like the I functions, but load the iterator into a L, if that module is installed.
642
643
=cut
644
645
sub table_iter {
646
0
0
1
my $self = shift;
647
648
0
eval "use Data::Org::Table";
649
0
0
croak 'Data::Org::Table is not installed' if (@!);
650
651
0
Data::Org::Table->new ($self->load_iter(@_), $self->fields, 0);
652
}
653
654
sub table {
655
0
0
1
my ($self, $limit) = @_;
656
0
$self->table_iter($self->iter(), $limit);
657
}
658
sub table_parms {
659
0
0
1
my $self = shift;
660
0
$self->table_iter($self->iter(@_));
661
}
662
sub table_lparms {
663
0
0
1
my $self = shift;
664
0
my $limit = shift;
665
0
$self->table_iter($self->iter(@_), $limit);
666
}
667
668
package Iterator::Records::db;
669
5
5
49
use Iterator::Simple;
5
9
5
265
670
5
5
35
use Carp;
5
10
5
496
671
672
our $dbi_ok = 1;
673
our $sqlite_ok = 0;
674
5
5
910
eval "use DBI;";
0
0
0
0
675
$dbi_ok = 0 if $@;
676
if ($dbi_ok) {
677
5
5
37
use vars qw(@ISA);
5
26
5
4747
678
@ISA = qw(DBI::db);
679
680
eval "use DBD::SQLite;";
681
$sqlite_ok = 1 unless $@;
682
}
683
684
=head2 open ([filename])
685
686
The C method opens an SQLite database file. Opens an in-memory file if no filename is provided.
687
688
=cut
689
690
sub open {
691
0
0
0
croak "DBI is not installed" unless $dbi_ok;
692
0
0
croak "SQLite is not installed" unless $sqlite_ok;
693
0
my $class = shift;
694
0
0
my $file = shift || ':memory:';
695
0
my $dbh = DBI->connect('dbi:SQLite:dbname=' . $file);
696
0
bless($dbh, $class);
697
0
$dbh;
698
}
699
700
sub open_dbh {
701
0
0
0
croak "DBI is not installed" unless $dbi_ok;
702
0
my $class = shift;
703
0
my $dbh = shift;
704
0
bless ($dbh, $class);
705
0
$dbh;
706
}
707
708
=head2 connect(...)
709
710
The C method is just the DBI connect method; we get it via inheritance.
711
712
=head2 get (query, [parms])
713
714
The C method takes some SQL, executes it with the parameters passed in (if any), retrieves the first row, and returns
715
the value of the first field of that row.
716
717
=cut
718
719
sub get {
720
0
0
my $self = shift;
721
0
my $query = shift;
722
0
my $sth = $self->prepare($query);
723
0
$sth->execute(@_);
724
0
my $row = $sth->fetchrow_arrayref;
725
0
$row->[0];
726
}
727
728
=head2 select
729
730
The C method retrieves an array of arrayrefs for the rows returned from the query.
731
In scalar mode, returns the arrayref from C.
732
733
=cut
734
735
sub select {
736
0
0
my $self = shift;
737
0
my $query = shift;
738
0
0
return unless defined wantarray;
739
0
my $sth = $self->prepare($query);
740
0
$sth->execute(@_);
741
0
my $ret = $sth->fetchall_arrayref;
742
0
0
return wantarray ? @$ret : $ret;
743
}
744
745
=head2 select_one
746
747
The C method runs a query and returns the first row as an arrayref.
748
749
=cut
750
751
sub select_one {
752
0
0
my $self = shift;
753
0
my $query = shift;
754
0
my $sth = $self->prepare($query);
755
0
$sth->execute(@_);
756
0
$sth->fetchrow_arrayref;
757
}
758
759
760
=head2 iterator (query, [parms), itparms (query, fields)
761
762
This is the actual reason for putting this into the Iterator::Records namespace - given a query against the database, we return
763
an iterator factory for iterators over the rows of the query. Like C, the basic C call will assemble a query
764
and execute it. It will then ask DBI for the names of the fields in the query and use that information to build an C object
765
that, when iterated, will return the query results. If iterated again, it will run a new query.
766
767
If you want to have parameterized queries instead, use C, then pass parameters to the factory it creates. In this case, since
768
the query can't be run in advance, you have to provide the field names you expect. (They don't have to match the ones the database will give
769
you, though, in this case.)
770
771
=cut
772
773
# Here's the subtle part. We have to execute the query once to get the field names from the DBI driver. So the first time the iterator factory
774
# is called, it should return an iterator over *that instance*. But the next time, it has to create a new one.
775
# 2019-04-23 - turns out SQLite is perfectly capable of returning NAMES after the prepare but before execute - not all drivers can, but SQLite can. So this is largely unnecessary.
776
sub iterator {
777
0
0
my $self = shift;
778
0
my $query = shift;
779
0
my $sth = $self->prepare($query);
780
0
$sth->execute(@_);
781
0
my $names = $sth->{NAME};
782
0
my $first_time = 1;
783
my $factory = sub {
784
0
0
0
if ($first_time) {
785
0
$first_time = 0;
786
} else {
787
0
$sth = $self->prepare($query);
788
0
$sth->execute(@_);
789
}
790
sub {
791
0
$sth->fetchrow_arrayref;
792
}
793
0
};
0
794
0
Iterator::Records->new ($factory, $names);
795
}
796
797
sub itparms {
798
0
0
my $self = shift;
799
0
my $query = shift;
800
0
my $fields = shift;
801
0
my $sth = $self->prepare($query);
802
0
0
$fields = $sth->{NAME} unless defined $fields;
803
804
my $factory = sub {
805
0
0
$sth->execute(@_);
806
sub {
807
0
$sth->fetchrow_arrayref;
808
}
809
0
};
0
810
0
Iterator::Records->new ($factory, $fields);
811
}
812
813
=head2 insert
814
815
The C command calls C after the insertion, and returns that value. Just a little shorthand. Since retrieval of the ID for the last
816
row inserted is very database-specific, it may not work for your particular configuration.
817
818
=cut
819
820
sub insert {
821
0
0
my $self = shift;
822
0
my $query = shift;
823
0
my $sth = $self->prepare($query);
824
0
$sth->execute(@_);
825
0
$self->last_insert_id('', '', '', '');
826
}
827
828
=head2 load_table (table, iterator), load_sql (insert query, iterator)
829
830
For bulk loading, we have single-call methods C and C. The former will build an appropriate insert query for the table in question using the iterator's field list.
831
The second takes an arbitrary insert query, then executes it on each record coming from the iterator. This method can take either an L object, or any coderef or activated
832
iterator that returns arrayrefs; if given the latter it will simply pass them to the execute call.
833
834
Each returns the number of rows inserted.
835
836
=cut
837
838
sub load_table {
839
0
0
my ($self, $table, $iterator) = @_;
840
0
my $fields = $iterator->fields();
841
0
my @inserts = map { '?' } @$fields;
0
842
0
my $sql = "insert into $table values (" . join (', ', @inserts) . ')';
843
0
$self->load_sql ($sql, $iterator->iter()); # TODO: an error in our SQL will show this line. Do better.
844
}
845
846
sub load_sql {
847
0
0
my ($self, $query, $iterator) = @_;
848
0
0
croak "Source for bulk load not iterable" unless Iterator::Simple::is_iterable($iterator);
849
0
my $iter = Iterator::Simple::iter($iterator);
850
0
my $sth = $self->prepare($query);
851
0
my $count = 0;
852
0
while (my $rec = $iter->()) {
853
0
$count += 1;
854
0
$sth->execute(@$rec);
855
}
856
0
$count;
857
}
858
859
=head2 do
860
861
The C command works a little differently from the standard API; DBI's version wants a hashref of attributes that I never use
862
and regularly screw up.
863
864
=cut
865
866
sub do {
867
0
0
my $self = shift;
868
0
my $query = shift;
869
0
my $sth = $self->prepare($query);
870
0
$sth->execute(@_);
871
}
872
873
package Iterator::Records::st;
874
5
5
66
use vars qw(@ISA);
5
12
5
425
875
@ISA = qw(DBI::st);
876
877
# We don't actually have anything to override in the statement, but it has to be defined or the DBI machinery won't work.
878
879
880
=head1 AUTHOR
881
882
Michael Roberts, C<< >>
883
884
=head1 BUGS
885
886
Please report any bugs or feature requests to C, or through
887
the web interface at L. I will be notified, and then you'll
888
automatically be notified of progress on your bug as I make changes.
889
890
891
892
893
=head1 SUPPORT
894
895
You can find documentation for this module with the perldoc command.
896
897
perldoc Iterator::Records
898
899
900
You can also look for information at:
901
902
=over 4
903
904
=item * RT: CPAN's request tracker (report bugs here)
905
906
L
907
908
=item * AnnoCPAN: Annotated CPAN documentation
909
910
L
911
912
=item * CPAN Ratings
913
914
L
915
916
=item * Search CPAN
917
918
L
919
920
=back
921
922
923
=head1 ACKNOWLEDGEMENTS
924
925
926
=head1 LICENSE AND COPYRIGHT
927
928
Copyright 2015 Michael Roberts.
929
930
This program is free software; you can redistribute it and/or modify it
931
under the terms of the the Artistic License (2.0). You may obtain a
932
copy of the full license at:
933
934
L
935
936
Any use, modification, and distribution of the Standard or Modified
937
Versions is governed by this Artistic License. By using, modifying or
938
distributing the Package, you accept this license. Do not use, modify,
939
or distribute the Package, if you do not accept this license.
940
941
If your Modified Version has been derived from a Modified Version made
942
by someone other than you, you are nevertheless required to ensure that
943
your Modified Version complies with the requirements of this license.
944
945
This license does not grant you the right to use any trademark, service
946
mark, tradename, or logo of the Copyright Holder.
947
948
This license includes the non-exclusive, worldwide, free-of-charge
949
patent license to make, have made, use, offer to sell, sell, import and
950
otherwise transfer the Package with respect to any patent claims
951
licensable by the Copyright Holder that are necessarily infringed by the
952
Package. If you institute patent litigation (including a cross-claim or
953
counterclaim) against any party alleging that the Package constitutes
954
direct or contributory patent infringement, then this Artistic License
955
to you shall terminate on the date that such litigation is filed.
956
957
Disclaimer of Warranty: THE PACKAGE IS PROVIDED BY THE COPYRIGHT HOLDER
958
AND CONTRIBUTORS "AS IS' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES.
959
THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
960
PURPOSE, OR NON-INFRINGEMENT ARE DISCLAIMED TO THE EXTENT PERMITTED BY
961
YOUR LOCAL LAW. UNLESS REQUIRED BY LAW, NO COPYRIGHT HOLDER OR
962
CONTRIBUTOR WILL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, OR
963
CONSEQUENTIAL DAMAGES ARISING IN ANY WAY OUT OF THE USE OF THE PACKAGE,
964
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
965
966
967
=cut
968
969
1; # End of Iterator::Records