blib/lib/PDL/IO/FastRaw.pm

filter: Statementstmt 90.0% 100 / 111 Branchbran 65.0% 26 / 40 Conditioncond 54.1% 13 / 24 MC/DCmcdc 23.0% 3 / 13 Subroutinesub 85.7% 12 / 14 Podpod 37.5% 3 / 8 totaltotal 74.7% 157 / 210 CC 31 SCAR 21.4
CC
31
Cov
79%
gluefraw
18.1
mapfraw
9.1
_read_frawhdr
5.4
CRAP 8.5
« blib/lib/PDL/Parallel/threads.pm ↑ Summary blib/lib/PDL/Image2D.pm »
1 =head1 NAME
2  
3 PDL::IO::FastRaw -- A simple, fast and convenient io format for PerlDL.
4  
5 =head1 SYNOPSIS
6  
7 use PDL;
8 use PDL::IO::FastRaw;
9  
10 writefraw($pdl,"fname"); # write a raw file
11  
12 $pdl2 = readfraw("fname"); # read a raw file
13 $pdl2 = PDL->readfraw("fname");
14  
15 gluefraw($pdlx, "fname"); # append to existing file
16 $pdlx->gluefraw("fname");
17  
18 $pdl3 = mapfraw("fname2",{ReadOnly => 1}); # mmap a file, don't read yet
19  
20 $pdl4 = maptextfraw("fname3",{...}); # map a text file into a 1-D pdl.
21  
22  
23 =head1 DESCRIPTION
24  
25 This is a very simple and fast io format for PerlDL.
26 The disk data consists of two files, a header metadata file
27 in ASCII and a binary file consisting simply of consecutive
28 bytes, shorts or whatever.
29  
30 It is hoped that this will not only make for a simple PerlDL module
31 for saving and retrieving these files but also make it easy
32 for other programs to use these files.
33  
34 The format of the ASCII header is simply
35  
36 <typeid>
37 <ndims>
38 <dim0> <dim1> ...
39  
40 You should probably stick with the default header name. You may want
41 to specify your own header, however, such as when you have a large
42 collection of data files with identical dimensions and data types.
43 Under these circumstances, simply specify the C<Header> option in the
44 options hash.
45  
46 The binary files are in general
47 NOT interchangeable between different architectures since the binary
48 file is simply dumped from the memory region of the ndarray.
49 This is what makes the approach efficient.
50  
51 It is also possible to mmap the file which can give a large
52 speedup in certain situations as well as save a lot of memory
53 by using a disk file as virtual memory. When a file is mapped,
54 parts of it are read only as they are accessed in the memory
55 (or as the kernel decides: if you are reading the pages in order,
56 it may well preread some for you).
57  
58 Note that memory savings and copy-on-write are operating-system
59 dependent - see Core.xs and your operating system documentation
60 for exact semantics of whatever. Basically, if you write to a
61 mmapped file without C<ReadOnly>, the change will be reflected
62 in the file immediately. C<ReadOnly> doesn't really make it impossible
63 to write to the ndarray but maps the memory privately so the file
64 will not be changed when you change the ndarray. Be aware though
65 that mmapping a 40Mb file without C<ReadOnly> spends no virtual
66 memory but with C<ReadOnly> it does reserve 40Mb.
67  
68 =head2 Example: Converting ASCII to raw
69  
70 You have a whole slew of data files in ASCII from an experiment
71 that you ran in your lab. You're still tweaking the analysis
72 and plots, so you'd like if your data could load as fast as
73 possible. Eventually you'll read the data into your scripts
74 using C<readfraw>, but the first thing you might do is create
75 a script that converts all the data files to raw files:
76  
77 #!/usr/bin/perl
78 # Assumes that the data files end with a .asc or .dat extension
79 # and saves the raw file output with a .bdat extension.
80 # call with
81 # >./convert_to_raw.pl file1.dat file2.dat ...
82 # or
83 # >./convert_to_raw.pl *.dat
84
85 use PDL;
86 use PDL::IO::FastRaw; # for saving raw files
87 use PDL::IO::Misc; # for reading ASCII files with rcols
88 while(shift) { # run through the entire supplied list of file names
89 ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
90 print "Saving contents of $_ to $newName\n";
91 $data = rcols($_);
92 writefraw($data, $newName);
93 }
94  
95  
96 =head2 Example: readfraw
97  
98 Now that you've got your data into a raw file format, you can
99 start working on your analysis scripts. If you scripts used C<rcols>
100 in the past, the reading portion of the script should go much,
101 much faster now:
102  
103 #!/usr/bin/perl
104 # My plotting script.
105 # Assume I've specified the files to plot on the command line like
106 # >./plot_script.pl file1.bdat file2.bdat ...
107 # or
108 # >./plot_script.pl *.bdat
109
110 use PDL;
111 use PDL::IO::FastRaw;
112 while(shift) { # run through the entire supplied list of file names
113 $data = readfraw($_);
114 my_plot_func($data);
115 }
116  
117 =head2 Example: Custom headers
118  
119 In the first example, I allow C<writefraw> to use the standard header
120 file name, which would be C<file.bdat.hdr>. However, I often measure
121 time series that have identical length, so all of those header files
122 are redundant. To fix that, I simply pass the Header option to the
123 C<writefraw> command. A modified script would look like this:
124  
125 #!/usr/bin/perl
126 # Assumes that the data files end with a .asc or .dat extension
127 # and saves the raw file output with a .bdat extension.
128 # call with
129 # >./convert_to_raw.pl [-hHeaderFile] <fileglob> [-hHeaderFile] <fileglob> ...
130
131 use PDL;
132 use PDL::IO::FastRaw; # for saving raw files
133 use PDL::IO::Misc; # for reading ASCII files with rcols
134 my $header_file = undef;
135 CL_OPTION: while($_ = shift @ARGV) { # run through the entire list of command-line options
136 if(/-h(.*)/) {
137 $header_file = $1;
138 next CL_OPTION;
139 }
140 ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
141 print "Saving contents of $_ to $newName\n";
142 $data = rcols($_);
143 writefraw($data, $newName, {Header => $header_file});
144 }
145  
146 Modifying the read script is left as an exercise for the reader. :]
147  
148  
149 =head2 Example: Using mapfraw
150  
151 Sometimes you'll want to use C<mapfraw> rather than the read/write
152 functions. In fact, the original author of the module doesn't
153 use the read/write functions anymore, prefering to always use
154 C<mapfraw>. How would you go about doing this?
155  
156 Assuming you've already saved your data into the raw format, the
157 only change you would have to make to the script in example 2 would
158 be to change the call to C<readfraw> to C<mapfraw>. That's it.
159 You will probably see differences in performance, though I (David
160 Mertens) couldn't tell you about them because I haven't played
161 around with C<mapfraw> much myself.
162  
163 What if you eschew the use of C<writefraw> and prefer to only use
164 C<mapfraw>? How would you save your data to a raw format? In that
165 case, you would have to create a C<mapfraw> ndarray with the correct
166 dimensions first using
167  
168 $ndarray_on_hd = mapfraw('fname', {Creat => 1, Dims => [dim1, dim2, ...]});
169  
170 Note that you must specify the dimensions and you must tell
171 C<mapfraw> to create the new ndarray for you by setting the
172 C<Creat> option to a true value, not C<Create> (note the missing
173 final 'e').
174  
175 =head1 FUNCTIONS
176  
177 =cut
178  
179 package PDL::IO::FastRaw;
180 1 ▶use strict;
Subroutine BEGIN: called CC=1 SCAR=0.0
181 1 ▶use warnings;
Subroutine BEGIN: called CC=1 SCAR=0.0
182  
183 1 our $VERSION = '0.000003';
184 1 $VERSION = eval $VERSION;
185  
186 1 require Exporter;
187 1 ▶use PDL::Core '';
Subroutine BEGIN: called CC=1 SCAR=0.0
188 1 ▶use PDL::Exporter;
Subroutine BEGIN: called CC=1 SCAR=0.0
189  
190 1 our @ISA = qw/PDL::Exporter/;
191 1 our @EXPORT_OK = qw/writefraw readfraw mapfraw maptextfraw gluefraw/;
192 1 our %EXPORT_TAGS = (Func=>\@EXPORT_OK);
193  
194 # Exported functions
195  
196 1 *writefraw = \&PDL::writefraw;
197 1 *gluefraw = \&PDL::gluefraw;
198 5 ▶sub readfraw {PDL->readfraw(@_)}
Subroutine readfraw: called CC=1 SCAR=0.0
Pod documented
199 3 ▶sub mapfraw {PDL->mapfraw(@_)}
Subroutine mapfraw: called CC=1 SCAR=0.0
Pod documented
200 0 ▶sub maptextfraw {PDL->maptextfraw(@_)}
Subroutine maptextfraw: not called CC=1 SCAR=6.9
Pod documented
201  
202 8 ▶sub _read_frawhdr {
Subroutine _read_frawhdr: called CC=5 SCAR=16.8
203 8 my($name,$opts) = @_;
204 8 ▶ my $hname = $opts->{Header} || "$name.hdr";
Condition100%2/2
$opts->{'Header'} || "$name.hdr"
l!l
26
MC/DC100%1/1
$opts->{'Header'} || "$name.hdr"
$opts->{'Header'}
Truth table100%2/2
A
A= $opts->{'Header'}
Aresult
11
00
205 8 ▶ open my $h, '<', $hname
Branch: 1 branch
BranchTrueFalseTotal
open my $h, "<", $hname or barf("Couldn't open '${hname}' for reading: $!") 8 0 8
206 or barf "Couldn't open '$hname' for reading: $!";
207 8 chomp(my $tid = <$h>);
208 8 chomp(my $ndims = <$h>);
209 0 ▶ chomp(my $str = <$h>); if(!defined $str) {barf("Format error in '$hname'");}
Branch: 1 branch
BranchTrueFalseTotal
unless (defined $str) 0 8 8
210 8 my @dims = split ' ',$str;
211 8 ▶ if($#dims != $ndims-1) {
Branch: 1 branch
BranchTrueFalseTotal
if ($#dims != $ndims - 1) 0 8 8
212 0 barf("Format error reading fraw header file '$hname'");
213 }
214 return {
215 8 Type => $tid,
216 Dims => \@dims,
217 NDims => $ndims
218 };
219 }
220  
221 7 ▶sub _writefrawhdr {
Subroutine _writefrawhdr: called CC=5 SCAR=16.2
222 7 my($pdl,$name,$opts) = @_;
223 7 ▶ my $hname = $opts->{Header} || "$name.hdr";
Condition100%2/2
$opts->{'Header'} || "$name.hdr"
l!l
25
MC/DC100%1/1
$opts->{'Header'} || "$name.hdr"
$opts->{'Header'}
Truth table100%2/2
A
A= $opts->{'Header'}
Aresult
11
00
224 7 ▶ open my $h, '>', $hname
Branch: 1 branch
BranchTrueFalseTotal
open my $h, ">", $hname or barf("Couldn't open '${hname}' for writing: $!") 7 0 7
225 or barf "Couldn't open '$hname' for writing: $!";
226 print $h map "$_\n", $pdl->get_datatype,
227 $opts->{NDims} // $pdl->getndims,
228 7 ▶ join(' ', $opts->{Dims} ? @{$opts->{Dims}} : $pdl->dims);
Branch: 1 branch
BranchTrueFalseTotal
$opts->{'Dims'} ? : 2 5 7
Condition66%2/3
$opts->{"NDims"} // $pdl->getndims
l!l&&r!l&&!r
160
MC/DC0%0/2
$opts->{"NDims"} // $pdl->getndims
$opts->{"NDims"} $pdl->getndims
Truth table66%2/3
A // B
A= $opts->{"NDims"}B= $pdl->getndims
ABresult
1X1
011
000
229 }
230  
231 =head2 writefraw
232  
233 =for ref
234  
235 Write a raw format binary file
236  
237 =for usage
238  
239 writefraw($pdl,"fname");
240 writefraw($pdl,"fname", {Header => 'headerfname'});
241  
242 =for options
243  
244 The C<writefraw> command
245 supports the following option:
246  
247 =over 8
248  
249 =item Header
250  
251 Specify the header file name.
252  
253 =back
254  
255 =cut
256  
257 5 ▶sub PDL::writefraw {
Subroutine writefraw: called CC=2 SCAR=7.0
Pod undocumented
258 5 my($pdl,$name,$opts) = @_;
259 5 _writefrawhdr($pdl,$name,$opts);
260 5 ▶ open my $d, '>', $name
Branch: 1 branch
BranchTrueFalseTotal
open my $d, ">", $name or barf("Couldn't open '${name}' for writing: $!") 5 0 5
261 or barf "Couldn't open '$name' for writing: $!";
262 5 binmode $d;
263 5 print $d ${$pdl->get_dataref};
264 }
265  
266 =head2 readfraw
267  
268 =for ref
269  
270 Read a raw format binary file
271  
272 =for usage
273  
274 $pdl2 = readfraw("fname");
275 $pdl2 = PDL->readfraw("fname");
276 $pdl2 = readfraw("fname", {Header => 'headerfname'});
277  
278 =for options
279  
280 The C<readfraw> command
281 supports the following option:
282  
283 =over 8
284  
285 =item Header
286  
287 Specify the header file name.
288  
289 =back
290  
291 =cut
292  
293 5 ▶sub PDL::readfraw {
Subroutine readfraw: called CC=3 SCAR=11.0
Pod undocumented
294 5 my $class = shift;
295 5 my($name,$opts) = @_;
296 5 ▶ open my $d, '<', $name or barf "Couldn't open '$name' for reading: $!";
Branch: 1 branch
BranchTrueFalseTotal
open my $d, "<", $name or barf("Couldn't open '${name}' for reading: $!") 5 0 5
297 5 binmode $d;
298 5 my $hdr = _read_frawhdr($name,$opts);
299 5 my $pdl = $class->zeroes(PDL::Type->new($hdr->{Type}), @{$hdr->{Dims}});
300 5 my $len = length ${$pdl->get_dataref};
301 5 my $index = 0;
302 5 my $data;
303 my $retlen;
304 5 ▶ while (($retlen = sysread $d, $data, $len) != 0) {
Branch: 1 branch
BranchTrueFalseTotal
if (($retlen = sysread($d, $data, $len)) != 0) 5 5 10
305 5 substr(${$pdl->get_dataref},$index,$len) = $data;
306 5 $index += $retlen;
307 5 $len -= $retlen;
308 }
309 5 $pdl->upd_data();
310 5 return $pdl;
311 }
312  
313 =head2 gluefraw
314  
315 =for ref
316  
317 Append a single data item to an existing binary file written by
318 L</writefraw>. Error if dims not compatible with existing data.
319  
320 =for usage
321  
322 gluefraw($file, $pdl[, $opts]);
323  
324 =cut
325  
326 1 ▶sub PDL::gluefraw {
Subroutine gluefraw: called CC=12 SCAR=28.9
Pod undocumented
327 1 my $usage = 'Usage: gluefraw($pdl,"filename"[,$opts])';
328 1 my ($pdl,$name,$opts) = @_;
329 1 ▶ barf $usage if @_ < 2 or @_ > 3 or !UNIVERSAL::isa($pdl, 'PDL') or ref $name;
Branch: 1 branch
BranchTrueFalseTotal
if @_ < 2 or @_ > 3 or not UNIVERSAL::isa($pdl, "PDL") or ref $name 0 1 1
Condition33%3/9
@_ < 2 or @_ > 3 or not UNIVERSAL::isa($pdl, "PDL") or ref $name
l!l&&r!l&&!r
001
@_ < 2 or @_ > 3 or not UNIVERSAL::isa($pdl, "PDL")
l!l&&r!l&&!r
001
@_ < 2 or @_ > 3
l!l&&r!l&&!r
001
MC/DC0%0/4
@_ < 2 or @_ > 3 or not UNIVERSAL::isa($pdl, "PDL") or ref $name
@_ < 2 @_ > 3 not UNIVERSAL::isa($pdl, "PDL") ref $name
Truth table20%1/5
A or B or not(C) or D
A= @_ < 2B= @_ > 3C= not UNIVERSAL::isa($pdl, "PDL")D= ref $name
ABCDresult
1XXX1
01XX1
001X1
00011
00000
330 1 ▶ barf "'$name' must be real filename: $!" if !-f $name;
Branch: 1 branch
BranchTrueFalseTotal
unless -f $name 0 1 1
331 1 ▶ $opts ||= {};
Condition50%1/2
$opts ||= {}
l!l
01
MC/DC0%0/1
$opts ||= {}
$opts
Truth table50%1/2
A
A= $opts
Aresult
11
00
332 1 my $hdr = _read_frawhdr($name,$opts);
333 0 barf "gluefraw: ndarray has type '@{[$pdl->type]}' but file has type '$hdr->{Type}'"
334 1 ▶ if $pdl->type != PDL::Type->new($hdr->{Type});
Branch: 1 branch
BranchTrueFalseTotal
if $pdl->type != "PDL::Type"->new($hdr->{"Type"}) 0 1 1
335 1 ▶ my @dims = ref $hdr->{Dims} ? @{$hdr->{Dims}} : $hdr->{Dims};
Branch: 1 branch
BranchTrueFalseTotal
ref $hdr->{'Dims'} ? : 1 0 1
336 1 ▶ barf "gluefraw: header dims needs at least 2 dims, got (@dims)" if @dims < 2;
Branch: 1 branch
BranchTrueFalseTotal
if @dims < 2 0 1 1
337 1 my @ldims = @dims[0..$#dims-1];
338 0 ▶ barf "gluefraw: incompatible lower dims, ndarray (@{[$pdl->dims]}) vs header (@ldims)"
Branch: 1 branch
BranchTrueFalseTotal
unless PDL::all($pdl->shape == pdl(@ldims)) 0 1 1
339 if !PDL::all($pdl->shape == pdl(@ldims));
340 1 ▶ open my $d, '>>', $name or barf "Couldn't open '$name' for appending: $!";
Branch: 1 branch
BranchTrueFalseTotal
open my $d, ">>", $name or barf("Couldn't open '${name}' for appending: $!") 1 0 1
341 1 binmode $d;
342 1 print $d ${$pdl->get_dataref};
343 1 $dims[-1]++;
344 1 $hdr->{Dims} = \@dims;
345 1 _writefrawhdr($pdl, $name, { %$opts, %$hdr });
346 }
347  
348 =head2 mapfraw
349  
350 =for ref
351  
352 Memory map a raw format binary file (see the module docs also)
353  
354 =for usage
355  
356 $pdl3 = mapfraw("fname2",{ReadOnly => 1});
357  
358 =for options
359  
360 The C<mapfraw> command
361 supports the following options (not all combinations make sense):
362  
363 =over 8
364  
365 =item Dims, Datatype
366  
367 If creating a new file or if you want to specify your own header
368 data for the file, you can give an array reference and a scalar,
369 respectively.
370  
371 =item Creat
372  
373 Create the file. Also writes out a header for the file.
374  
375 =item Trunc
376  
377 Set the file size. Automatically enabled with C<Creat>. NOTE: This also
378 clears the file to all zeroes.
379  
380 =item ReadOnly
381  
382 Disallow writing to the file.
383  
384 =item Header
385  
386 Specify the header file name.
387  
388 =back
389  
390 =cut
391  
392 3 ▶sub PDL::mapfraw {
Subroutine mapfraw: called CC=9 SCAR=22.1
Pod undocumented
393 3 my $class = shift;
394 3 my($name,$opts) = @_;
395 3 my $hdr;
396 3 ▶ if($opts->{Dims}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($opts->{'Dims'}) { } 1 2 3
397 1 ▶ $hdr->{Type} = $opts->{Datatype} // double->enum;
Condition33%1/3
$opts->{"Datatype"} // "double"->enum
l!l&&r!l&&!r
100
MC/DC0%0/2
$opts->{"Datatype"} // "double"->enum
$opts->{"Datatype"} "double"->enum
Truth table33%1/3
A // B
A= $opts->{"Datatype"}B= "double"->enum
ABresult
1X1
011
000
398 1 $hdr->{Dims} = $opts->{Dims};
399 1 $hdr->{NDims} = scalar(@{$opts->{Dims}});
400 } else {
401 2 $hdr = _read_frawhdr($name,$opts);
402 }
403 3 my $s = PDL::Core::howbig($hdr->{Type});
404 3 for(@{$hdr->{Dims}}) {
405 6 $s *= $_;
406 }
407 3 my $pdl = $class->zeroes(PDL::Type->new($hdr->{Type}));
408 $pdl->set_data_by_file_map(
409 $name,
410 $s,
411 1,
412 ($opts->{ReadOnly}?0:1),
413 ($opts->{Creat}?1:0),
414 (0644),
415 3 ▶ ($opts->{Creat} || $opts->{Trunc} ? 1:0)
Branch: 3 branches
BranchTrueFalseTotal
$opts->{'ReadOnly'} ? : 0 3 3
$opts->{'Creat'} ? : 1 2 3
$opts->{'Creat'} || $opts->{'Trunc'} ? : 1 2 3
Condition66%2/3
$opts->{"Creat"} or $opts->{"Trunc"}
l!l&&r!l&&!r
102
MC/DC50%1/2
$opts->{"Creat"} or $opts->{"Trunc"}
$opts->{"Creat"} $opts->{"Trunc"}
Truth table66%2/3
A or B
A= $opts->{"Creat"}B= $opts->{"Trunc"}
ABresult
1X1
011
000
416 );
417 3 $pdl->setdims($hdr->{Dims});
418 3 $pdl->set_donttouchdata;
419 3 ▶ _writefrawhdr($pdl,$name,$opts) if $opts->{Creat};
Branch: 1 branch
BranchTrueFalseTotal
if $opts->{"Creat"} 1 2 3
420 3 $pdl;
421 }
422  
423 =head2 maptextfraw
424  
425 =for ref
426  
427 Memory map a text file (see the module docs also).
428  
429 Note that this function maps the raw format so if you are
430 using an operating system which does strange things to e.g.
431 line delimiters upon reading a text file, you get the raw (binary)
432 representation.
433  
434 The file doesn't really need to be text but it is just mapped
435 as one large binary chunk.
436  
437 This function is just a convenience wrapper which firsts C<stat>s
438 the file and sets the dimensions and datatype.
439  
440 =for usage
441  
442 $pdl4 = maptextfraw("fname", {options}
443  
444 =for options
445  
446 The options other than Dims, Datatype of C<mapfraw> are
447 supported.
448  
449 =cut
450  
451 0 ▶sub PDL::maptextfraw {
Subroutine maptextfraw: not called CC=1 SCAR=6.9
Pod undocumented
452 0 my($class, $name, $opts) = @_;
453 0 $opts = {%$opts}; # Copy just in case
454 0 my @s = stat $name;
455 0 $opts->{Dims} = [$s[7]];
456 0 $opts->{Datatype} = &PDL::byte;
457 0 return PDL::mapfraw($class, $name, $opts);
458 }
459  
460 =head1 BUGS
461  
462 Should be documented better. C<writefraw> and C<readfraw> should
463 also have options (the author nowadays only uses C<mapfraw> ;)
464  
465 =head1 AUTHOR
466  
467 Copyright (C) Tuomas J. Lukka 1997.
468 All rights reserved. There is no warranty. You are allowed
469 to redistribute this software / documentation under certain
470 conditions. For details, see the file COPYING in the PDL
471 distribution. If this file is separated from the PDL distribution,
472 the copyright notice should be included in the file.
473  
474 =cut
475  
476 1 1;
« blib/lib/PDL/Parallel/threads.pm ↑ Summary blib/lib/PDL/Image2D.pm »