blib/lib/Cn/Util.pm

filter: Statementstmt 94.3% 414 / 439 Branchbran 81.8% 203 / 248 Conditioncond 64.1% 68 / 106 MC/DCmcdc 47.4% 28 / 59 Subroutinesub 97.2% 35 / 36 Podpod 100% 10 / 10 totaltotal 84.4% 758 / 898 CC 181 SCAR 27.7
CC
181
Cov
86%
_structure_describe
29.0
structure
26.5
dir
26.4
CRAP 15.9
↑ Summary
1 # ABSTRACT: Small everyday helpers: directory listing, shell commands, JSON files, number formatting, and data-structure shapes
2 11 ▶use strict;
Subroutine BEGIN: called CC=1 SCAR=0.0
3 11 ▶use feature 'say';
Subroutine BEGIN: called CC=1 SCAR=0.0
4 11 ▶use warnings FATAL => 'all';
Subroutine BEGIN: called CC=1 SCAR=0.0
5 # Devel::Confess is loaded for its side effect, which is process-wide: it
6 # installs $SIG{__DIE__} and $SIG{__WARN__} so that every die and warn in the
7 # program carries a stack trace. `use Cn::Util` has always given the caller
8 # this, so it stays at the top, where loading the module is enough.
9 11 ▶use Devel::Confess;
Subroutine BEGIN: called CC=1 SCAR=0.0
10  
11 package Cn::Util;
12 11 require 5.010001;
13 11 our $VERSION = 0.0242;
14 11 require Exporter;
15 11 our @ISA = ('Exporter');
16 11 our @EXPORT = qw(dir execute file2string format_commas json_file_to_ref random_key random_value ref_to_json_file st structure);
17  
18 # Startup cost. Everything other than the pragmas, Exporter and
19 # Devel::Confess is loaded on first use, not at `use Cn::Util`. Loading DDP,
20 # autodie, Capture::Tiny and JSON::MaybeXS eagerly cost ~40 ms of interpreter
21 # startup on every script that touched this module, even ones that only called
22 # format_commas().
23 #
24 # The rules that decide what can be deferred:
25 # - PRAGMAS (warnings, feature, autodie) act on the *compiler* for the
26 # enclosing lexical scope, so they can never be deferred. autodie was the
27 # single most expensive line here (~15 ms); this file does not use it, and
28 # the bare open()s it was covering carry explicit `or die`.
29 # - Exporter must exist at compile time (it supplies import()), but it is
30 # free -- it is already in memory as part of the core.
31 # - Everything else is a plain module, so `require` inside the sub that
32 # needs it works. `require` is idempotent and a repeat call is one %INC
33 # hash lookup, so putting it in the hot sub costs nothing measurable.
34 # - A runtime `require` does NOT run import(), so the imported short names
35 # (find, capture, decode_json, colored, blessed) are not available. Each
36 # call site below is therefore fully qualified. Calling a not-yet-loaded
37 # sub with a block prototype is also unparseable at compile time, which is
38 # why Capture::Tiny::capture is handed an explicit sub{} instead of a bare
39 # {}.
40  
41 # ---------------------------------------------------------------------------
42 # `use Cn::Util` hands the calling package three things besides @EXPORT:
43 # autodie, and DDP's p() and np().
44 #
45 # autodie is LEXICAL, and that is the hard case. It is Fatal underneath, and
46 # Fatal::import does a bare `caller()` with no $Level support: calling
47 # autodie->import() directly from in here makes Fatal believe the importing
48 # file is Cn/Util.pm. Its runtime leak-guard then sees calls arriving from a
49 # different file and correctly declines to apply -- open() in the caller just
50 # returns undef again, silently. Verified: the naive version does nothing at
51 # all.
52 #
53 # So the import has to be performed in a context where caller() reports the
54 # caller's package and file. A string eval carrying a #line directive does
55 # exactly that, and is the same trick Import::Into uses (not depended on here
56 # because it is not a core module). $line/$file come from our own caller, so
57 # autodie's scope starts at the `use Cn::Util` line, which is where a
58 # hand-written `use autodie` would have started it too.
59 #
60 # The eval only compiles a closure, and the import runs when that closure is
61 # called after the eval has returned, which is also what Import::Into does.
62 # Perls up to at least 5.12 keep a string eval's own %^H in force while the
63 # eval runs, and autodie 2.06 (core in 5.10.1 and 5.12.5) keeps the guard that
64 # un-installs it at the end of the scope in %^H: imported inside the eval,
65 # autodie was un-installed again as the eval returned, and a missing file
66 # opened in the caller returned false. t/00-load.t checks this.
67 #
68 # p() and np() are installed by _install_ddp_stubs(), below.
69 #
70 # NB: `use Cn::Util ();` skips import() entirely, per normal Perl semantics,
71 # so that is the escape hatch for a caller that wants none of this.
72 # ---------------------------------------------------------------------------
73 11 ▶sub import {
Subroutine import: called CC=2 SCAR=7.0
74 11 my $class = shift;
75 11 $class->export_to_level(1, $class, @_); # the @EXPORT subs, one level up
76  
77 11 my ($pkg, $file, $line) = caller;
78 11 _install_ddp_stubs($pkg);
79 11 require autodie;
80 11 ▶ my $autodie = eval qq{package $pkg;\n#line $line "$file"\nsub { autodie->import(':default') }}
Branch: 1 branch
BranchTrueFalseTotal
my $autodie = eval qq[package $pkg;\n#line $line "$file"\nsub { autodie->import(':default') }] or die "Cn::Util: could not propagate autodie to ${file}: $@" 11 0 11
81 or die "Cn::Util: could not propagate autodie to $file: $@";
82 11 $autodie->();
83 11 return;
84 }
85  
86 # Scripts rely on `use Cn::Util` giving them DDP's p() and np() without a
87 # `use DDP` of their own. These stubs keep that convenience without paying
88 # DDP's ~16 ms unless something actually prints. They are installed at the
89 # caller's `use` line, which is compile time for the rest of the caller, and
90 # with DDP's own prototype, because that is what makes `p @files` pass \@files
91 # rather than a flat list. On first call, DDP->import() is run as if from the
92 # caller's package, so it overwrites the stub there with the real thing.
93 #
94 # A p() or np() the caller already has -- its own, or DDP's from an earlier
95 # `use DDP` -- is left alone. Util.pm, which this module was before it was
96 # Cn::Util, compiled the stubs into main:: whenever it loaded, which redefined
97 # DDP's p() under its FATAL warnings when a script said `use DDP` first.
98 11 my %DDP_LOADED; # package => true once DDP's real p() and np() are installed there
99 my %DDP_STUB; # package => { name => the stub installed for it }, p and np
100  
101 11 ▶sub _install_ddp_stubs {
Subroutine _install_ddp_stubs: called CC=3 SCAR=11.0
102 11 my ($pkg) = @_;
103 11 ▶ no strict 'refs';
Subroutine BEGIN: called CC=1 SCAR=0.0
104 11 for my $name ('p', 'np') {
105 22 my $glob = "${pkg}::$name";
106 22 ▶ next if defined &{$glob};
Branch: 1 branch
BranchTrueFalseTotal
if defined &{$glob;} 1 21 22
107 21 ▶ *{$glob} = $DDP_STUB{$pkg}{$name} = sub (\[@$%&];%) {
Subroutine __ANON__: called CC=2 SCAR=7.0
108 4 _load_ddp($pkg);
109 # A `p` that DDP did not replace -- an `alias` in ~/.dataprinter
110 # exports it under another name -- would otherwise be this stub
111 # again, and the goto would call it forever.
112 die "Cn::Util: DDP did not install ${pkg}::$name\n"
113 2 ▶ if \&{$glob} == $DDP_STUB{$pkg}{$name};
Branch: 1 branch
BranchTrueFalseTotal
if \&{$glob;} == $DDP_STUB{$pkg}{$name} 0 2 2
114 2 goto &{$glob};
115 21 };
116 }
117 11 return;
118 }
119  
120 4 ▶sub _load_ddp {
Subroutine _load_ddp: called CC=8 SCAR=21.2
121 4 my ($pkg) = @_;
122 # Marked loaded only once the import has succeeded. Marking it first made
123 # a second call after a failed load return here at once, and the stub's
124 # goto then called the stub itself, forever.
125 4 ▶ return if $DDP_LOADED{$pkg};
Branch: 1 branch
BranchTrueFalseTotal
if $DDP_LOADED{$pkg} 0 4 4
126 4 require DDP;
127 11 ▶ no strict 'refs';
Subroutine BEGIN: called CC=1 SCAR=0.0
128 # DDP->import() exports both p() and np(), but a stub was installed only
129 # for a name the caller had no sub under, so the other -- or one the caller
130 # defined over a stub after its `use` line -- is the caller's own. Its
131 # glob is localised for the import, so DDP writes into a temporary that is
132 # discarded, and the caller's sub comes back untouched.
133 my %own = map {
134 2 my $glob = "${pkg}::$_";
135 4 ▶ $_ => (defined &{$glob} && \&{$glob} != ($DDP_STUB{$pkg}{$_} // 0));
Condition80%4/5
defined &{$glob;} && \&{$glob;} != ($DDP_STUB{$pkg}{$_} // 0)
!ll&&!rl&&r
031
$DDP_STUB{$pkg}{$_} // 0
l!l
31
MC/DC50%1/2
defined &{$glob;} && \&{$glob;} != ($DDP_STUB{$pkg}{$_} // 0)
defined &{$glob;} \&{$glob;} != ($DDP_STUB{$pkg}{$_} // 0)
MC/DC100%1/1
$DDP_STUB{$pkg}{$_} // 0
$DDP_STUB{$pkg}{$_}
Truth table66%2/3
A && B
A= defined &{$glob;}B= \&{$glob;} != ($DDP_STUB{$pkg}{$_} // 0)
ABresult
0X0
100
111
Truth table100%2/2
A
A= $DDP_STUB{$pkg}{$_}
Aresult
11
00
136 } 'p', 'np';
137 2 ▶ local *{"${pkg}::p"} if $own{p};
Branch: 1 branch
BranchTrueFalseTotal
if $own{"p"} 1 1 2
138 0 ▶ local *{"${pkg}::np"} if $own{np};
Branch: 1 branch
BranchTrueFalseTotal
if $own{"np"} 0 2 2
139 3 ▶ local $SIG{__WARN__} = sub {
Subroutine __ANON__: called CC=2 SCAR=7.6
140 3 ▶ warn @_ unless $_[0] =~ m{\ASubroutine \Q$pkg\E::n?p redefined};
Branch: 1 branch
BranchTrueFalseTotal
unless $_[0] =~ /\ASubroutine \Q$pkg\E::n?p redefined/ 0 3 3
141 2 };
142 # show_memsize needs Devel::Size, which is not a prerequisite; asked for
143 # without it, DDP warned "Devel::Size not found" on every first p() or np().
144 2 ▶ my $memsize = eval { require Devel::Size; 1 } ? 1 : 0;
Branch: 1 branch
BranchTrueFalseTotal
eval { do { require Devel::Size; 1 } } ? : 2 0 2
145 2 my $ok = eval qq{package $pkg;\nDDP->import({output => 'STDOUT', array_max => 10, show_memsize => $memsize, show_dualvar => 'off'}); 1};
146 2 ▶ die "Cn::Util: could not load DDP into $pkg: $@" unless $ok;
Branch: 1 branch
BranchTrueFalseTotal
unless $ok 0 2 2
147 2 $DDP_LOADED{$pkg} = 1;
148 2 return;
149 }
150  
151 39 ▶sub dir {
Subroutine dir: called CC=26 SCAR=32.7
Pod documented
152 39 my %args = (
153 tests => 'f',
154 dir => '.',
155 regex => '.',
156 recursive => 0,
157 on_match => undef, # optional callback: streams results, keeps RAM flat
158 cpu => 1, # worker processes for a recursive walk; 1 = no forking
159 @_,
160 );
161  
162 39 my %allowed = map { $_ => 1 } qw(tests dir regex recursive on_match cpu);
163 39 ▶ if (my @bad = grep { !$allowed{$_} } keys %args) {
Branch: 1 branch
BranchTrueFalseTotal
if (my(@bad) = grep({not $allowed{$_};} keys %args)) 1 38 39
164 1 die "Unrecognized arguments: @bad\nAccepted args: "
165 . join(', ', sort keys %allowed);
166 }
167 # Added 's' (non-zero size), removed 't' (tty test - meaningless on a path)
168 # Checked here so that an undef gets this message rather than "Use of
169 # uninitialized value" from the first line that reads it. recursive is
170 # left out: undef is as false there as 0.
171 38 for my $name (qw(tests dir regex cpu)) {
172 149 ▶ die "$name must be defined" unless defined $args{$name};
Branch: 1 branch
BranchTrueFalseTotal
unless defined $args{$name} 1 148 149
173 }
174 die "on_match must be a code reference"
175 37 ▶ if defined $args{on_match} && ref $args{on_match} ne 'CODE';
Branch: 1 branch
BranchTrueFalseTotal
if defined $args{"on_match"} and ref $args{"on_match"} ne "CODE" 1 36 37
Condition100%3/3
defined $args{"on_match"} and ref $args{"on_match"} ne "CODE"
!ll&&!rl&&r
3421
MC/DC100%2/2
defined $args{"on_match"} and ref $args{"on_match"} ne "CODE"
defined $args{"on_match"} ref $args{"on_match"} ne "CODE"
Truth table100%3/3
A and B
A= defined $args{"on_match"}B= ref $args{"on_match"} ne "CODE"
ABresult
0X0
100
111
176 die "Unrecognized characters in tests: $args{tests}"
177 36 ▶ if $args{tests} =~ /[^rwxoRWXOezsfdlpSbcugkTB]/;
Branch: 1 branch
BranchTrueFalseTotal
if $args{"tests"} =~ /[^rwxoRWXOezsfdlpSbcugkTB]/ 1 35 36
178 die "cpu must be a positive integer, got '$args{cpu}'"
179 35 ▶ unless $args{cpu} =~ /\A[1-9][0-9]*\z/;
Branch: 1 branch
BranchTrueFalseTotal
unless $args{"cpu"} =~ /\A[1-9][0-9]*\z/ 2 33 35
180 33 ▶ my $re = eval { qr/$args{regex}/ }
Branch: 1 branch
BranchTrueFalseTotal
my $re = eval { do { qr/$args{"regex"}/ } } or die qq[Invalid regex '$args{"regex"}': $@] 32 1 33
181 or die "Invalid regex '$args{regex}': $@";
182 # Compile the tests once. The first test stats the path; every later test
183 # reuses the stat buffer via the special '_' filehandle, so N tests cost
184 # one stat(2) syscall instead of N (~2.4x faster for 4 tests).
185 # -l needs lstat, so it is hoisted to the front, and the test after it
186 # must re-stat the real path (otherwise '_' holds lstat data, or -l _
187 # would be a fatal error).
188 32 my @t = split //, $args{tests};
189 32 @t = ((grep { $_ eq 'l' } @t), (grep { $_ ne 'l' } @t));
190 32 my @parts;
191 32 my $fresh = 1; # 1 = the next test must stat the path itself rather than reuse '_'
192 32 for my $t (@t) {
193 0 ▶ if ($t eq 'l') { push @parts, '-l $_[0]'; $fresh = 1 }
Branch: 2 branches
BranchTrueFalseTotal
if ($t eq 'l') { } 0 28 28
elsif ($fresh) { } 28 0 28
194 28 elsif ($fresh) { push @parts, "-$t \$_[0]"; $fresh = 0 }
195 0 else { push @parts, "-$t _" }
196 }
197 my $tester = @parts
198 ? (eval 'sub { ' . join(' && ', @parts) . ' }'
199 or die "Failed to compile tests: $@")
200 32 ▶ : sub { 1 }; # empty tests string now means "no filtering", not "nothing matches"
Subroutine __ANON__: called CC=1 SCAR=0.0
Branch: 1 branch
BranchTrueFalseTotal
@parts ? : 28 4 32
Condition50%1/2
eval 'sub { ' . join(" && ", @parts) . ' }' or die "Failed to compile tests: $@"
l!l
280
MC/DC0%0/1
eval 'sub { ' . join(" && ", @parts) . ' }' or die "Failed to compile tests: $@"
eval 'sub { ' . join(" && ", @parts) . ' }'
Truth table50%1/2
A
A= eval 'sub { ' . join(" && ", @parts) . ' }'
Aresult
11
00
201  
202 32 my $target_dir = $args{dir};
203 # On Windows every "\" in dir becomes "/", so that every path dir() returns
204 # there uses "/" alone. This began as matching File::Find, which did the
205 # same to its starting path (the "canonicalize directory separators" step
206 # of _find_opt in 1.44, on the Strawberry Perl 5.42 smoker) while the
207 # recursive walk used it, and mixed separators made that walk report its
208 # own root. Only on MSWin32: elsewhere "\" is an ordinary character in a
209 # file name.
210 32 my $is_win32 = $^O eq 'MSWin32';
211 32 ▶ $target_dir =~ tr{\\}{/} if $is_win32;
Branch: 1 branch
BranchTrueFalseTotal
if $is_win32 0 32 32
212 # Strip trailing slashes, but keep a root intact: "/", and on Windows "C:/",
213 # because "C:" alone is the current directory on drive C, not its root.
214 # Collapsing first is what turns "///" into "/"; it used to be left as it
215 # was, to be joined as "////etc".
216 32 $target_dir =~ s{/+\z}{/}s;
217 32 ▶ $target_dir =~ s{/\z}{}s
Branch: 2 branches
BranchTrueFalseTotal
unless $target_dir =~ ($is_win32 ? qr"\A(?:[A-Za-z]:)?/\z"s : qr"\A/\z"s) 30 2 32
$is_win32 ? : 0 32 32
218 unless $target_dir =~ ($is_win32 ? qr{\A(?:[A-Za-z]:)?/\z}s : qr{\A/\z}s);
219 # "." is left off the front of results; a root, the only $d that still
220 # ends in "/", must not become "//etc" or "C://etc".
221 720 ▶ my $join = sub {
Subroutine __ANON__: called CC=3 SCAR=11.0
222 720 my ($d, $item) = @_;
223 720 ▶ $d eq '.' ? $item : $d =~ m{/\z} ? "$d$item" : "$d/$item"
Branch: 2 branches
BranchTrueFalseTotal
$d eq '.' ? : 26 694 720
$d =~ m[/\z] ? : 42 652 694
224 32 };
225  
226 32 my $cb = $args{on_match};
227 32 my @items;
228 32 ▶ my $emit = $cb // sub { push @items, $_[0] };
Subroutine __ANON__: called CC=1 SCAR=0.1
Condition66%2/3
$cb // sub { push @items, $_[0]; }
l!l&&r!l&&!r
2300
MC/DC0%0/2
$cb // sub { push @items, $_[0]; }
$cb sub { push @items, $_[0]; }
Truth table66%2/3
A // B
A= $cbB= sub { push @items, $_[0]; }
ABresult
1X1
011
000
229  
230 32 ▶ if ($args{recursive}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($args{'recursive'}) { } 23 9 32
231 # Walk everything strictly below $start, depth first, handing each
232 # match to $sink. _dir_level returns a directory's subdirectories
233 # without the symlinks among them, so the walk does not follow links;
234 # but opendir follows $start itself, so a dir that is a symlink to a
235 # directory is listed through the link.
236 #
237 # File::Find did this walk up to 0.024. Over ~/perl5 (279,000
238 # entries, warm cache, 2026-11-09) it took 1.06 s of user CPU to this
239 # loop's 0.43 s, for the same system time: the two make the same stat
240 # and lstat calls. It also lstat()ed its starting path, so a symlinked
241 # dir needed listing by hand before it, and on Windows it rewrote the
242 # separators of that path, which the "\" conversion above is for.
243 11 ▶ my $find = sub {
Subroutine __ANON__: called CC=2 SCAR=6.9
244 11 my ($start, $sink) = @_;
245 11 my @todo = ($start);
246 11 ▶ push @todo, _dir_level(pop @todo, $re, $tester, $join, $sink) while @todo;
Branch: 1 branch
BranchTrueFalseTotal
@todo and do { push @todo, _dir_level(pop @todo, $re, $tester, $join, $sink) } 206 9 215
247 9 return;
248 23 };
249 23 ▶ if ($args{cpu} == 1) {
Branch: 1 branch
BranchTrueFalseTotal
if ($args{'cpu'} == 1) { } 11 12 23
250 11 $find->($target_dir, $emit);
251 } else {
252 12 _dir_parallel($target_dir, $args{cpu}, $re, $tester, $join, $find, $emit);
253 }
254 } else {
255 # cpu is ignored here on purpose: one readdir plus one stat per entry
256 # is ~6 us warm, far below the cost of forking and shipping results back.
257 9 ▶ opendir my $dh, $target_dir
Branch: 1 branch
BranchTrueFalseTotal
opendir my $dh, $target_dir or die "Cannot open directory ${target_dir}: $!" 8 1 9
258 or die "Cannot open directory $target_dir: $!";
259 8 ▶ while (defined(my $item = readdir $dh)) { # defined(): a file named "0" is falsy
Branch: 1 branch
BranchTrueFalseTotal
if (defined(my $item = readdir $dh)) 96 8 104
260 96 ▶ next if $item =~ /\A\.\.?\z/; # \A..\z: "$" would match before a trailing \n
Branch: 1 branch
BranchTrueFalseTotal
if $item =~ /\A\.\.?\z/ 16 80 96
261 80 ▶ next if $item !~ $re;
Branch: 1 branch
BranchTrueFalseTotal
unless $item =~ /$re/ 11 69 80
262 69 my $path = $join->($target_dir, $item);
263 69 ▶ $emit->($path) if $tester->($path);
Branch: 1 branch
BranchTrueFalseTotal
if &$tester($path) 62 7 69
264 }
265 8 closedir $dh;
266 }
267 27 ▶ return $cb ? () : @items;
Branch: 1 branch
BranchTrueFalseTotal
$cb ? : 2 25 27
268 }
269  
270 # Jobs handed out per worker. Parallel::ForkManager starts a new job whenever
271 # a worker finishes, so more jobs than workers is what evens out a tree whose
272 # subdirectories differ wildly in size; each job costs one fork plus one
273 # Storable round-trip through a temp file, so too many jobs loses again.
274 # Measured 2026-09-23, 20-core NVMe box, warm cache, best of 3, seconds:
275 #
276 # tree (files) serial k=1 k=2 k=4 k=8 k=16
277 # ~/perl5 (150k), cpu=8 1.23 0.95 0.25 0.29 0.36 0.47
278 # ~/perl5 (150k), cpu=4 1.23 1.03 0.82 0.39 0.45 0.61
279 # perlbrew/build (120k), cpu=8 0.75 0.19 0.24 0.25 0.33 0.44
280 #
281 # k=1 and k=2 collapse on the skewed tree; 4 is the smallest value that does
282 # not. Re-run: edit the value below, then time
283 # perl -Ilib -MCn::Util -e 'dir(dir => $ARGV[0], recursive => 1, cpu => 8)' TREE
284 11 my $DIR_JOBS_PER_CPU = 4;
285  
286 # Hand each entry of directory $d that matches $re and $tester to $emit, and
287 # return its subdirectories, leaving out symlinks to directories so that a
288 # recursive walk does not follow them. One level of a recursive dir(): the
289 # serial walk is a loop over it, and so are the parallel walk's breadth-first
290 # levels and each of its workers.
291 218 ▶sub _dir_level {
Subroutine _dir_level: called CC=8 SCAR=20.8
292 218 my ($d, $re, $tester, $join, $emit) = @_;
293 218 ▶ opendir my $dh, $d or die "Cannot open directory $d: $!";
Branch: 1 branch
BranchTrueFalseTotal
opendir my $dh, $d or die "Cannot open directory ${d}: $!" 215 3 218
294 215 my @sub;
295 215 ▶ while (defined(my $item = readdir $dh)) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined(my $item = readdir $dh)) 1081 215 1296
296 1081 ▶ next if $item =~ /\A\.\.?\z/;
Branch: 1 branch
BranchTrueFalseTotal
if $item =~ /\A\.\.?\z/ 430 651 1081
297 651 my $path = $join->($d, $item);
298 651 ▶ $emit->($path) if $item =~ $re && $tester->($path);
Branch: 1 branch
BranchTrueFalseTotal
if $item =~ /$re/ and &$tester($path) 334 317 651
Condition100%3/3
$item =~ /$re/ and &$tester($path)
!ll&&!rl&&r
67250334
MC/DC100%2/2
$item =~ /$re/ and &$tester($path)
$item =~ /$re/ &$tester($path)
Truth table100%3/3
A and B
A= $item =~ /$re/B= &$tester($path)
ABresult
0X0
100
111
299 651 ▶ push @sub, $path if !-l $path && -d _; # -l: lstat, so -d _ is the entry itself, not its target
Branch: 1 branch
BranchTrueFalseTotal
if not -l $path and -d _ 340 311 651
Condition100%3/3
not -l $path and -d _
!ll&&!rl&&r
18293340
MC/DC100%2/2
not -l $path and -d _
not -l $path -d _
Truth table100%3/3
not(A) and B
A= not -l $pathB= -d _
ABresult
0X0
100
111
300 }
301 215 closedir $dh;
302 215 return @sub;
303 }
304  
305 # Recursive dir() across $cpu worker processes. Results come back to this
306 # process and go through $emit here, so an on_match closure still sees and
307 # mutates the caller's variables -- it just runs after a whole job's batch
308 # arrives rather than per file, and match order differs from the serial walk.
309 12 ▶sub _dir_parallel {
Subroutine _dir_parallel: called CC=12 SCAR=29.7
310 12 my ($root, $cpu, $re, $tester, $join, $find, $emit) = @_;
311 12 require Parallel::ForkManager; # not core, and only cpu > 1 needs it
312  
313 # One subtree per job would starve the pool whenever a single directory
314 # holds most of the tree (~/perl5: one of 3 top-level dirs has 99% of the
315 # files, and a top-level split gave no speedup at any cpu). So walk
316 # breadth-first here until a level is wide enough to share out, testing
317 # the entries of every level walked; the final level's directories are
318 # already tested and become the jobs.
319 12 my $want = $cpu * $DIR_JOBS_PER_CPU;
320 12 my @level = ($root);
321 12 ▶ while (@level && @level < $want) {
Branch: 1 branch
BranchTrueFalseTotal
if (@level and @level < $want) 12 11 23
Condition66%2/3
@level and @level < $want
!ll&&!rl&&r
01112
MC/DC50%1/2
@level and @level < $want
@level @level < $want
Truth table66%2/3
A and B
A= @levelB= @level < $want
ABresult
0X0
100
111
322 12 @level = map { _dir_level($_, $re, $tester, $join, $emit) } @level;
323 }
324 11 ▶ return unless @level;
Branch: 1 branch
BranchTrueFalseTotal
unless @level 0 11 11
325  
326 11 my @jobs;
327 11 push @{ $jobs[$_ % $want] }, $level[$_] for 0 .. $#level;
328  
329 # A worker hands its results back in a Storable file of its own, named by
330 # its pid, and leaves with POSIX::_exit, rather than through
331 # Parallel::ForkManager's finish(), whose CORE::exit ran the caller's END
332 # blocks and destructors in every worker: an END that removed a lock file
333 # or printed a summary did it a dozen times, and a database handle was
334 # closed from under the parent. On Windows a worker is a thread of this
335 # process, and _exit would end the whole of it, so there it still calls
336 # finish(). newdir() removes the directory only in this process, by pid.
337 11 require POSIX;
338 11 require Storable;
339 11 require File::Temp;
340 11 my $is_win32 = $^O eq 'MSWin32';
341 11 my $tmp = File::Temp->newdir;
342  
343 11 my @errors;
344 11 my $pm = Parallel::ForkManager->new($cpu);
345 # The default 1 s poll in wait_all_children added a flat 1.01 s to every
346 # call, measured; 0 makes it block in waitpid instead.
347 11 $pm->set_waitpid_blocking_sleep(0);
348 92 ▶ $pm->run_on_finish(sub {
Subroutine __ANON__: called CC=8 SCAR=24.0
349 92 my ($pid, $exit, undef, $signal) = @_;
350 92 my $file = "$tmp/$pid";
351 92 ▶ my $data = -e $file ? eval { Storable::retrieve($file) } : undef;
Branch: 1 branch
BranchTrueFalseTotal
-e $file ? : 92 0 92
352 92 ▶ unlink $file if -e $file;
Branch: 1 branch
BranchTrueFalseTotal
if -e $file 92 0 92
353 92 ▶ if ($signal || $exit || !$data) {
Branch: 1 branch
BranchTrueFalseTotal
if ($signal or $exit or not $data) 0 92 92
Condition33%2/6
$signal or $exit or not $data
l!l&&r!l&&!r
0092
$signal or $exit
l!l&&r!l&&!r
0092
MC/DC0%0/3
$signal or $exit or not $data
$signal $exit not $data
Truth table25%1/4
A or B or not(C)
A= $signalB= $exitC= not $data
ABCresult
1XX1
01X1
0011
0000
354 0 push @errors, "dir: worker $pid died (exit $exit, signal $signal)";
355 0 return;
356 }
357 92 ▶ push @errors, $data->{error} if defined $data->{error};
Branch: 1 branch
BranchTrueFalseTotal
if defined $data->{"error"} 1 91 92
358 92 $emit->($_) for @{ $data->{paths} };
359 11 });
360 11 for my $job (@jobs) {
361 92 ▶ $pm->start and next;
Branch: 1 branch
BranchTrueFalseTotal
$pm->start and next 92 0 92
362 0 my @found;
363 0 ▶ my $ok = eval { $find->($_, sub { push @found, $_[0] }) for @$job; 1 };
Subroutine __ANON__: not called CC=1 SCAR=6.9
364 0 ▶ my $error = $ok ? undef : "$@"; # taken now: the eval below clears $@ as it starts
Branch: 1 branch
BranchTrueFalseTotal
$ok ? : 0 0 0
365 0 my $stored = eval { Storable::store({ paths => \@found, error => $error }, "$tmp/$$") };
366 0 ▶ my $code = $stored ? 0 : 1; # 1 = the results could not be written
Branch: 1 branch
BranchTrueFalseTotal
$stored ? : 0 0 0
367 0 ▶ $pm->finish($code) if $is_win32;
Branch: 1 branch
BranchTrueFalseTotal
if $is_win32 0 0 0
368 0 POSIX::_exit($code);
369 }
370 11 $pm->wait_all_children;
371 11 ▶ die join "\n", @errors if @errors;
Branch: 1 branch
BranchTrueFalseTotal
if @errors 1 10 11
372 10 return;
373 }
374  
375 14 ▶sub execute {
Subroutine execute: called CC=16 SCAR=27.8
Pod documented
376 14 my ($cmd, $return, $die) = @_;
377 14 ▶ $return = 'exit' unless defined $return;
Branch: 1 branch
BranchTrueFalseTotal
unless defined $return 4 10 14
378 14 ▶ $die = 1 unless defined $die;
Branch: 1 branch
BranchTrueFalseTotal
unless defined $die 11 3 14
379 # \A..\z: "$" also matches before a trailing newline, so "exit\n" passed
380 # here, ran the command, and only then died with "broke pigeonholes".
381 14 ▶ if ($return !~ m/\A(?:exit|stdout|stderr|all)\z/) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($return =~ /\A(?:exit|stdout|stderr|all)\z/) 2 12 14
382 2 die "you gave \$return = \"$return\", while this subroutine only accepts exit, stdout, stderr or all";
383 }
384 12 require Capture::Tiny; # ~7.5 ms
385 # Capture::Tiny redirects fd 1 and fd 2 and nothing else, so the child still
386 # inherits the parent's fd 0. A child that prompts -- rm on a
387 # write-protected file, cp -i, apt -- prompts only when fd 0 is a tty, then
388 # writes its question into the captured stderr where nobody can see it and
389 # blocks on the terminal for an answer the user does not know is wanted.
390 # (Observed 2026-09-13: `rm -r` on read-only cl.tex hung jobsearch/tex/tar.pl
391 # indefinitely.) Pointing fd 0 at /dev/null turns the prompt into an instant
392 # EOF, so such a child declines and exits instead of hanging.
393 #
394 # What goes on fd 0 is the read end of an empty pipe whose write end is
395 # closed, and only the descriptor is swapped, with dup2; the STDIN handle is
396 # left alone. Before system() forks, perl flushes every handle, and flushing
397 # one it reads from lseek()s its descriptor back to the position perl has
398 # read up to and, if that seek succeeds, discards the read-ahead buffer. A
399 # pipe cannot seek, so the buffer survives; the null device can, so with it
400 # on fd 0, as in 0.023, the buffered input was lost:
401 # `printf 'a\nb\nc\n' | perl -e 'while (<STDIN>) { print; execute("true") }'`
402 # printed only "a". Whatever holds fd 0 -- STDIN, or another handle when
403 # STDIN is closed -- is saved and put back, and fd 0 is closed again if it
404 # was closed before.
405 #
406 # fd 0 is saved before the pipe is made: when it is free, pipe() takes it,
407 # and a dup of fd 0 made afterwards saved the pipe, which was then
408 # "restored", leaving fd 0 open on it.
409 12 require POSIX;
410 12 my $saved_fd0 = POSIX::dup(0); # undef = fd 0 was closed
411 12 ▶ pipe my $eof, my $writer or die "cannot make a pipe for the command's input: $!";
Branch: 1 branch
BranchTrueFalseTotal
pipe my $eof, my $writer or die "cannot make a pipe for the command's input: $!" 12 0 12
412 12 close $writer;
413 12 my $eof_on_fd0 = fileno $eof == 0; # 1 = fd 0 was free and pipe() took it
414 12 ▶ unless ($eof_on_fd0) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($eof_on_fd0) 11 1 12
415 11 ▶ defined POSIX::dup2(fileno $eof, 0) or die "cannot point fd 0 at an empty pipe: $!";
Branch: 1 branch
BranchTrueFalseTotal
defined POSIX::dup2(fileno $eof, 0) or die "cannot point fd 0 at an empty pipe: $!" 11 0 11
416 11 close $eof;
417 }
418 # An explicit sub{} rather than capture { ... }: the prototype that makes
419 # the block form legal is not visible at compile time under a runtime
420 # require, so perl would try to parse the block as an anonymous hash.
421 # The eval is so that fd 0 is restored even when capture() dies; 0.023 left
422 # it on the null device for the rest of the program.
423 12 my ($stdout, $stderr, $exit);
424 12 my $ok = eval {
425 11 ▶ ($stdout, $stderr, $exit) = Capture::Tiny::capture(sub {
Subroutine __ANON__: called CC=1 SCAR=0.0
426 11 system( $cmd )
427 12 });
428 11 1;
429 };
430 12 my $err = $@;
431 12 ▶ if (defined $saved_fd0) {
Branch: 2 branches
BranchTrueFalseTotal
if (defined $saved_fd0) { } 11 1 12
elsif ($eof_on_fd0) { } 1 0 1
432 11 ▶ defined POSIX::dup2($saved_fd0, 0) or die "cannot restore fd 0: $!";
Branch: 1 branch
BranchTrueFalseTotal
defined POSIX::dup2($saved_fd0, 0) or die "cannot restore fd 0: $!" 11 0 11
433 11 POSIX::close($saved_fd0);
434 } elsif ($eof_on_fd0) {
435 1 close $eof;
436 } else {
437 0 POSIX::close(0);
438 }
439 12 ▶ die $err unless $ok;
Branch: 1 branch
BranchTrueFalseTotal
unless $ok 1 11 12
440 # Truthiness, as the documentation describes it. `$die == 1` made 'yes' die
441 # with "isn't numeric" and made 2 mean "do not die".
442 11 ▶ if ($die && $exit != 0) {
Branch: 1 branch
BranchTrueFalseTotal
if ($die and $exit != 0) 3 8 11
Condition100%3/3
$die and $exit != 0
!ll&&!rl&&r
173
MC/DC100%2/2
$die and $exit != 0
$die $exit != 0
Truth table100%3/3
A and B
A= $dieB= $exit != 0
ABresult
0X0
100
111
443 3 say STDERR "exit = $exit";
444 3 say STDERR "STDOUT = $stdout";
445 3 say STDERR "STDERR = $stderr";
446 3 die "$cmd\n failed";
447 }
448 8 ▶ return $exit if $return eq 'exit';
Branch: 1 branch
BranchTrueFalseTotal
if $return eq "exit" 3 5 8
449 5 chomp $stdout;
450 5 chomp $stderr;
451 5 ▶ return $stdout if $return eq 'stdout';
Branch: 1 branch
BranchTrueFalseTotal
if $return eq "stdout" 3 2 5
452 2 ▶ return $stderr if $return eq 'stderr';
Branch: 1 branch
BranchTrueFalseTotal
if $return eq "stderr" 1 1 2
453 1 return { exit => $exit, stdout => $stdout, stderr => $stderr }; # 'all', the only mode left
454 }
455  
456 4 ▶sub file2string {
Subroutine file2string: called CC=4 SCAR=13.9
Pod documented
457 4 my ($file) = @_;
458 4 ▶ open my $fh, '<', $file or die "can't read $file: $!"; # was autodie
Branch: 1 branch
BranchTrueFalseTotal
open my $fh, "<", $file or die "can't read ${file}: $!" 3 1 4
459 # open() succeeds on a directory on POSIX systems. Linux, OpenBSD and
460 # FreeBSD 13+ then fail the read with EISDIR, but NetBSD's read(2) returns
461 # the raw directory entries, so 0.0241 handed back binary garbage there
462 # (CPAN Testers report 52217fbc-dd66-11f1-9541-8d78add113b8, perl 5.42.3,
463 # amd64-netbsd-thread-multi). So a directory is refused by fstat, whatever
464 # read() would do; Windows never gets here, as its open() fails.
465 3 ▶ if (-d $fh) {
Branch: 1 branch
BranchTrueFalseTotal
if (-d $fh) 1 2 3
466 1 require POSIX;
467 1 $! = POSIX::EISDIR();
468 1 die "can't read $file: $!";
469 }
470 # readline returns undef only on an error -- an empty file reads as ''.
471 # Tested with // rather than through a lexical, which on perl 5.10.1 took
472 # a third copy of the file at return: a 200 MB file peaked at 575 MB,
473 # against 384 MB this way (5.44 is 195 MB either way, through
474 # copy-on-write).
475 2 ▶ return do { local $/; <$fh> } // die "can't read $file: $!";
Condition50%1/2
do { local $/; readline $fh } // die("can't read ${file}: $!")
l!l
20
MC/DC0%0/1
do { local $/; readline $fh } // die("can't read ${file}: $!")
do { local $/; readline $fh }
Truth table50%1/2
A
A= do { local $/; readline $fh }
Aresult
11
00
476 }
477  
478 31 ▶sub format_commas { # https://stackoverflow.com/questions/33442240/perl-printf-to-use-commas-as-thousands-separator
Subroutine format_commas: called CC=9 SCAR=22.0
Pod documented
479 31 my ($n, $format) = @_;
480 31 ▶ $format = '.%02d' unless defined $format;
Branch: 1 branch
BranchTrueFalseTotal
unless defined $format 20 11 31
481 # '%.0u' is the documented way to ask for a whole number, but sprintf
482 # prints nothing for it only when the hundredths are 0: 1234.56 came out
483 # "1,23456". So '%.0u' is recognised here, rounds half up to a whole
484 # number, and prints no fraction at all.
485 31 my $whole = $format eq '%.0u';
486 # Infinity and NaN are returned as perl prints them. Run through the
487 # arithmetic below they came out "Inf.NaN", "-Inf.NaN" and "NaN.NaN".
488 # $n - $n is NaN for exactly these, and NaN is the one value not equal to
489 # itself.
490 31 my $diff = $n - $n;
491 31 ▶ return "$n" if $diff != $diff;
Branch: 1 branch
BranchTrueFalseTotal
if $diff != $diff 3 28 31
492 28 my $neg = $n < 0;
493 28 my $abs = abs $n;
494 28 my ($int, $cents);
495 28 ▶ if ($whole) {
Branch: 1 branch
BranchTrueFalseTotal
if ($whole) { } 11 17 28
496 11 $int = int($abs + .5);
497 11 $cents = 0;
498 } else {
499 17 $int = int $abs;
500 # int(100 * (.005 + frac)) is the original expression, and is exactly
501 # round-half-up to hundredths -- .005 * 100 is the +0.5 before truncation.
502 # It is kept verbatim so no already-correct result shifts by a cent.
503 17 $cents = int(100 * (.005 + ($abs - $int)));
504 # ...but it can land on 100 (.999 rounds up to 1.00), which used to be
505 # printed straight into the fraction: 999999.999 came out "999,999.100".
506 # The carry has to go into the integer part instead.
507 17 ▶ if ($cents >= 100) {
Branch: 1 branch
BranchTrueFalseTotal
if ($cents >= 100) 5 12 17
508 5 $int++;
509 5 $cents -= 100;
510 }
511 }
512 # Split off the sign and work on the magnitude: the old code fed a negative
513 # through int()/sprintf and produced "-1,234.-49" for -1234.5, and lost the
514 # sign entirely for -0.5. A value that rounds to zero stays unsigned, so
515 # -0.001 is "0.00" rather than "-0.00".
516 28 ▶ my $sign = ($neg && ($int || $cents)) ? '-' : '';
Branch: 1 branch
BranchTrueFalseTotal
$neg && ($int || $cents) ? : 5 23 28
Condition100%6/6
$neg and $int || $cents
!ll&&!rl&&r
2125
$int or $cents
l!l&&r!l&&!r
412
MC/DC100%3/3
$neg and $int || $cents
$neg $int $cents
Truth table100%4/4
A and B or C
A= $negB= $intC= $cents
ABCresult
0XX0
1000
11X1
1011
517 # '%.0f' rather than interpolating $int: a large magnitude stringifies in
518 # scientific notation, and the reverse/unpack grouping shredded that into
519 # "1e,+20" for 1e20.
520 28 my $digits = sprintf '%.0f', $int;
521 28 ▶ return $sign . reverse(join(',', unpack('(A3)*', reverse $digits))) . ($whole ? '' : sprintf($format, $cents));
Branch: 1 branch
BranchTrueFalseTotal
$whole ? : 11 17 28
522 }
523  
524 5 ▶sub json_file_to_ref {
Subroutine json_file_to_ref: called CC=6 SCAR=18.0
Pod documented
525 5 my ($json_filename) = @_;
526 5 ▶ die "$json_filename doesn't exist or isn't a file" unless -f $json_filename;
Branch: 1 branch
BranchTrueFalseTotal
unless -f $json_filename 1 4 5
527 4 ▶ die "$json_filename has 0 size" if -s $json_filename == 0;
Branch: 1 branch
BranchTrueFalseTotal
if -s $json_filename == 0 1 3 4
528 3 ▶ open my $fh, '<:raw', $json_filename or die "can't read $json_filename: $!";
Branch: 1 branch
BranchTrueFalseTotal
open my $fh, "<:raw", $json_filename or die "can't read ${json_filename}: $!" 3 0 3
529 3 my $json = do { local $/; <$fh> }; # slurp; $/ restored immediately
530 3 close $fh;
531 # Python/NumPy emit bare NaN / Infinity / -Infinity, which are not valid
532 # JSON. Rewrite them to null (-> undef) only when they sit in JSON value
533 # position (after '[', ',' or ':'). That test alone also matched text
534 # inside a string -- "values: [1, NaN, 2]" became "values: [1, null, 2]"
535 # -- so the first alternative matches each whole string literal and
536 # (*SKIP)(*FAIL) resumes the scan after its closing quote.
537 #
538 # That scan is slow: on a 43 MB file of 300,000 records it took 2.1 s, to
539 # decode_json's 0.30 s (Cpanel::JSON::XS), measured 2026-11-09. The two
540 # index() calls that rule it out took 4.5 ms there, so a file with neither
541 # word, which is most of them, is not scanned at all.
542 3 ▶ $json =~ s/"(?:[^"\\]++|\\.)*+"(*SKIP)(*FAIL)|[\[,:]\s*\K-?(?:NaN|Infinity)(?=\s*[,\]}])/null/gs
Branch: 1 branch
BranchTrueFalseTotal
if index($json, "NaN") != -1 or index($json, "Infinity") != -1 2 1 3
Condition66%2/3
index($json, "NaN") != -1 or index($json, "Infinity") != -1
l!l&&r!l&&!r
201
MC/DC50%1/2
index($json, "NaN") != -1 or index($json, "Infinity") != -1
index($json, "NaN") != -1 index($json, "Infinity") != -1
Truth table66%2/3
A or B
A= index($json, "NaN") != -1B= index($json, "Infinity") != -1
ABresult
1X1
011
000
543 if index($json, 'NaN') >= 0 || index($json, 'Infinity') >= 0;
544 3 require JSON::MaybeXS; # ~4.3 ms
545 3 return JSON::MaybeXS::decode_json($json);
546 }
547  
548 # These return *some* key/value from the hash, not a random one -- whichever
549 # pair the hash's internal iterator yields first.
550 #
551 # They used to be `foreach my $key (keys %{$hash}) { return $key }`, which is
552 # O(n): `keys` in list context materializes EVERY key into a temporary list,
553 # and then the loop throws all but the first away. On a 50,000-key hash that
554 # was 1.48 ms per call; via each() it is 0.15 us, ~10,000x faster.
555 #
556 # The bare `keys` in void context resets the iterator, so repeated calls keep
557 # returning the first pair instead of walking off the end of the hash. It is
558 # reset again after the each(), which would otherwise leave it one pair in, so
559 # that a caller's own `while (each %h)` afterwards skipped the first pair.
560 # The old `keys`-in-list-context version left it at the start, and so do these.
561 #
562 # Falling off a foreach also returned '' rather than undef in scalar context;
563 # each() on an empty hash returns an empty list, so these now yield a true
564 # undef, which is what t/random.t always asserted.
565 4 ▶sub random_key {
Subroutine random_key: called CC=1 SCAR=0.0
Pod documented
566 4 my ($hash) = @_;
567 4 keys %{ $hash }; # reset the iterator
568 4 my ($key) = each %{ $hash };
569 4 keys %{ $hash };
570 4 return $key;
571 }
572  
573 4 ▶sub random_value {
Subroutine random_value: called CC=1 SCAR=0.0
Pod documented
574 4 my ($hash) = @_;
575 4 keys %{ $hash }; # reset the iterator
576 4 my (undef, $value) = each %{ $hash };
577 4 keys %{ $hash };
578 4 return $value;
579 }
580  
581 6 ▶sub ref_to_json_file {
Subroutine ref_to_json_file: called CC=7 SCAR=19.5
Pod documented
582 6 my ($ref, $json_filename) = @_;
583 6 my $ref_json_filename = ref $json_filename;
584 6 ▶ unless ($ref_json_filename eq '') {
Branch: 1 branch
BranchTrueFalseTotal
unless ($ref_json_filename eq "") 1 5 6
585 1 die "$json_filename isn't a scalar/string";
586 }
587 5 require JSON::MaybeXS;
588 5 ▶ open my $fh, '>:raw', $json_filename # Write it unmangled
Branch: 1 branch
BranchTrueFalseTotal
open my $fh, ">:raw", $json_filename or die "can't write ${json_filename}: $!" 5 0 5
589 or die "can't write $json_filename: $!"; # was autodie
590 5 say $fh JSON::MaybeXS::encode_json($ref);
591 # A full disk shows up at close, when the buffer is flushed. Unchecked,
592 # perl up to 5.20 reported "Wrote" and returned the name of a truncated
593 # file, and later perls died with "unable to close filehandle" from the
594 # implicit close under FATAL warnings.
595 5 ▶ close $fh or die "can't write $json_filename: $!";
Branch: 1 branch
BranchTrueFalseTotal
close $fh or die "can't write ${json_filename}: $!" 4 1 5
596 # Colored by the rule structure()'s color => 'auto' follows. Up to 0.024
597 # it always was, so a log or a pipe that STDOUT went to got the escape
598 # codes too.
599 4 my $shown = $json_filename;
600 4 ▶ if (($ENV{CLICOLOR_FORCE} || -t STDOUT) && !$ENV{NO_COLOR}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($ENV{"CLICOLOR_FORCE"} || -t STDOUT and not $ENV{"NO_COLOR"}) 1 3 4
Condition83%5/6
$ENV{"CLICOLOR_FORCE"} || -t STDOUT and not $ENV{"NO_COLOR"}
!ll&&!rl&&r
211
$ENV{"CLICOLOR_FORCE"} or -t STDOUT
l!l&&r!l&&!r
202
MC/DC66%2/3
$ENV{"CLICOLOR_FORCE"} || -t STDOUT and not $ENV{"NO_COLOR"}
$ENV{"CLICOLOR_FORCE"} -t STDOUT not $ENV{"NO_COLOR"}
Truth table60%3/5
A or B and not(C)
A= $ENV{"CLICOLOR_FORCE"}B= -t STDOUTC= not $ENV{"NO_COLOR"}
ABCresult
00X0
1X00
0100
1X11
0111
601 1 require Term::ANSIColor;
602 1 $shown = Term::ANSIColor::colored(['blue on_red'], $shown);
603 }
604 4 say "Wrote $shown";
605 4 return $json_filename;
606 }
607  
608 # ----------------------------------------------------------------------------
609 # structure(): print the *shape* of a data structure instead of its values.
610 #
611 # p() from Data::Printer shows you every value; when a 50,000-element array
612 # holds the same kind of record over and over, that is the wrong tool. This
613 # merges all elements of an array into ONE representative node, so you see the
614 # schema once. Keys absent from some of the merged records are flagged '?';
615 # values that disagree about their type become a UNION.
616 #
617 # Two entry points, mirroring DDP's own p()/np() split:
618 # st() - prototyped, so `st %hash` works like `p %hash`. Use this
619 # interactively.
620 # structure() - plain sub, takes a reference: structure(\%hash), or the
621 # return value of another call. Use this from other code.
622 #
623 # Scalar::Util and Term::ANSIColor are require'd in structure(), which st()
624 # also funnels through, so the per-value helpers below can call them fully
625 # qualified without re-checking %INC on every node. Scalar::Util is not loaded
626 # at `use` time because it drags in List::Util's XS bootstrap (~4 ms) and only
627 # this family ever touches it.
628  
629 11 my %STRUCTURE_COLOR = (
630 key => 'bold', # hash key names
631 opt => 'red', # the '?' optional marker
632 HASH => 'bold cyan',
633 ARRAY => 'bold yellow',
634 UNION => 'bold magenta',
635 elem => 'bright_black', # the '[*]' / '<alt>' / '(empty)' markers
636 class => 'bold white', # blessed package names
637 number => 'green',
638 string => 'bright_blue',
639 bool => 'bright_magenta',
640 null => 'bright_black',
641 CODE => 'yellow',
642 other => 'red', # anything unexpected
643 );
644  
645 117 ▶sub _structure_color {
Subroutine _structure_color: called CC=3 SCAR=16.5
646 117 my ($name, $text, $use_color) = @_;
647 117 ▶ return $text unless $use_color;
Branch: 1 branch
BranchTrueFalseTotal
unless $use_color 117 0 117
648 0 ▶ return Term::ANSIColor::colored([$STRUCTURE_COLOR{$name} // $STRUCTURE_COLOR{other}], $text);
Condition0%0/3
$STRUCTURE_COLOR{$name} // $STRUCTURE_COLOR{"other"}
l!l&&r!l&&!r
000
MC/DC0%0/2
$STRUCTURE_COLOR{$name} // $STRUCTURE_COLOR{"other"}
$STRUCTURE_COLOR{$name} $STRUCTURE_COLOR{"other"}
Truth table0%0/3
A // B
A= $STRUCTURE_COLOR{$name}B= $STRUCTURE_COLOR{"other"}
ABresult
1X1
011
000
649 }
650  
651 # Classify one value. Blessed containers keep their package name so that
652 # Foo=HASH and a plain HASH don't get merged into the same node.
653 52 ▶sub _structure_kind {
Subroutine _structure_kind: called CC=13 SCAR=29.6
654 52 my ($v) = @_;
655 52 ▶ return 'null' unless defined $v;
Branch: 1 branch
BranchTrueFalseTotal
unless defined $v 1 51 52
656 51 my $r = ref $v;
657 51 ▶ return (Scalar::Util::looks_like_number($v) ? 'number' : 'string') if $r eq '';
Branch: 2 branches
BranchTrueFalseTotal
if $r eq "" 4 47 51
&Scalar::Util::looks_like_number($v) ? : 2 2 4
658 47 ▶ if (my $class = Scalar::Util::blessed($v)) {
Branch: 1 branch
BranchTrueFalseTotal
if (my $class = builtin::blessed($v)) 3 44 47
659 3 ▶ return 'bool' if $class eq 'JSON::PP::Boolean' || $class eq 'Types::Serialiser::Boolean';
Branch: 1 branch
BranchTrueFalseTotal
if $class eq "JSON::PP::Boolean" or $class eq "Types::Serialiser::Boolean" 0 3 3
Condition33%1/3
$class eq "JSON::PP::Boolean" or $class eq "Types::Serialiser::Boolean"
l!l&&r!l&&!r
003
MC/DC0%0/2
$class eq "JSON::PP::Boolean" or $class eq "Types::Serialiser::Boolean"
$class eq "JSON::PP::Boolean" $class eq "Types::Serialiser::Boolean"
Truth table33%1/3
A or B
A= $class eq "JSON::PP::Boolean"B= $class eq "Types::Serialiser::Boolean"
ABresult
1X1
011
000
660 3 ▶ return 'Regexp' if $class eq 'Regexp';
Branch: 1 branch
BranchTrueFalseTotal
if $class eq "Regexp" 0 3 3
661 3 ▶ my $rt = Scalar::Util::reftype($v) // '';
Condition50%1/2
builtin::reftype($v) // ""
l!l
30
MC/DC0%0/1
builtin::reftype($v) // ""
builtin::reftype($v)
Truth table50%1/2
A
A= builtin::reftype($v)
Aresult
11
00
662 3 ▶ return "$class=$rt" if $rt eq 'HASH' || $rt eq 'ARRAY' || $rt eq 'SCALAR' || $rt eq 'REF';
Branch: 1 branch
BranchTrueFalseTotal
if $rt eq "HASH" or $rt eq "ARRAY" or $rt eq "SCALAR" or $rt eq "REF" 3 0 3
Condition33%3/9
$rt eq "HASH" or $rt eq "ARRAY" or $rt eq "SCALAR" or $rt eq "REF"
l!l&&r!l&&!r
300
$rt eq "HASH" or $rt eq "ARRAY" or $rt eq "SCALAR"
l!l&&r!l&&!r
300
$rt eq "HASH" or $rt eq "ARRAY"
l!l&&r!l&&!r
300
MC/DC0%0/4
$rt eq "HASH" or $rt eq "ARRAY" or $rt eq "SCALAR" or $rt eq "REF"
$rt eq "HASH" $rt eq "ARRAY" $rt eq "SCALAR" $rt eq "REF"
Truth table20%1/5
A or B or C or D
A= $rt eq "HASH"B= $rt eq "ARRAY"C= $rt eq "SCALAR"D= $rt eq "REF"
ABCDresult
1XXX1
01XX1
001X1
00011
00000
663 0 return $class;
664 }
665 44 return $r; # HASH ARRAY SCALAR REF CODE GLOB
666 }
667  
668 # Merge a list of same-position values into a single structural node:
669 # { kind => 'hash', keys => { k => { node => .., optional => 0|1 } }, n, class }
670 # { kind => 'array', node => .., n => #instances, lens => [lengths], class }
671 # { kind => 'sref', node => .. } # \$x and \\$x
672 # { kind => 'leaf', types => { typename => 1, ... } }
673 # { kind => 'union', alts => { kindname => node } }
674 # $stack holds the refaddrs of the containers we are currently inside, so a
675 # self-referential structure terminates instead of recursing forever.
676 #
677 # The values arrive as $chunks, a list of array references whose elements,
678 # taken together, are the values. The elements of the arrays being merged are
679 # then described where they lie -- the arrays themselves are the chunks --
680 # rather than copied into one list, and values are copied only to split a
681 # union by kind. Copying at every level, as up to 0.024 did, took 413 MB on
682 # top of a 362 MB array of 300,000 five-key records, and 2.0 s; this takes
683 # 237 MB and 1.4 s (perl 5.44, 2026-11-09). Most of what is left is the
684 # copies of hash values, one list per key, and the cycle guard's $stack.
685 55 ▶sub _structure_describe {
Subroutine _structure_describe: called CC=29 SCAR=33.7
686 55 my ($chunks, $opt, $depth, $stack) = @_;
687 55 my %count; # kind => how many values are of that kind
688 55 for my $chunk (@{ $chunks }) {
689 # A plain scalar, the commonest value, is classified here rather than
690 # by a call to _structure_kind, which cost 0.3 s of the 1.7 s that
691 # 300,000 five-key records took. The answers are the same.
692 $count{ !defined $_ ? 'null'
693 : ref $_ ? _structure_kind($_)
694 61 ▶ : Scalar::Util::looks_like_number($_) ? 'number' : 'string' }++ for @{ $chunk };
Branch: 3 branches
BranchTrueFalseTotal
!defined($_) ? : 2 74 76
ref $_ ? : 43 31 74
&Scalar::Util::looks_like_number($_) ? : 23 8 31
695 }
696 55 my @kinds = sort keys %count;
697  
698 55 ▶ if (@kinds > 1) {
Branch: 1 branch
BranchTrueFalseTotal
if (@kinds > 1) 3 52 55
699 3 my %by_kind;
700 3 for my $chunk (@{ $chunks }) {
701 5 push @{ $by_kind{ _structure_kind($_) } }, $_ for @{ $chunk };
702 }
703 return {
704 kind => 'union',
705 3 alts => { map { $_ => _structure_describe([$by_kind{$_}], $opt, $depth, $stack) } @kinds },
706 };
707 }
708 52 ▶ my $k = $kinds[0] // 'null';
Condition50%1/2
$kinds[0] // "null"
l!l
520
MC/DC0%0/1
$kinds[0] // "null"
$kinds[0]
Truth table50%1/2
A
A= $kinds[0]
Aresult
11
00
709 52 my ($class, $reftype) = $k =~ /\A(.+)=(HASH|ARRAY|SCALAR|REF)\z/;
710 52 ▶ $reftype //= $k;
Condition66%2/3
$reftype //= $k
l!l&&r!l&&!r
1510
MC/DC0%0/2
$reftype //= $k
$reftype $k
Truth table66%2/3
A //= B
A= $reftypeB= $k
ABresult
1X1
011
000
711  
712 52 ▶ if ($reftype eq 'HASH' || $reftype eq 'ARRAY' || $reftype eq 'SCALAR' || $reftype eq 'REF') {
Branch: 1 branch
BranchTrueFalseTotal
if ($reftype eq "HASH" or $reftype eq "ARRAY" or $reftype eq "SCALAR" or $reftype eq "REF") 32 20 52
Condition100%9/9
$reftype eq "HASH" or $reftype eq "ARRAY" or $reftype eq "SCALAR" or $reftype eq "REF"
l!l&&r!l&&!r
28420
$reftype eq "HASH" or $reftype eq "ARRAY" or $reftype eq "SCALAR"
l!l&&r!l&&!r
27124
$reftype eq "HASH" or $reftype eq "ARRAY"
l!l&&r!l&&!r
18925
MC/DC100%4/4
$reftype eq "HASH" or $reftype eq "ARRAY" or $reftype eq "SCALAR" or $reftype eq "REF"
$reftype eq "HASH" $reftype eq "ARRAY" $reftype eq "SCALAR" $reftype eq "REF"
Truth table100%5/5
A or B or C or D
A= $reftype eq "HASH"B= $reftype eq "ARRAY"C= $reftype eq "SCALAR"D= $reftype eq "REF"
ABCDresult
1XXX1
01XX1
001X1
00011
00000
713 32 ▶ my $all = @{ $chunks } == 1 ? $chunks->[0] : [ map { @{ $_ } } @{ $chunks } ];
Branch: 1 branch
BranchTrueFalseTotal
@{$chunks;} == 1 ? : 31 1 32
714 # Drop anything we're already inside: its shape is an ancestor's shape.
715 # Copied only when there is something to drop.
716 39 my $fresh = (grep { $stack->{ Scalar::Util::refaddr($_) } } @{ $all })
717 32 ▶ ? [ grep { !$stack->{ Scalar::Util::refaddr($_) } } @{ $all } ]
Branch: 1 branch
BranchTrueFalseTotal
grep({$$stack{builtin::refaddr($_)};} @{$all;}) ? : 1 31 32
718 : $all;
719 32 ▶ return { kind => 'leaf', types => { '(cycle)' => 1 } } unless @{ $fresh };
Branch: 1 branch
BranchTrueFalseTotal
unless @{$fresh;} 1 31 32
720 31 ▶ if ($opt->{max_depth} && $depth >= $opt->{max_depth}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($opt->{"max_depth"} and $depth >= $opt->{"max_depth"}) 1 30 31
Condition100%3/3
$opt->{"max_depth"} and $depth >= $opt->{"max_depth"}
!ll&&!rl&&r
2821
MC/DC100%2/2
$opt->{"max_depth"} and $depth >= $opt->{"max_depth"}
$opt->{"max_depth"} $depth >= $opt->{"max_depth"}
Truth table100%3/3
A and B
A= $opt->{"max_depth"}B= $depth >= $opt->{"max_depth"}
ABresult
0X0
100
111
721 1 return { kind => 'leaf', types => { '(...)' => 1 } };
722 }
723 30 local @{ $stack }{ map { Scalar::Util::refaddr($_) } @{ $fresh } } = (1) x @{ $fresh };
724  
725 30 ▶ if ($reftype eq 'HASH') {
Branch: 2 branches
BranchTrueFalseTotal
if ($reftype eq 'HASH') { } 16 14 30
elsif ($reftype eq 'ARRAY') { } 9 5 14
726 # Each hash puts one value under each of its keys, so a key whose
727 # list is shorter than @$fresh is missing from some of them.
728 16 my %vals; # key => [ its value in each hash that has it ]
729 16 for my $h (@{ $fresh }) {
730 21 push @{ $vals{$_} }, $h->{$_} for keys %{ $h };
731 }
732 # A wide hash used as a dictionary has no interesting key names:
733 # merge every value into one representative entry instead.
734 16 ▶ if ($opt->{hash_max} && keys %vals > $opt->{hash_max}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($opt->{"hash_max"} and keys %vals > $opt->{"hash_max"}) 1 15 16
Condition66%2/3
$opt->{"hash_max"} and keys %vals > $opt->{"hash_max"}
!ll&&!rl&&r
0151
MC/DC50%1/2
$opt->{"hash_max"} and keys %vals > $opt->{"hash_max"}
$opt->{"hash_max"} keys %vals > $opt->{"hash_max"}
Truth table66%2/3
A and B
A= $opt->{"hash_max"}B= keys %vals > $opt->{"hash_max"}
ABresult
0X0
100
111
735 return {
736 kind => 'hash',
737 class => $class,
738 1 n => scalar @{ $fresh },
739 wide => scalar keys %vals,
740 keys => { '*' => {
741 node => _structure_describe([values %vals], $opt, $depth + 1, $stack),
742 optional => 0,
743 }},
744 };
745 }
746 15 my %out;
747 15 for my $key (keys %vals) {
748 $out{$key} = {
749 node => _structure_describe([$vals{$key}], $opt, $depth + 1, $stack),
750 19 ▶ optional => (@{ $vals{$key} } < @{ $fresh } ? 1 : 0),
Branch: 1 branch
BranchTrueFalseTotal
@{$vals{$key};} < @{$fresh;} ? : 1 18 19
751 };
752 }
753 15 return { kind => 'hash', class => $class, n => scalar @{ $fresh }, keys => \%out };
754 }
755 elsif ($reftype eq 'ARRAY') {
756 9 my @lens = map { scalar @{ $_ } } @{ $fresh };
757 return {
758 kind => 'array',
759 class => $class,
760 11 node => ((grep { $_ } @lens) ? _structure_describe($fresh, $opt, $depth + 1, $stack) : undef),
761 9 ▶ n => scalar @{ $fresh },
Branch: 1 branch
BranchTrueFalseTotal
grep({$_;} @lens) ? : 9 0 9
762 lens => \@lens,
763 };
764 }
765 return { # SCALAR / REF
766 kind => 'sref',
767 class => $class,
768 5 node => _structure_describe([[ map { ${ $_ } } @{ $fresh } ]], $opt, $depth + 1, $stack),
769 };
770 }
771 20 return { kind => 'leaf', types => { $k => 1 } };
772 }
773  
774 9 ▶sub _structure_array_desc {
Subroutine _structure_array_desc: called CC=6 SCAR=17.9
775 9 my ($node, $c) = @_;
776 9 my @lens = @{ $node->{lens} };
777 9 my ($min, $max) = ($lens[0], $lens[0]);
778 9 ▶ for (@lens) { $min = $_ if $_ < $min; $max = $_ if $_ > $max }
Branch: 2 branches
BranchTrueFalseTotal
if $_ < $min 0 11 11
if $_ > $max 1 10 11
779 9 ▶ my $range = $min == $max ? $min : "$min-$max";
Branch: 1 branch
BranchTrueFalseTotal
$min == $max ? : 8 1 9
780 9 my $out = $c->('ARRAY', "ARRAY[$range]");
781 9 ▶ $out .= " (from $node->{n} instances)" if $node->{n} > 1;
Branch: 1 branch
BranchTrueFalseTotal
if $node->{"n"} > 1 1 8 9
782 9 return $out;
783 }
784  
785 55 ▶sub _structure_render {
Subroutine _structure_render: called CC=17 SCAR=28.4
786 55 my ($label, $node, $indent, $opt, $c, $lines) = @_;
787 55 ▶ my $pad = (' ' x $indent) . ($label // '');
Condition100%2/2
$label // ""
l!l
4213
MC/DC100%1/1
$label // ""
$label
Truth table100%2/2
A
A= $label
Aresult
11
00
788 55 ▶ my $class = defined $node->{class} ? $c->('class', "$node->{class}=") : '';
Branch: 1 branch
BranchTrueFalseTotal
defined $node->{'class'} ? : 1 54 55
789  
790 55 ▶ if ($node->{kind} eq 'hash') {
Branch: 4 branches
BranchTrueFalseTotal
if ($node->{'kind'} eq 'hash') { } 16 39 55
elsif ($node->{'kind'} eq 'array') { } 9 30 39
elsif ($node->{'kind'} eq 'sref') { } 5 25 30
elsif ($node->{'kind'} eq 'union') { } 3 22 25
791 16 my @k = sort keys %{ $node->{keys} };
792 16 ▶ my $count = $node->{wide} // scalar @k;
Condition100%2/2
$node->{"wide"} // scalar @k
l!l
115
MC/DC100%1/1
$node->{"wide"} // scalar @k
$node->{"wide"}
Truth table100%2/2
A
A= $node->{"wide"}
Aresult
11
00
793 16 ▶ push @{ $lines }, $pad . $class . $c->('HASH', "HASH{$count}")
Branch: 2 branches
BranchTrueFalseTotal
$node->{'n'} > 1 ? : 2 14 16
$node->{'wide'} ? : 1 15 16
794 . ($node->{n} > 1 ? " (from $node->{n} instances)" : '')
795 16 . ($node->{wide} ? ' ' . $c->('elem', '(keys merged)') : '');
796 16 for my $key (@k) {
797 20 my $entry = $node->{keys}{$key};
798 my $name = ($key eq '*' && $node->{wide})
799 ? $c->('elem', '<*>') . ': '
800 20 ▶ : $c->('key', $key) . ($entry->{optional} ? $c->('opt', '?') : '') . ': ';
Branch: 2 branches
BranchTrueFalseTotal
$key eq '*' && $node->{'wide'} ? : 1 19 20
$entry->{'optional'} ? : 1 18 19
Condition66%2/3
$key eq "*" and $node->{"wide"}
!ll&&!rl&&r
1901
MC/DC50%1/2
$key eq "*" and $node->{"wide"}
$key eq "*" $node->{"wide"}
Truth table66%2/3
A and B
A= $key eq "*"B= $node->{"wide"}
ABresult
0X0
100
111
801 20 _structure_render($name, $entry->{node}, $indent + 1, $opt, $c, $lines);
802 }
803 }
804 elsif ($node->{kind} eq 'array') {
805 9 push @{ $lines }, $pad . $class . _structure_array_desc($node, $c);
806 9 ▶ if ($node->{node}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($node->{'node'}) { } 9 0 9
807 9 _structure_render($c->('elem', '[*]') . ': ', $node->{node}, $indent + 1, $opt, $c, $lines);
808 } else {
809 0 push @{ $lines }, (' ' x ($indent + 1)) . $c->('elem', '(empty)');
810 }
811 }
812 elsif ($node->{kind} eq 'sref') { # \$x: stay on the same line, no extra indent level
813 # The '\ ' joins the label rather than the pad, and $indent passes
814 # through unchanged, so whatever the referent holds is still indented
815 # from this line. Passing $pad with indent 0 put the line right but
816 # dropped the referent's own children back to depth 1.
817 5 ▶ _structure_render(($label // '') . $class . $c->('elem', '\\') . ' ', $node->{node}, $indent, $opt, $c, $lines);
Condition100%2/2
$label // ""
l!l
41
MC/DC100%1/1
$label // ""
$label
Truth table100%2/2
A
A= $label
Aresult
11
00
818 }
819 elsif ($node->{kind} eq 'union') {
820 3 push @{ $lines }, $pad . $c->('UNION', 'UNION');
821 3 for my $alt (sort keys %{ $node->{alts} }) {
822 8 _structure_render($c->('elem', "<$alt>") . ' ', $node->{alts}{$alt}, $indent + 1, $opt, $c, $lines);
823 }
824 }
825 else { # leaf
826 22 push @{ $lines }, $pad . join($c->('elem', '|'),
827 22 map { $c->($_, $_) } sort keys %{ $node->{types} });
828 }
829 55 return;
830 }
831  
832 14 ▶sub structure {
Subroutine structure: called CC=14 SCAR=32.8
Pod documented
833 14 my ($ref, %args) = @_;
834 14 my %opt = (
835 output => 'stdout', # 'stdout' | 'stderr' | 'string' (string: return it, print nothing)
836 color => 'auto', # 'auto' | 1 | 0
837 max_depth => 0, # 0 = unlimited
838 hash_max => 32, # hashes wider than this collapse to one '<*>' entry; 0 = never
839 as => undef, # optional header line, like DDP's "as"
840 %args,
841 );
842 14 my %allowed = map { $_ => 1 } qw(output color max_depth hash_max as);
843 14 ▶ if (my @bad = grep { !$allowed{$_} } keys %opt) {
Branch: 1 branch
BranchTrueFalseTotal
if (my(@bad) = grep({not $allowed{$_};} keys %opt)) 0 14 14
844 0 die "Unrecognized arguments: @bad\nAccepted args: " . join(', ', sort keys %allowed);
845 }
846 # undef as in dir(): this message, not "Use of uninitialized value".
847 14 for my $name (qw(output color)) {
848 28 ▶ die "$name must be defined" unless defined $opt{$name};
Branch: 1 branch
BranchTrueFalseTotal
unless defined $opt{$name} 1 27 28
849 }
850 13 ▶ die "output must be one of stdout, stderr, string" if $opt{output} !~ m/\A(stdout|stderr|string)\z/;
Branch: 1 branch
BranchTrueFalseTotal
unless $opt{"output"} =~ /\A(stdout|stderr|string)\z/ 0 13 13
851  
852 13 require Scalar::Util; # ~4 ms (List::Util XS bootstrap); see header note
853  
854 13 ▶ my $fh = $opt{output} eq 'stderr' ? \*STDERR : \*STDOUT;
Branch: 1 branch
BranchTrueFalseTotal
$opt{'output'} eq 'stderr' ? : 0 13 13
855 my $use_color = $opt{color} eq 'auto'
856 ? ((($ENV{CLICOLOR_FORCE} || ($opt{output} ne 'string' && -t $fh)) && !$ENV{NO_COLOR}) ? 1 : 0)
857 13 ▶ : $opt{color};
Branch: 2 branches
BranchTrueFalseTotal
$opt{'color'} eq 'auto' ? : 0 13 13
($ENV{'CLICOLOR_FORCE'} || $opt{'output'} ne 'string' && -t $fh) && !$ENV{'NO_COLOR'} ? : 0 0 0
Condition0%0/9
$ENV{"CLICOLOR_FORCE"} || $opt{"output"} ne "string" && -t $fh and not $ENV{"NO_COLOR"}
!ll&&!rl&&r
000
$ENV{"CLICOLOR_FORCE"} or $opt{"output"} ne "string" and -t $fh
l!l&&r!l&&!r
000
$opt{"output"} ne "string" and -t $fh
!ll&&!rl&&r
000
MC/DC0%0/4
$ENV{"CLICOLOR_FORCE"} || $opt{"output"} ne "string" && -t $fh and not $ENV{"NO_COLOR"}
$ENV{"CLICOLOR_FORCE"} $opt{"output"} ne "string" -t $fh not $ENV{"NO_COLOR"}
Truth table0%0/6
A or B and C and not(D)
A= $ENV{"CLICOLOR_FORCE"}B= $opt{"output"} ne "string"C= -t $fhD= not $ENV{"NO_COLOR"}
ABCDresult
00XX0
010X0
1XX00
01100
1XX11
01111
858 13 ▶ require Term::ANSIColor if $use_color; # ~0.8 ms, skipped entirely when piping
Branch: 1 branch
BranchTrueFalseTotal
if $use_color 0 13 13
859 13 ▶ my $c = sub { _structure_color($_[0], $_[1], $use_color) };
Subroutine __ANON__: called CC=1 SCAR=0.0
860  
861 13 my @lines;
862 13 ▶ push @lines, $c->('key', $opt{as}) if defined $opt{as};
Branch: 1 branch
BranchTrueFalseTotal
if defined $opt{"as"} 0 13 13
863 13 _structure_render(undef, _structure_describe([[$ref]], \%opt, 0, {}), 0, \%opt, $c, \@lines);
864 13 my $out = join("\n", @lines) . "\n";
865 0 ▶ print {$fh} $out unless $opt{output} eq 'string';
Branch: 1 branch
BranchTrueFalseTotal
unless $opt{"output"} eq "string" 0 13 13
866 13 return $out;
867 }
868  
869 # st(): structure() with DDP's calling convention, so you can write
870 #
871 # st %hash; st @array; st $ref; st %hash, max_depth => 2;
872 #
873 # instead of backslashing by hand. The \[@$%&] prototype makes perl pass a
874 # reference to the caller's variable, which is the whole trick -- but it also
875 # means the argument must BE a variable: st(\%h), st({...}) and
876 # st(func()) are compile-time errors, exactly as they are for DDP's p().
877 # Call structure() directly in those cases.
878 #
879 # NOTE: the obvious short name 's' is impossible. perl's tokenizer resolves a
880 # bare 's' to the substitution operator before it ever looks for a sub of that
881 # name, so `s $ref` dies with "Substitution pattern not terminated" and even
882 # `s($ref)` is parsed as s(pat)(repl). Only `&s($ref)` would work, which
883 # defeats the point. Hence 'st'.
884 1 ▶sub st (\[@$%&];%) {
Subroutine st: called CC=3 SCAR=13.3
Pod documented
885 1 my ($item, @args) = @_;
886 # `st %h` / `st @a` / `st &c` hand us the HASH/ARRAY/CODE ref directly, but
887 # `st $x` hands us a ref to the scalar $x -- one layer more than we want.
888 # The prototype rejects everything except a plain variable, so a SCALAR/REF
889 # here is always that added layer: peel exactly one. `st $hashref` then
890 # describes the hash, and `my $x = 42; st $x` prints 'number', not '\ number'.
891 0 ▶ my $ref = (ref($item) eq 'SCALAR' || ref($item) eq 'REF') ? ${ $item } : $item;
Branch: 1 branch
BranchTrueFalseTotal
ref $item eq 'SCALAR' || ref $item eq 'REF' ? : 0 1 1
Condition33%1/3
ref $item eq "SCALAR" or ref $item eq "REF"
l!l&&r!l&&!r
001
MC/DC0%0/2
ref $item eq "SCALAR" or ref $item eq "REF"
ref $item eq "SCALAR" ref $item eq "REF"
Truth table33%1/3
A or B
A= ref $item eq "SCALAR"B= ref $item eq "REF"
ABresult
1X1
011
000
892 1 return structure($ref, @args);
893 }
894  
895 11 1;
896 # from md2pod.pl πατερ ημων ο εν τοις ουρανοις, ἁγιασθήτω τὸ ὄνομά σου
897 =encoding utf8
898  
899 =head1 NAME
900  
901 Cn::Util - Small everyday helpers: directory listing, shell commands, JSON files, number formatting, and data-structure shapes
902  
903 =head1 Synopsis
904  
905 use Cn::Util;
906
907 my @scripts = dir(dir => '.', regex => '\.pl$', recursive => 1);
908 my $branch = execute('git rev-parse --abbrev-ref HEAD', 'stdout');
909 my $text = file2string('notes.txt');
910 print format_commas(1234567.891), "\n"; # 1,234,567.89
911
912 my $data = json_file_to_ref('results.json');
913 ref_to_json_file($data, 'copy.json');
914
915 st $data; # print the shape of $data, not its values
916 p $data; # Data::Printer's p(), loaded on first use
917  
918 =head1 Description
919  
920 C<Cn::Util> is a collection of the functions I reach for in nearly every script: listing a directory with file tests and a regex, running a shell command and keeping its output, slurping a file, reading and writing JSON, putting commas in a number, and seeing the I<shape> of a large data structure without scrolling through every value in it.
921  
922 Loading it is meant to be cheap. Everything that is not needed to compile the caller is loaded the first time a function needs it, so a script that only calls C<format_commas> does not pay for C<JSON::MaybeXS>, C<Capture::Tiny> or C<Data::Printer>.
923  
924 =head2 What C<use Cn::Util> does
925  
926 Besides exporting the ten functions below, C<use Cn::Util> does three things to the file and package that loads it:
927  
928 =over
929  
930 =item * It turns on C<autodie ':default'> in that file, from the C<use> line onward, so a bare C<open> or C<close> that fails dies with a message rather than returning false.
931  
932 =item * It gives that package C<p> and C<np> from C<Data::Printer> (C<DDP>). They are stubs until first called, so C<DDP> is loaded only by a script that prints something. A C<p> or C<np> the package already has, such as one from an earlier C<use DDP>, is left alone.
933  
934 =item * It loads C<Devel::Confess>, which is process-wide: every C<die> and C<warn> in the program carries a stack trace.
935  
936 =back
937  
938 C<use Cn::Util ();> imports nothing and skips the first two; C<Devel::Confess> is still loaded. C<use Cn::Util 'format_commas';> imports only the names given, and still does all three.
939  
940 =head1 Functions
941  
942 All functions are exported by default.
943  
944 =head2 dir
945  
946 my @files = dir(dir => '/data', regex => '\.csv$');
947 my @tree = dir(dir => '/data', regex => '\.csv$', recursive => 1, cpu => 8);
948 dir(dir => '/data', recursive => 1, on_match => sub { say $_[0] });
949  
950 Returns the paths under C<dir> whose base name matches C<regex> and that pass every file test in C<tests>. The paths are C<dir> joined to the entry with C</>; with the default C<< dir =E<gt> '.' >>, the leading C<./> is left off. C<.> and C<..> are never returned.
951  
952 It takes these named arguments:
953  
954 =over
955  
956 =item * C<dir> (default C<.>): The directory to list. Trailing slashes are removed, but C</> stays C</>. A C<dir> that is a symbolic link to a directory is listed through the link, recursively or not.
957  
958 =item * C<regex> (default C<.>): A pattern the base name must match, given as a string or a C<qr//>. An invalid pattern dies, naming it.
959  
960 =item * C<tests> (default C<f>): One or more of perl's file tests, as letters: C<rwxoRWXOezsfdlpSbcugkTB>. Every one must pass. C<'d'> lists directories; C<''> applies no test at all.
961  
962 =item * C<recursive> (default C<0>): C<1> descends into subdirectories, without following symbolic links.
963  
964 =item * C<cpu> (default C<1>): For a recursive listing, the number of worker processes (through C<Parallel::ForkManager>). Ignored when C<recursive> is C<0>.
965  
966 =item * C<on_match> (no default): A code reference called with each path as it is found; C<dir> then returns an empty list, so a very large tree is never held in memory.
967  
968 =back
969  
970 The file tests share one C<stat> per path, so asking for four of them costs no more than asking for one. C<-l> is checked first, with C<lstat>.
971  
972 With C<cpu> greater than 1, C<dir> walks the tree breadth-first until one level holds four directories per worker, then shares those directories out. The results are the same as a serial walk, in a different order. C<on_match> still runs in the calling process, so it can change the caller's variables, but it is called once a worker's whole batch arrives rather than as each file is found. A worker leaves without running the program's C<END> blocks or destructors, except on Windows, where it is a thread of the calling process.
973  
974 An unknown argument, an undefined C<tests>, C<dir>, C<regex> or C<cpu>, a letter that is not a file test, a C<cpu> that is not a positive integer, an C<on_match> that is not a code reference, and a directory that cannot be opened, whether C<dir> itself or one a recursive listing meets, all die.
975  
976 =head2 execute
977  
978 my $exit = execute('make test'); # 0, or dies
979 my $out = execute('git log -1 --format=%H', 'stdout');
980 my $err = execute('perl -c script.pl', 'stderr');
981 my $all = execute('ls /nonexistent', 'all', 0); # does not die
982 # $all = { exit => 512, stdout => '', stderr => "ls: cannot access ..." }
983  
984 Runs C<$cmd> through the shell with C<system>, capturing its standard output and standard error with C<Capture::Tiny>. The second argument says what to return:
985  
986 =over
987  
988 =item * C<exit> (the default): the wait status, C<$?> -- 0 for success, 256 for C<exit 1>
989  
990 =item * C<stdout>: standard output, chomped
991  
992 =item * C<stderr>: standard error, chomped
993  
994 =item * C<all>: a hash reference with C<exit>, C<stdout> and C<stderr>
995  
996 =back
997  
998 The third argument, C<$die>, is true by default: a command that exits non-zero prints its exit status, standard output and standard error to C<STDERR> and dies. Pass a false value, such as 0, to have the status returned instead.
999  
1000 The command's standard input is an empty pipe. A command that would prompt -- C<rm> on a write-protected file, C<cp -i> -- reads end-of-file and declines rather than waiting, unseen, for an answer. The caller's C<STDIN> is not touched, so input it has already read ahead is still there afterwards, even when C<execute> dies.
1001  
1002 =head2 file2string
1003  
1004 my $text = file2string('notes.txt');
1005  
1006 Returns the whole of a file as one string, read without any decoding layer. A file that cannot be opened or read, such as a directory, dies, naming it.
1007  
1008 =head2 format_commas
1009  
1010 format_commas(1234567.891); # '1,234,567.89'
1011 format_commas(1234567, '%.0u'); # '1,234,567'
1012 format_commas(999999.999); # '1,000,000.00'
1013 format_commas(-0.5); # '-0.50'
1014 format_commas(1e20); # '100,000,000,000,000,000,000.00'
1015  
1016 Puts a comma between each group of three digits of the integer part and appends the hundredths, rounded half up. The second argument is the C<sprintf> format the hundredths are printed with, C<'.%02d'> by default; C<'%.0u'> rounds half up to a whole number and prints no fraction. Rounding that carries into the integer part does so (C<0.999> is C<1.00>), a negative number keeps its sign, a number that rounds to zero has none, and a large number is grouped as digits rather than printed in exponent form. Infinity and NaN are returned as perl prints them.
1017  
1018 =head2 json_file_to_ref
1019  
1020 my $ref = json_file_to_ref('results.json');
1021  
1022 Reads a JSON file and returns the decoded data, through C<JSON::MaybeXS>. Python's C<json> module and NumPy write C<NaN>, C<Infinity> and C<-Infinity> as bare words, which are not JSON; where one of those stands as a value it is read as C<null>, so it becomes C<undef>. The same words inside a string are left alone. A file that does not exist, or is empty, dies.
1023  
1024 =head2 ref_to_json_file
1025  
1026 ref_to_json_file($data, 'results.json'); # prints "Wrote results.json"
1027  
1028 Writes a data structure to a file as JSON, through C<JSON::MaybeXS>, prints C<Wrote> and the file name, and returns the file name. The name is colored by the rule C<structure>'s C<< color =E<gt> 'auto' >> follows: when C<STDOUT> is a terminal, or C<CLICOLOR_FORCE> is set, and C<NO_COLOR> is not. The file name must be a plain string; a reference dies, and so does a write that fails, such as on a full disk, before anything is printed.
1029  
1030 =head2 random_key, random_value
1031  
1032 my $key = random_key(\%hash);
1033 my $value = random_value(\%hash);
1034  
1035 Return one key, or one value, of a hash -- whichever pair perl's hash iterator yields first, which is quick no matter how big the hash is. This is I<any> key, not a randomly chosen one: the same hash returns the same pair each time. It is useful for looking at one record of a hash of records. An empty hash returns C<undef>.
1036  
1037 Both reset the hash's iterator, so do not call them on a hash in the middle of an C<each> loop.
1038  
1039 =head2 structure
1040  
1041 my $text = structure($ref, output => 'string');
1042 structure(\%config, max_depth => 2, as => 'config');
1043  
1044 Prints the I<shape> of a data structure rather than its values: C<p> shows every element of a 50,000-element array of records, while C<structure> merges them into one and shows the record once.
1045  
1046 my @people = (
1047 { name => 'Ann', age => 31, langs => ['perl', 'R'] },
1048 { name => 'Bo', age => 28, langs => [], email => undef },
1049 );
1050 st @people;
1051  
1052 prints
1053  
1054 ARRAY[2]
1055 [*]: HASH{4} (from 2 instances)
1056 age: number
1057 email?: null
1058 langs: ARRAY[0-2] (from 2 instances)
1059 [*]: string
1060 name: string
1061  
1062 A key that only some of the merged hashes have is marked C<?>. Values in the same place that are of different kinds are shown as a C<UNION> of each kind. Arrays show the range of their lengths. Blessed references show their class. A reference back to a containing structure is shown as C<(cycle)> rather than followed.
1063  
1064 It takes these options after the reference:
1065  
1066 =over
1067  
1068 =item * C<output> (default C<stdout>): C<stdout> or C<stderr> to print there; C<string> to print nothing. The text is returned in every case.
1069  
1070 =item * C<color> (default C<auto>): C<1> or C<0>. C<auto> colours the output when it goes to a terminal, or when C<CLICOLOR_FORCE> is set, unless C<NO_COLOR> is set.
1071  
1072 =item * C<max_depth> (default C<0>): How many levels to descend before printing C<(...)>; C<0> is no limit.
1073  
1074 =item * C<hash_max> (default C<32>): A hash with more keys than this is taken to be a lookup table, and its values are merged into one entry, C<< E<lt>*E<gt> >>, rather than listed by key. C<0> lists every key.
1075  
1076 =item * C<as> (no default): A line to print above the structure.
1077  
1078 =back
1079  
1080 =head2 st
1081  
1082 st %hash;
1083 st @array;
1084 st $ref, max_depth => 2;
1085  
1086 C<structure>, called the way C<p> is called: the argument is a variable, not a reference to one, and the options follow it. As with C<p>, it must be a variable -- C<st(\%h)> and C<st(func())> do not compile; call C<structure> for those.
1087  
1088 =head1 Installation
1089  
1090 cpanm Cn::Util
1091  
1092 or, from a copy of the repository:
1093  
1094 perl Makefile.PL
1095 make
1096 make test
1097 make install
1098  
1099 =head1 Compatibility
1100  
1101 C<Cn::Util> runs on perl 5.10.1 and later, and is tested there. C<execute> hands its command to the shell, so the command has to be written for the shell of the system it runs on. On Windows, the paths C<dir> returns are joined with C</>, which perl accepts there, and a C<\> in the C<dir> argument is returned as C</>, so C<< dir =E<gt> q{C:\Temp} >> lists C<C:/Temp/a.txt>; recursive listings with C<cpu> rely on C<Parallel::ForkManager>'s emulated C<fork>.
1102  
1103 =head1 Author
1104  
1105 David E. Condon L<mailto:dec986@gmail.com>
1106  
1107 =head1 License
1108  
1109 This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
1110  
1111 =head1 Thanks
1112  
1113 A lot of this work used Claude AI, which was paid for by the University of Idaho's IMCI
↑ Summary