lib/App/Test/Generator/Mutator.pm

filter: Statementstmt 100% 119 / 119 Branchbran 90.3% 47 / 52 Conditioncond 90.0% 18 / 20 MC/DCmcdc 66.6% 8 / 12 Subroutinesub 100% 20 / 20 Podpod 100% 5 / 5 totaltotal 95.1% 217 / 228 CC 38 SCAR 19.1
CC
38
Cov
96%
generate_mutants
13.0
_is_redundant_mutation
10.0
_dedup_mutants
7.0
CRAP 6.8
« bin/pod-example-tester ↑ Summary lib/App/Test/Generator/Emitter/Perl.pm »
1 package App::Test::Generator::Mutator;
2  
3 18 ▶use 5.036;
Subroutine BEGIN: called CC=1 SCAR=0.0
4 18 ▶use autodie qw(:io); # covers open/close/read/write; excludes system (which legitimately fails)
Subroutine BEGIN: called CC=1 SCAR=0.0
5 18 ▶use Carp qw(croak);
Subroutine BEGIN: called CC=1 SCAR=0.0
6 18 ▶use Config;
Subroutine BEGIN: called CC=1 SCAR=0.0
7 18 ▶use File::Copy::Recursive qw(dircopy);
Subroutine BEGIN: called CC=1 SCAR=0.0
8 18 ▶use File::Spec;
Subroutine BEGIN: called CC=1 SCAR=0.0
9 18 ▶use File::Temp qw(tempdir);
Subroutine BEGIN: called CC=1 SCAR=0.0
10 18 ▶use PPI;
Subroutine BEGIN: called CC=1 SCAR=0.0
11 18 ▶use Readonly;
Subroutine BEGIN: called CC=1 SCAR=0.0
12  
13 18 ▶use App::Test::Generator::Mutation::BooleanNegation;
Subroutine BEGIN: called CC=1 SCAR=0.0
14 18 ▶use App::Test::Generator::Mutation::ConditionalInversion;
Subroutine BEGIN: called CC=1 SCAR=0.0
15 18 ▶use App::Test::Generator::Mutation::NumericBoundary;
Subroutine BEGIN: called CC=1 SCAR=0.0
16 18 ▶use App::Test::Generator::Mutation::ReturnUndef;
Subroutine BEGIN: called CC=1 SCAR=0.0
17  
18 # --------------------------------------------------
19 # Valid mutation level values
20 # --------------------------------------------------
21 18 Readonly my $LEVEL_FULL => 'full';
22 18 Readonly my $LEVEL_FAST => 'fast';
23  
24 # --------------------------------------------------
25 # Default values for optional constructor arguments
26 # --------------------------------------------------
27 18 Readonly my $DEFAULT_LIB_DIR => 'lib';
28 18 Readonly my $DEFAULT_MUTATION_LEVEL => $LEVEL_FULL;
29  
30 # --------------------------------------------------
31 # Error message constants — named so test code can
32 # match against them without duplicating the literal
33 # --------------------------------------------------
34 18 Readonly my $ERR_FILE_REQUIRED => 'file required';
35 18 Readonly my $ERR_WORKSPACE_NOT_SET => 'Workspace not prepared -- call prepare_workspace first';
36 18 Readonly my $ERR_RELATIVE_NOT_SET => 'Relative path not set -- call prepare_workspace first';
37  
38 18 our $VERSION = '0.50';
39  
40 =encoding utf8
41  
42 =head1 NAME
43  
44 App::Test::Generator::Mutator - Generate and apply mutation tests
45  
46 =head1 VERSION
47  
48 Version 0.50
49  
50 =head1 SYNOPSIS
51  
52 A "mutant" is a copy of your code with one small, deliberate mistake.
53 If your tests still pass, the mutant "survived" and your tests missed the
54 mistake. If a test fails, the mutant was "killed", which is good.
55  
56 Pattern 1: list the mutants for a file. Nothing is changed on disk.
57  
58 use App::Test::Generator::Mutator;
59  
60 my $mutator = App::Test::Generator::Mutator->new(
61 file => 'lib/My/Module.pm',
62 lib_dir => 'lib',
63 );
64  
65 my $mutants = $mutator->generate_mutants();
66 printf "Generated %d mutants\n", scalar @{$mutants};
67 printf "line %d: %s\n", $_->line, $_->description for @{$mutants};
68  
69 Pattern 2: test every mutant. Run this from the top directory of the
70 project. The mutated file is copied over the real file for the test run
71 and the real file is always put back. This is what
72 C<bin/test-generator-mutate> does.
73  
74 my $file = 'lib/My/Module.pm';
75 my $read = sub { open my $fh, '<', $_[0]; local $/; <$fh> };
76 my $original = $read->($file);
77  
78 for my $m (@{$mutants}) {
79 my $workspace = $mutator->prepare_workspace(); # clean copy each time
80 $mutator->apply_mutant($m);
81 my $mutated = $read->("$workspace/$file");
82  
83 { open my $fh, '>', $file; print $fh $mutated }
84 my $survived = $mutator->run_tests();
85 { open my $fh, '>', $file; print $fh $original }
86  
87 printf "%s: %s\n", ($survived ? 'SURVIVED' : 'KILLED'), $m->description;
88 }
89  
90 Pattern 3: use C<fast> mode to get fewer mutants and a shorter run.
91  
92 my $quick = App::Test::Generator::Mutator->new(
93 file => 'lib/My/Module.pm',
94 mutation_level => 'fast',
95 );
96 my $fewer = $quick->generate_mutants();
97 printf "fast: %d mutants, full: %d mutants\n",
98 scalar @{$fewer}, scalar @{$mutants};
99  
100 Pattern 4: protect lines of your own code from mutation. Put these two
101 comment lines around them in the file to be mutated. Each marker must
102 be alone on its line.
103  
104 ## MUTANT_SKIP_BEGIN
105 return 1 if $n <= 1; # no mutant will be made for this line
106 ## MUTANT_SKIP_END
107  
108 =head1 DESCRIPTION
109  
110 B<App::Test::Generator::Mutator> is a mutation engine that programmatically
111 alters Perl source files to evaluate the effectiveness of a project's test
112 suite. It analyses modules, generates systematic code mutations (such as
113 conditional inversions, logical operator changes, and numeric boundary
114 flips), and applies them within an isolated workspace so tests can be
115 executed safely against each modified variant.
116  
117 By tracking which mutants are killed (cause tests to fail) versus those that
118 survive (tests still pass), the module enables calculation of a mutation
119 score, providing a quantitative measure of how well the test suite detects
120 unintended behavioural changes.
121  
122 =head2 new
123  
124 Construct a new Mutator for a given source file.
125  
126 my $mutator = App::Test::Generator::Mutator->new(
127 file => 'lib/My/Module.pm',
128 lib_dir => 'lib',
129 mutation_level => 'full',
130 );
131  
132 =head3 Arguments
133  
134 =over 4
135  
136 =item * C<file>
137  
138 Path to the Perl source file to mutate. Required. Must exist on disk.
139  
140 =item * C<lib_dir>
141  
142 Root library directory. Optional - defaults to C<lib>.
143 Use a relative path. C<file> must begin with this path.
144  
145 =item * C<mutation_level>
146  
147 Controls the breadth of mutation. C<full> applies all mutations;
148 C<fast> deduplicates and removes redundant mutants first.
149 Optional - defaults to C<full>.
150  
151 =back
152  
153 =head3 Returns
154  
155 A blessed hashref. Croaks if C<file> is missing or does not exist.
156 The file is only checked here. It is read later, by C<generate_mutants>.
157  
158 =head3 API SPECIFICATION
159  
160 =head4 input
161  
162 {
163 file => { type => SCALAR },
164 lib_dir => { type => SCALAR, optional => 1 },
165 mutation_level => { type => SCALAR, optional => 1 },
166 }
167  
168 =head4 output
169  
170 {
171 type => OBJECT,
172 isa => 'App::Test::Generator::Mutator',
173 }
174  
175 =head3 EXAMPLE
176  
177 my $m = App::Test::Generator::Mutator->new(
178 file => 'lib/Acme/Widget.pm',
179 mutation_level => 'fast',
180 );
181  
182 =head3 MESSAGES
183  
184 =over 4
185  
186 =item C<file required>
187  
188 C<file> was not supplied.
189  
190 =item C<< file not found: PATH >>
191  
192 C<file> was supplied but does not exist on disk.
193  
194 =back
195  
196 The mathematical description is in L</FORMAL SPECIFICATION>.
197  
198 =cut
199  
200 384 ▶sub new {
Subroutine new: called CC=5 SCAR=16.2
Pod documented
201 384 my ($class, %args) = @_;
202  
203 # file is required and must exist on disk
204 384 ▶ croak $ERR_FILE_REQUIRED unless defined $args{file};
Branch: 1 branch
BranchTrueFalseTotal
unless defined $args{"file"} 6 378 384
205 378 ▶ croak "file not found: $args{file}" unless -f $args{file};
Branch: 1 branch
BranchTrueFalseTotal
unless -f $args{"file"} 7 371 378
206  
207 return bless {
208 file => $args{file},
209 lib_dir => $args{lib_dir} || $DEFAULT_LIB_DIR,
210 371 ▶ mutation_level => $args{mutation_level} || $DEFAULT_MUTATION_LEVEL,
Condition66%4/6
$args{'lib_dir'} || $DEFAULT_LIB_DIR
l!l&&r!l&&!r
1951760
$args{'mutation_level'} || $DEFAULT_MUTATION_LEVEL
l!l&&r!l&&!r
1572140
MC/DC0%0/2
$args{'lib_dir'} || $DEFAULT_LIB_DIR
$args{'lib_dir'} $DEFAULT_LIB_DIR
MC/DC0%0/2
$args{'mutation_level'} || $DEFAULT_MUTATION_LEVEL
$args{'mutation_level'} $DEFAULT_MUTATION_LEVEL
Truth table66%2/3
A || B
A= $args{'lib_dir'}B= $DEFAULT_LIB_DIR
ABresult
1X1
011
000
Truth table66%2/3
A || B
A= $args{'mutation_level'}B= $DEFAULT_MUTATION_LEVEL
ABresult
1X1
011
000
211  
212 # Instantiate all registered mutation strategies
213 mutations => [
214 App::Test::Generator::Mutation::BooleanNegation->new(),
215 App::Test::Generator::Mutation::ReturnUndef->new(),
216 App::Test::Generator::Mutation::NumericBoundary->new(),
217 App::Test::Generator::Mutation::ConditionalInversion->new(),
218 ],
219 }, $class;
220 }
221  
222 =head2 generate_mutants
223  
224 Parse the target file and generate all mutants by running each registered
225 mutation strategy against the PPI document.
226  
227 my $mutants = $mutator->generate_mutants(); # scalar context: arrayref
228 for my $m (@{$mutants}) { ... }
229  
230 my @mutants = $mutator->generate_mutants(); # list context: flat list (backward-compat)
231  
232 =head3 Arguments
233  
234 None beyond C<$self>.
235  
236 =head3 Returns
237  
238 An arrayref of L<App::Test::Generator::Mutant> objects. In C<fast> mode,
239 redundant and duplicate mutants are removed before returning.
240 Lines within C<## MUTANT_SKIP_BEGIN> / C<## MUTANT_SKIP_END> annotation
241 blocks are excluded from the candidate list entirely.
242 After this method returns,
243 C<$self-E<gt>{skip_lines}> contains a hashref mapping excluded
244 line numbers to 1.
245  
246 =head3 API SPECIFICATION
247  
248 =head4 input
249  
250 {
251 self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
252 }
253  
254 =head4 output
255  
256 {
257 type => ARRAYREF,
258 elements => { type => OBJECT, isa => 'App::Test::Generator::Mutant' },
259 }
260  
261 =head3 EXAMPLE
262  
263 my $mutants = $mutator->generate_mutants();
264 printf "%d mutants generated\n", scalar @{$mutants};
265  
266 for my $m (@{$mutants}) {
267 printf "line %d: %s\n", $m->line, $m->description;
268 }
269  
270 =head3 MESSAGES
271  
272 =over 4
273  
274 =item C<< Unable to parse FILE >>
275  
276 PPI could not parse the source file (syntax error or unreadable file).
277 FILE is the path passed to C<new>.
278  
279 =item C<< FILE: MUTANT_SKIP_BEGIN at line N with no prior MUTANT_SKIP_END >>
280  
281 A C<## MUTANT_SKIP_BEGIN> marker was found while already inside a skip block.
282  
283 =item C<< FILE: MUTANT_SKIP_END at line N with no matching MUTANT_SKIP_BEGIN >>
284  
285 A C<## MUTANT_SKIP_END> marker was found with no preceding C<## MUTANT_SKIP_BEGIN>.
286  
287 =item C<< FILE: MUTANT_SKIP_BEGIN at line N has no matching MUTANT_SKIP_END >>
288  
289 The source file ended while still inside a skip block.
290  
291 =back
292  
293 The mathematical description is in L</FORMAL SPECIFICATION>.
294  
295 =head3 PSEUDOCODE
296  
297 parse file with PPI
298 scan lines for MUTANT_SKIP_BEGIN / MUTANT_SKIP_END pairs, giving skip_lines
299 for each registered mutation strategy:
300 skip if strategy does not apply_to(doc)
301 for each mutant from strategy->mutate(doc):
302 include unless mutant.line is in skip_lines
303 if mutation_level == 'fast':
304 deduplicate and remove redundant mutants
305 return arrayref (or flat list in list context)
306  
307 =cut
308  
309 73 ▶sub generate_mutants {
Subroutine generate_mutants: called CC=13 SCAR=25.7
Pod documented
310 73 my $self = $_[0];
311  
312 # Parse the target file into a PPI document
313 73 ▶ my $doc = PPI::Document->new($self->{file}) or croak "Unable to parse $self->{file}";
Branch: 1 branch
BranchTrueFalseTotal
my $doc = "PPI::Document"->new($self->{"file"}) or croak(qq[Unable to parse $self->{"file"}]) 73 0 73
314  
315 # Build set of lines excluded by ## MUTANT_SKIP_BEGIN / ## MUTANT_SKIP_END
316 73 my %skip_lines;
317 73 my $in_skip = 0;
318 73 my $skip_start = 0;
319 73 my $line_num = 0;
320  
321 73 for my $line (split /\n/, $doc->serialize()) {
322 1847 $line_num++;
323  
324 # Match only lines where the annotation is the entire content —
325 # prevents false positives in comments or POD that mention the tag
326 1847 ▶ if($line =~ /^\s*##\s*MUTANT_SKIP_BEGIN\s*$/) {
Branch: 1 branch
BranchTrueFalseTotal
if ($line =~ /^\s*##\s*MUTANT_SKIP_BEGIN\s*$/u) 10 1837 1847
327 10 ▶ croak "$self->{file}: MUTANT_SKIP_BEGIN at line $line_num with no prior MUTANT_SKIP_END"
Branch: 1 branch
BranchTrueFalseTotal
if $in_skip 2 8 10
328 if $in_skip;
329 8 $in_skip = 1;
330 8 $skip_start = $line_num;
331 }
332 1845 ▶ $skip_lines{$line_num} = 1 if $in_skip;
Branch: 1 branch
BranchTrueFalseTotal
if $in_skip 24 1821 1845
333  
334 # Match only lines where the annotation is the entire content —
335 # prevents false positives in comments or POD that mention the tag
336 1845 ▶ if($line =~ /^\s*##\s*MUTANT_SKIP_END\s*$/) {
Branch: 1 branch
BranchTrueFalseTotal
if ($line =~ /^\s*##\s*MUTANT_SKIP_END\s*$/u) 6 1839 1845
337 6 ▶ croak "$self->{file}: MUTANT_SKIP_END at line $line_num with no matching MUTANT_SKIP_BEGIN"
Branch: 1 branch
BranchTrueFalseTotal
unless $in_skip 3 3 6
338 unless $in_skip;
339 3 $in_skip = 0;
340 }
341 }
342 # Unclosed MUTANT_SKIP_BEGIN is fatal
343 68 ▶ croak "$self->{file}: MUTANT_SKIP_BEGIN at line $skip_start has no matching MUTANT_SKIP_END" if $in_skip;
Branch: 1 branch
BranchTrueFalseTotal
if $in_skip 3 65 68
344  
345 # Store skip lines for use by the report generator
346 65 $self->{skip_lines} = \%skip_lines;
347  
348 65 my @mutants;
349  
350 # Run each registered mutation strategy against the document,
351 # excluding any candidates on skip-annotated lines. applies_to()
352 # is a cheap pre-filter -- skip the mutate() walk entirely for
353 # strategies that have nothing to match in this document.
354 65 for my $mutation (@{$self->{mutations}}) {
355 260 ▶ next unless $mutation->applies_to($doc);
Branch: 1 branch
BranchTrueFalseTotal
unless $mutation->applies_to($doc) 60 200 260
356 200 push @mutants, grep { !$skip_lines{$_->line} } $mutation->mutate($doc);
357 }
358  
359 # In fast mode deduplicate and remove redundant mutants before returning.
360 # Returns arrayref in scalar context and a flat list in list context so
361 # existing callers (my @m = generate_mutants()) continue to work unchanged.
362 65 ▶ my $result = $self->{mutation_level} eq $LEVEL_FAST
Branch: 1 branch
BranchTrueFalseTotal
$self->{'mutation_level'} eq $LEVEL_FAST ? : 16 49 65
363 ? _dedup_mutants(\@mutants)
364 : \@mutants;
365  
366 65 ▶ return wantarray ? @{$result} : $result;
Branch: 1 branch
BranchTrueFalseTotal
wantarray ? : 54 11 65
367 }
368  
369 =head2 prepare_workspace
370  
371 Prepare an isolated temporary workspace for a single mutant.
372  
373 The entire C<lib_dir> tree is copied into the workspace so that all module
374 dependencies resolve correctly when the test suite runs against the mutant.
375 Only after this copy is complete is the single target file overwritten by
376 C<apply_mutant>.
377  
378 my $workspace = $mutator->prepare_workspace();
379 $mutator->apply_mutant($mutant);
380 local $ENV{PERL5LIB} = "$workspace/lib";
381 my $survived = (system('prove', 't') == 0);
382  
383 =head3 Arguments
384  
385 None beyond C<$self>.
386  
387 =head3 Returns
388  
389 A string containing the absolute path to the temporary directory created.
390 The directory is removed automatically when the program ends
391 (L<File::Temp>'s C<CLEANUP =E<gt> 1> behaviour). It is not removed earlier.
392  
393 =head3 Side effects
394  
395 Creates a temporary directory. Recursively copies C<lib_dir> into it.
396 Sets C<< $self->{_workspace} >>, C<< $self->{_relative} >>, and
397 C<< $self->{_lib_basename} >>. Does not modify C<< $self->{lib_dir} >>.
398  
399 =head3 Notes
400  
401 Call C<prepare_workspace> again before B<each> C<apply_mutant> call.
402 C<apply_mutant> does not undo the previous mutant, so two mutants applied
403 to the same workspace are both present in the file.
404  
405 Each call makes a new directory and a new full copy of C<lib_dir>. The
406 object then points at the new one. Older workspaces stay on disk until the
407 program ends, so a long run needs disk space for one copy per call.
408  
409 =head3 API SPECIFICATION
410  
411 =head4 input
412  
413 {
414 self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
415 }
416  
417 =head4 output
418  
419 {
420 type => SCALAR,
421 }
422  
423 =head3 EXAMPLE
424  
425 my $workspace = $mutator->prepare_workspace();
426 # workspace is an absolute temp dir path
427 # original lib_dir value is unchanged
428 printf "original lib_dir still: %s\n", $mutator->{lib_dir};
429  
430 =head3 MESSAGES
431  
432 =over 4
433  
434 =item C<dircopy failed: $!>
435  
436 The C<lib_dir> tree could not be copied into the temporary workspace directory,
437 usually a permissions error.
438  
439 =back
440  
441 The mathematical description is in L</FORMAL SPECIFICATION>.
442  
443 =cut
444  
445 27 ▶sub prepare_workspace {
Subroutine prepare_workspace: called CC=2 SCAR=6.9
Pod documented
446 27 my $self = $_[0];
447  
448 # Create a self-cleaning temporary directory
449 27 my $tmp = tempdir(CLEANUP => 1);
450  
451 # Normalise lib_dir to its final component so workspace paths
452 # are relative regardless of whether an absolute path was passed in
453 27 my $lib_basename = (File::Spec->splitdir($self->{lib_dir}))[-1];
454  
455 # Derive the file's path relative to lib_dir for use by apply_mutant
456 27 my $relative = $self->{file};
457 27 $relative =~ s/^\Q$self->{lib_dir}\E\/?//;
458  
459 # Copy the entire lib tree so all dependencies resolve in the workspace
460 27 ▶ dircopy($self->{lib_dir}, File::Spec->catfile($tmp, $lib_basename)) or croak "dircopy failed: $!";
Branch: 1 branch
BranchTrueFalseTotal
dircopy($self->{"lib_dir"}, "File::Spec"->catfile($tmp, $lib_basename)) or croak("dircopy failed: $!") 26 1 27
461  
462 # Store normalised state under private keys — do NOT mutate lib_dir,
463 # which callers may inspect after this call expecting the original value.
464 26 $self->{_workspace} = $tmp;
465 26 $self->{_relative} = $relative;
466 26 $self->{_lib_basename} = $lib_basename;
467  
468 26 return $tmp;
469 }
470  
471 =head2 apply_mutant
472  
473 Apply a single mutant's transform to the target file in the workspace.
474  
475 $mutator->apply_mutant($mutant);
476  
477 =head3 Arguments
478  
479 =over 4
480  
481 =item * C<$mutant>
482  
483 An L<App::Test::Generator::Mutant> object whose C<transform> closure
484 will be applied to the workspace copy of the target file.
485  
486 =back
487  
488 =head3 Returns
489  
490 Nothing useful. Do not test the return value.
491 Modifies the workspace copy of the target file in place.
492  
493 =head3 Side effects
494  
495 Overwrites the target file in the workspace with the mutated version.
496 The real project file is never touched by this method.
497  
498 =head3 API SPECIFICATION
499  
500 =head4 input
501  
502 {
503 self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
504 mutant => { type => OBJECT, isa => 'App::Test::Generator::Mutant' },
505 }
506  
507 =head4 output
508  
509 { type => UNDEF }
510  
511 =head3 EXAMPLE
512  
513 for my $m (@{$mutants}) {
514 my $workspace = $mutator->prepare_workspace();
515 $mutator->apply_mutant($m);
516 # "$workspace/lib/My/Module.pm" now holds exactly one mutation
517 }
518  
519 =head3 MESSAGES
520  
521 =over 4
522  
523 =item C<Workspace not prepared -- call prepare_workspace first>
524  
525 C<apply_mutant> was called before C<prepare_workspace>.
526  
527 =item C<Relative path not set -- call prepare_workspace first>
528  
529 Internal: the relative-path field was not set by C<prepare_workspace>.
530  
531 =item C<< Failed to parse TARGET >>
532  
533 PPI could not parse the workspace copy of the target file.
534  
535 =item C<Can't call method "transform" on an undefined value>
536  
537 A Perl error, not a message from this module. C<$mutant> was C<undef>.
538  
539 =back
540  
541 The mathematical description is in L</FORMAL SPECIFICATION>.
542  
543 =cut
544  
545 24 ▶sub apply_mutant {
Subroutine apply_mutant: called CC=4 SCAR=14.0
Pod documented
546 24 my ($self, $mutant) = @_;
547  
548 # Workspace must be prepared before applying any mutant
549 my $workspace = $self->{_workspace}
550 24 ▶ or croak $ERR_WORKSPACE_NOT_SET;
Branch: 1 branch
BranchTrueFalseTotal
my $workspace = $self->{"_workspace"} or croak($ERR_WORKSPACE_NOT_SET) 16 8 24
551  
552 my $relative = $self->{_relative}
553 16 ▶ or croak $ERR_RELATIVE_NOT_SET;
Branch: 1 branch
BranchTrueFalseTotal
my $relative = $self->{"_relative"} or croak($ERR_RELATIVE_NOT_SET) 16 0 16
554  
555 # Construct the full path to the file in the workspace
556 my $target = File::Spec->catfile(
557 $workspace,
558 $self->{_lib_basename},
559 16 $relative,
560 );
561  
562 # Parse the workspace copy and apply the mutation transform
563 16 ▶ my $doc = PPI::Document->new($target) or croak "Failed to parse $target";
Branch: 1 branch
BranchTrueFalseTotal
my $doc = "PPI::Document"->new($target) or croak("Failed to parse $target") 16 0 16
564  
565 16 $mutant->transform->($doc);
566  
567 16 $doc->save($target);
568 }
569  
570 =head2 run_tests
571  
572 Run the project's test suite and return whether all tests passed.
573  
574 my $survived = $mutator->run_tests();
575  
576 =head3 Arguments
577  
578 None beyond C<$self>.
579  
580 =head3 Returns
581  
582 A true value (1) if all tests passed (mutant survived). A false value
583 (the empty string, not 0) if any test failed or C<prove> could not be
584 started (mutant killed). Test it with C<if>, not with C<== 0>.
585  
586 =head3 Side effects
587  
588 Executes an external process running the test suite. The output of
589 C<prove> goes to the terminal.
590  
591 =head3 Notes
592  
593 Runs C<prove -l t> in the current directory. This tests the F<lib>
594 directory of the project, B<not> the workspace. To test a mutant, copy
595 the mutated file from the workspace over the project file first, and put
596 the original back afterwards. See Pattern 2 in the L</SYNOPSIS>.
597  
598 The C<prove> program is looked for next to the running C<perl> first,
599 then in Perl's own C<bin> directory, and only then on C<PATH>.
600  
601 There is no time limit. A mutant that causes an endless loop makes this
602 call wait for ever.
603  
604 =head3 API SPECIFICATION
605  
606 =head4 input
607  
608 {
609 self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
610 }
611  
612 =head4 output
613  
614 { type => SCALAR }
615  
616 =head3 EXAMPLE
617  
618 my $survived = $mutator->run_tests();
619 if($survived) {
620 print "mutant survived\n";
621 } else {
622 print "mutant killed\n";
623 }
624  
625 The mathematical description is in L</FORMAL SPECIFICATION>.
626  
627 =cut
628  
629 6 ▶sub run_tests {
Subroutine run_tests: called CC=3 SCAR=11.2
Pod documented
630 6 my $self = $_[0];
631  
632 # Derive prove from $^X so CPAN smokers that install to a non-PATH
633 # location still resolve the correct perl/prove pair. Config{bin}
634 # is a reliable fallback; bare 'prove' is last resort only.
635 6 my ($vol, $dir) = File::Spec->splitpath($^X);
636 6 my $prove = File::Spec->catpath($vol, $dir, 'prove');
637 6 ▶ $prove = File::Spec->catfile($Config{bin}, 'prove') unless -x $prove;
Branch: 1 branch
BranchTrueFalseTotal
unless -x $prove 0 6 6
638 6 ▶ $prove = 'prove' unless -x $prove;
Branch: 1 branch
BranchTrueFalseTotal
unless -x $prove 0 6 6
639  
640 6 return system($prove, '-l', 't') == 0;
641 }
642  
643 # --------------------------------------------------
644 # _dedup_mutants
645 #
646 # Purpose: Remove duplicate and redundant mutants
647 # from a list, used in fast mutation mode
648 # to reduce the number of mutants to run.
649 #
650 # Entry: $mutants - arrayref of Mutant objects.
651 #
652 # Exit: Returns an arrayref of deduplicated
653 # Mutant objects.
654 #
655 # Side effects: None.
656 #
657 # Notes: Deduplication key uses line, original,
658 # and description rather than the transform
659 # coderef, which is not stable as a string.
660 # --------------------------------------------------
661 38 ▶sub _dedup_mutants {
Subroutine _dedup_mutants: called CC=7 SCAR=19.5
662 38 my ($mutants) = @_;
663 38 my @rc;
664 my %seen;
665  
666 38 for my $m (@{$mutants}) {
667 # Build a stable key from metadata — not from the coderef
668 194 ▶ my $key = join '|',
Condition100%6/6
$m->line // ""
l!l
1922
$m->original // ""
l!l
1922
$m->description // ""
l!l
1922
MC/DC100%1/1
$m->line // ""
$m->line
MC/DC100%1/1
$m->original // ""
$m->original
MC/DC100%1/1
$m->description // ""
$m->description
Truth table100%2/2
A
A= $m->line
Aresult
11
00
Truth table100%2/2
A
A= $m->original
Aresult
11
00
Truth table100%2/2
A
A= $m->description
Aresult
11
00
669 $m->line // '',
670 $m->original // '',
671 $m->description // '';
672  
673 194 ▶ next if $seen{$key}++;
Branch: 1 branch
BranchTrueFalseTotal
if $seen{$key}++ 7 187 194
674 187 ▶ next if _is_redundant_mutation($m);
Branch: 1 branch
BranchTrueFalseTotal
if _is_redundant_mutation($m) 12 175 187
675  
676 175 push @rc, $m;
677 }
678  
679 38 return \@rc;
680 }
681  
682 # --------------------------------------------------
683 # _is_redundant_mutation
684 #
685 # Return true if a mutant is considered
686 # redundant and should be skipped in fast
687 # mutation mode.
688 #
689 # Entry: $m - a Mutant object.
690 #
691 # Exit: Returns 1 if redundant, 0 otherwise.
692 #
693 # Notes: Checks for arithmetic no-ops, double
694 # negation inside conditionals, boolean
695 # literal flips, mutations inside comments,
696 # and equivalent numeric comparisons.
697 # Does not compare transform coderefs —
698 # they are not meaningful as strings.
699 # --------------------------------------------------
700 216 ▶sub _is_redundant_mutation {
Subroutine _is_redundant_mutation: called CC=10 SCAR=23.0
701 216 my ($m) = @_;
702  
703 216 ▶ my $orig = $m->original // '';
Condition100%2/2
$m->original // ""
l!l
2142
MC/DC100%1/1
$m->original // ""
$m->original
Truth table100%2/2
A
A= $m->original
Aresult
11
00
704  
705 # Arithmetic no-ops add nothing to mutation coverage
706 216 ▶ return 1 if $orig =~ /\+\s*0$/;
Branch: 1 branch
BranchTrueFalseTotal
if $orig =~ /\+\s*0$/u 7 209 216
707 209 ▶ return 1 if $orig =~ /-\s*0$/;
Branch: 1 branch
BranchTrueFalseTotal
if $orig =~ /-\s*0$/u 6 203 209
708  
709 # Double negation inside conditionals forces boolean context
710 # in Perl and is not a meaningful mutation
711 203 ▶ if($m->context && $m->context eq 'conditional') {
Branch: 1 branch
BranchTrueFalseTotal
if ($m->context and $m->context eq "conditional") 58 145 203
Condition100%3/3
$m->context and $m->context eq "conditional"
!ll&&!rl&&r
489758
MC/DC100%2/2
$m->context and $m->context eq "conditional"
$m->context $m->context eq "conditional"
Truth table100%3/3
A and B
A= $m->contextB= $m->context eq "conditional"
ABresult
0X0
100
111
712 58 ▶ return 1 if $orig =~ /^\!\!/;
Branch: 1 branch
BranchTrueFalseTotal
if $orig =~ /^\!\!/u 3 55 58
713 }
714  
715 # Boolean literal flip on a standalone 1 or 0 is trivial
716 200 ▶ return 1 if $orig =~ /^\s*(?:1|0)\s*$/;
Branch: 1 branch
BranchTrueFalseTotal
if $orig =~ /^\s*(?:1|0)\s*$/u 11 189 200
717  
718 # Mutations inside comments are unreachable code
719 189 ▶ return 1 if $m->line_content && $m->line_content =~ /^\s*#/;
Branch: 1 branch
BranchTrueFalseTotal
if $m->line_content and $m->line_content =~ /^\s*#/u 4 185 189
Condition100%3/3
$m->line_content and $m->line_content =~ /^\s*#/u
!ll&&!rl&&r
301554
MC/DC100%2/2
$m->line_content and $m->line_content =~ /^\s*#/u
$m->line_content $m->line_content =~ /^\s*#/u
Truth table100%3/3
A and B
A= $m->line_contentB= $m->line_content =~ /^\s*#/u
ABresult
0X0
100
111
720  
721 185 return 0;
722 }
723  
724 =head1 COMMON PITFALLS
725  
726 =over 4
727  
728 =item Calling C<apply_mutant> before C<prepare_workspace>
729  
730 C<apply_mutant> requires C<prepare_workspace> to have been called first. The
731 workspace holds the isolated copy of C<lib/> that receives mutations.
732  
733 =item Passing an absolute path as C<lib_dir>
734  
735 C<lib_dir> must be a B<relative> path (e.g. C<lib>). An absolute path causes
736 C<apply_mutant> to construct a doubled directory under the workspace and then
737 fail to find the target file.
738  
739 =item Checking C<< $mutator->{workspace} >> after the refactor
740  
741 Internal workspace state is stored in C<_workspace>, C<_relative>, and
742 C<_lib_basename> (note the underscore prefix). The original C<lib_dir> value
743 is never overwritten. Old code that reads C<< $mutator->{workspace} >> or
744 C<< $mutator->{relative} >> (without underscores) will not find these keys.
745  
746 =item Forgetting that C<run_tests> drives C<prove> from C<$^X>
747  
748 C<run_tests> resolves C<prove> from the same Perl binary used to run the script.
749 If you shell out to C<prove> directly in your own integration code, make sure
750 you are using the matching C<prove>.
751  
752 =item C<generate_mutants> in list vs scalar context
753  
754 C<generate_mutants> returns a flat list in list context and an arrayref in
755 scalar context. Assign to an arrayref (C<my $m = $mutator-E<gt>generate_mutants()>)
756 to guarantee you always get a reference regardless of calling context.
757  
758 =item Applying two mutants to one workspace
759  
760 C<apply_mutant> changes the workspace file and leaves it changed. If you
761 apply a second mutant without a new C<prepare_workspace> call, the file
762 holds both mutations and your results are wrong.
763  
764 =item Expecting C<run_tests> to test the workspace
765  
766 C<run_tests> runs the tests against the project's own F<lib> directory.
767 If you only call C<apply_mutant> and then C<run_tests>, the tests load the
768 original code, every mutant "survives", and the kill rate is 0%.
769  
770 =item An unknown C<mutation_level> is not an error
771  
772 Only the exact text C<fast> turns on fast mode. Any other value, for
773 example C<Fast> or C<quick>, is accepted silently and works like C<full>.
774  
775 =item How C<undef> and other false values are handled
776  
777 =over 4
778  
779 =item * C<< file => undef >> makes C<new> croak with C<file required>.
780 An empty string makes it croak with C<file not found>.
781  
782 =item * C<lib_dir> and C<mutation_level> use their default when the value
783 is C<undef>, an empty string or C<0>. You cannot have a library directory
784 that is really named C<0>.
785  
786 =item * C<apply_mutant(undef)> stops with a Perl error, after the
787 workspace check.
788  
789 =item * C<run_tests> returns an empty string for "failed". It never
790 returns C<undef>.
791  
792 =item * A mutant's C<line>, C<original> or C<description> may be C<undef>.
793 Fast mode treats each missing value as an empty string when it looks for
794 duplicates.
795  
796 =back
797  
798 =item Extra arguments to C<new> are ignored
799  
800 A wrong spelling such as C<libdir> or C<level> is not reported. The
801 default value is used and no warning is given.
802  
803 =item Skip markers must be alone on their line
804  
805 C<## MUTANT_SKIP_BEGIN> and C<## MUTANT_SKIP_END> are only seen when
806 nothing else except spaces is on the same line. Markers cannot be nested.
807 The two marker lines themselves are also skipped.
808  
809 =back
810  
811 =head1 ENCODING
812  
813 =over 4
814  
815 =item * The C<file> and C<lib_dir> arguments
816  
817 Paths are passed to the operating system unchanged. ASCII paths always
818 work. For non-ASCII directory or file names, pass the path as bytes in
819 the encoding of your file system (normally UTF-8).
820  
821 =item * Source file contents
822  
823 The file to mutate is read and written by L<PPI> as raw bytes. UTF-8
824 text, other non-ASCII text and emojis inside strings, comments and POD
825 are copied to the mutated file byte for byte. They are not damaged and
826 they do not need to be decoded first.
827  
828 =item * Text returned by a mutant
829  
830 C<< $mutant->original >>, C<< $mutant->description >> and
831 C<< $mutant->line_content >> can contain pieces of your source code.
832 These are byte strings. If the source is UTF-8 and you print them through
833 a UTF-8 file handle, decode them first with
834 C<Encode::decode('UTF-8', $text)>.
835  
836 =item * Skip markers
837  
838 The markers are ASCII. They are found correctly in a file of any
839 ASCII-compatible encoding, including UTF-8.
840  
841 =item * Error messages
842  
843 Messages contain the file path exactly as you gave it.
844  
845 =back
846  
847 =head1 LIMITATIONS
848  
849 =over 4
850  
851 =item * Single strategy registry
852  
853 The four built-in mutation strategies are hardcoded in C<new>. There is no
854 plugin mechanism for registering additional strategies without subclassing.
855 A future version should accept a C<strategies> arrayref argument.
856  
857 =item * No parallelism
858  
859 C<run_tests> is synchronous. For large test suites or large mutant sets,
860 wall-clock time scales linearly. The in-place mutation strategy also
861 serialises all mutants behind a single file lock.
862  
863 =item * PPI re-parse per apply_mutant call
864  
865 C<apply_mutant> re-parses the workspace copy of the target file for every
866 mutant. For very large single-file modules the PPI parse time may dominate.
867  
868 =item * apply_mutant does not restore on abnormal exit
869  
870 If the process is killed between the write and restore in
871 C<bin/test-generator-mutate>, the project file is left mutated.
872 A C<git restore lib/...> recovers it.
873  
874 =back
875  
876 =head1 SEE ALSO
877  
878 =over 4
879  
880 =item C<bin/test-generator-mutate>
881  
882 =item L<Devel::Mutator>
883  
884 =back
885  
886 =head1 AUTHOR
887  
888 Nigel Horne, C<< <njh at nigelhorne.com> >>
889  
890 =head1 FORMAL SPECIFICATION
891  
892 This section describes each method in Z notation. It is for readers who
893 want an exact, mathematical statement. You do not need it to use the module.
894  
895 How to read it: a name ending in C<?> is an input, a name ending in C<!> is
896 an output, and a name ending in C<'> is the value after the call.
897 E<0x0394> before the state name means "this call may change the object".
898 E<0x039E> means "this call does not change the object".
899  
900 LEVEL == { 'full', 'fast' }
901  
902 ┌─ Mutator ─────────────────────────────────────────
903 │ file, lib_dir : PATH
904 │ mutation_level : STRING
905 │ skip_lines : ℙ ℕ₁ ∪ { undef }
906 │ workspace : PATH ∪ { undef }
907 │ relative : PATH ∪ { undef }
908 ├───────────────────────────────────────────────────
909 │ workspace = undef ⇔ relative = undef
910 └───────────────────────────────────────────────────
911  
912 =head2 new
913  
914 ┌─ New ─────────────────────────────────────────────
915 │ Mutator′
916 │ file? : VALUE
917 │ lib_dir?, mutation_level? : VALUE
918 │ result! : Mutator
919 ├───────────────────────────────────────────────────
920 │ pre file? ≠ undef ∧ file? names an existing plain file
921 │ post file′ = file?
922 │ lib_dir′ = if lib_dir? is true
923 │ then lib_dir? else 'lib'
924 │ mutation_level′ = if mutation_level? is true
925 │ then mutation_level? else 'full'
926 │ skip_lines′ = undef ∧ workspace′ = undef ∧ relative′ = undef
927 │ result! = θ Mutator′
928 └───────────────────────────────────────────────────
929  
930 file? = undef ⇒ croak 'file required'
931 file? is not an existing file ⇒ croak 'file not found: ' ⁀ file?
932  
933 =head2 generate_mutants
934  
935 ┌─ GenerateMutants ─────────────────────────────────
936 │ Δ Mutator
937 │ result! : seq Mutant
938 │ all : seq Mutant
939 ├───────────────────────────────────────────────────
940 │ pre file is valid Perl that PPI can parse
941 │ skip markers in file are paired and not nested
942 │ post skip_lines′ = { n : ℕ₁ | line n of file lies between
943 │ a BEGIN marker and its END marker,
944 │ both marker lines included }
945 │ all = every mutant that the four strategies
946 │ produce for file, in strategy order
947 │ ∀ m : ran result! • m ∈ ran all ∧ m.line ∉ skip_lines′
948 │ mutation_level ≠ 'fast' ⇒
949 │ result! = all with skipped lines removed
950 │ mutation_level = 'fast' ⇒
951 │ (∀ i, j : dom result! • i ≠ j ⇒
952 │ key(result!(i)) ≠ key(result!(j)))
953 │ ∧ (∀ m : ran result! • ¬ redundant(m))
954 │ file′ = file ∧ lib_dir′ = lib_dir
955 │ workspace′ = workspace ∧ relative′ = relative
956 └───────────────────────────────────────────────────
957  
958 key(m) == (m.line, m.original, m.description)
959  
960 In list context the members of result! are returned as a list.
961  
962 =head2 prepare_workspace
963  
964 ┌─ PrepareWorkspace ────────────────────────────────
965 │ Δ Mutator
966 │ result! : PATH
967 ├───────────────────────────────────────────────────
968 │ pre lib_dir names a directory that can be copied
969 │ post result! names a directory created by this call
970 │ result! ≠ workspace
971 │ result!/basename(lib_dir) is a full copy of lib_dir
972 │ workspace′ = result!
973 │ relative′ = file with the leading lib_dir removed
974 │ file′ = file ∧ lib_dir′ = lib_dir
975 │ mutation_level′ = mutation_level
976 │ skip_lines′ = skip_lines
977 └───────────────────────────────────────────────────
978  
979 the copy fails ⇒ croak 'dircopy failed: ' ⁀ reason
980  
981 =head2 apply_mutant
982  
983 ┌─ ApplyMutant ─────────────────────────────────────
984 │ Ξ Mutator
985 │ mutant? : Mutant
986 │ target : PATH
987 ├───────────────────────────────────────────────────
988 │ pre workspace ≠ undef ∧ relative is true
989 │ mutant?.transform is a CODE reference
990 │ target = workspace/basename(lib_dir)/relative
991 │ target is valid Perl that PPI can parse
992 │ post content′(target) = mutant?.transform(content(target))
993 │ content′(file) = content(file)
994 └───────────────────────────────────────────────────
995  
996 workspace = undef ⇒ croak 'Workspace not prepared -- ...'
997 relative is false ⇒ croak 'Relative path not set -- ...'
998 target not parsed ⇒ croak 'Failed to parse ' ⁀ target
999  
1000 =head2 run_tests
1001  
1002 ┌─ RunTests ────────────────────────────────────────
1003 │ Ξ Mutator
1004 │ result! : { 1, '' }
1005 ├───────────────────────────────────────────────────
1006 │ post result! = 1 ⇔ the command 'prove -l t', run in the
1007 │ current directory, exits with status 0
1008 └───────────────────────────────────────────────────
1009  
1010 =head1 STATE DIAGRAM
1011  
1012 The object has three states. The state says what has been prepared so far.
1013  
1014 new(file => undef) or new(file => missing file)
1015 +--------------------------------------------------+
1016 | action: croak, no object |
1017 v |
1018 +-------------+ |
1019 | NO OBJECT |--------------------------------------+
1020 +-------------+
1021 |
1022 | new(file => existing file)
1023 | action: store file, lib_dir, mutation_level;
1024 | build the four mutation strategies
1025 v
1026 +---------------------+ generate_mutants()
1027 | |-----------------------------------------+
1028 | CONSTRUCTED | action: read file, set skip_lines, |
1029 | (no workspace yet) | return the mutants (state stays same) |
1030 | |<----------------------------------------+
1031 | |
1032 | | run_tests()
1033 | |-----------------------------------------+
1034 | | action: run 'prove -l t', return 1/'' |
1035 | |<----------------------------------------+
1036 | |
1037 | | apply_mutant(m)
1038 | |-----------------------------------------+
1039 | | action: croak 'Workspace not prepared' |
1040 | |<----------------------------------------+
1041 +---------------------+
1042 |
1043 | prepare_workspace()
1044 | action: make temp directory, copy lib_dir into it,
1045 | remember workspace path and relative file path
1046 v
1047 +---------------------+ generate_mutants() (as above)
1048 | |-----------------------------------------+
1049 | WORKSPACE CLEAN |<----------------------------------------+
1050 | (copy is the same |
1051 | as the original) | run_tests() (as above)
1052 | |-----------------------------------------+
1053 | |<----------------------------------------+
1054 | |
1055 | | prepare_workspace()
1056 | |-----------------------------------------+
1057 | | action: new directory, new clean copy |
1058 | |<----------------------------------------+
1059 +---------------------+
1060 | ^
1061 | | prepare_workspace()
1062 | | action: new directory, new clean copy;
1063 | | the mutated copy is left behind
1064 | apply_mutant(m)
1065 | action: rewrite the workspace copy of the file
1066 | with one mutation
1067 v |
1068 +---------------------+ apply_mutant(m2)
1069 | |-----------------------------------------+
1070 | WORKSPACE MUTATED | action: a second mutation is added on |
1071 | | top of the first (usually a mistake) |
1072 | |<----------------------------------------+
1073 | |
1074 | | generate_mutants(), run_tests()
1075 | |-----------------------------------------+
1076 | | action: as above, state stays the same |
1077 | |<----------------------------------------+
1078 +---------------------+
1079  
1080 From any state: when the program ends, every temporary workspace
1081 directory is deleted.
1082  
1083 A call that croaks leaves the object in the state it was in. The
1084 workspace file is only rewritten after the mutation has succeeded.
1085  
1086 No method of this class changes the real project file.
1087  
1088 =head1 LICENCE AND COPYRIGHT
1089  
1090 Copyright 2026 Nigel Horne.
1091  
1092 Usage is subject to the terms of GPL2.
1093 If you use it,
1094 please let me know.
1095  
1096 =cut
1097  
1098 18 1;
« bin/pod-example-tester ↑ Summary lib/App/Test/Generator/Emitter/Perl.pm »