bin/test-generator-mutate

filter: Statementstmt 73.1% 147 / 201 Branchbran 47.5% 38 / 80 Conditioncond 22.5% 7 / 31 MC/DCmcdc 11.7% 2 / 17 Subroutinesub 88.2% 15 / 17 Podpod n/a totaltotal 60.4% 209 / 346 CC 12 SCAR 15.1
CC
12
Cov
62%
__ANON__
12.0
group_mutants
12.0
mutation_difficulty
5.1
CRAP 4.5
« lib/App/Test/Generator/Sample/Module.pm ↑ Summary lib/App/Test/Generator/Planner.pm »
1 #!/usr/bin/env perl
2  
3 4 ▶use strict;
Subroutine BEGIN: called CC=1 SCAR=0.0
4 4 ▶use warnings;
Subroutine BEGIN: called CC=1 SCAR=0.0
5  
6 4 ▶use Carp qw(croak);
Subroutine BEGIN: called CC=1 SCAR=0.0
7 4 ▶use Config;
Subroutine BEGIN: called CC=1 SCAR=0.0
8 4 ▶use Cwd qw(abs_path);
Subroutine BEGIN: called CC=1 SCAR=0.0
9 4 ▶use File::Find qw(find);
Subroutine BEGIN: called CC=1 SCAR=0.0
10 4 ▶use File::Spec;
Subroutine BEGIN: called CC=1 SCAR=0.0
11 4 ▶use Getopt::Long qw(GetOptions);
Subroutine BEGIN: called CC=1 SCAR=0.0
12 4 ▶use JSON::MaybeXS;
Subroutine BEGIN: called CC=1 SCAR=0.0
13 4 ▶use Pod::Usage;
Subroutine BEGIN: called CC=1 SCAR=0.0
14 4 ▶use Readonly;
Subroutine BEGIN: called CC=1 SCAR=0.0
15 4 ▶use Time::HiRes qw(time);
Subroutine BEGIN: called CC=1 SCAR=0.0
16  
17 4 ▶use App::Test::Generator::Mutator;
Subroutine BEGIN: called CC=1 SCAR=0.0
18  
19 =pod
20  
21 =head1 NAME
22  
23 test-generator-mutate - Run mutation testing against a Perl test suite
24  
25 =head1 SYNOPSIS
26  
27 test-generator-mutate [options]
28  
29 test-generator-mutate --lib lib --tests t
30 test-generator-mutate --file lib/My/Module.pm
31 test-generator-mutate --json mutation.json
32 test-generator-mutate --min-score 75
33  
34 =head1 QUICK START
35  
36 test-generator-mutate --lib lib --min-score 85 --json mutation.json
37  
38 =head2 Skipping Lines from Mutation
39  
40 Some routines involve logic that must not be mutated because doing so
41 would cause the test suite to hang rather than produce a clean
42 pass/fail result. Two common cases:
43  
44 =over 4
45  
46 =item Recursion base cases
47  
48 The C<ConditionalInversion> and C<BooleanNegation> operators will negate
49 any C<if> condition, including the guard that terminates a recursive
50 function. A mutant that removes the base case causes infinite recursion
51 and the per-mutant C<prove> run hangs indefinitely (there is currently
52 no per-mutant timeout). Protect the base-case condition:
53  
54 sub factorial {
55 my ($n) = @_;
56 ## MUTANT_SKIP_BEGIN
57 return 1 if $n <= 1;
58 ## MUTANT_SKIP_END
59 return $n * factorial($n - 1);
60 }
61  
62 =item Process-management side-effects
63  
64 Forking, signalling, and C<waitpid> are inherently difficult to assert
65 against in a unit test. A mutant that corrupts a PID or signal value
66 may cause the test suite to hang or emit warnings from an C<END> block:
67  
68 sub disable_hot_reload {
69 ## MUTANT_SKIP_BEGIN
70 kill 'HUP', $pid if $pid;
71 waitpid $pid, 0;
72 ## MUTANT_SKIP_END
73 }
74  
75 =back
76  
77 To exclude a block of lines from mutation testing entirely, wrap it with
78 C<## MUTANT_SKIP_BEGIN> and C<## MUTANT_SKIP_END> annotations as shown
79 above.
80  
81 Skipped lines are silently excluded from the mutation candidate list and
82 do not appear in the killed or survived counts. The number of skipped
83 lines is shown in the per-file mutation report.
84  
85 Mismatched markers (a C<## MUTANT_SKIP_BEGIN> with no matching
86 C<## MUTANT_SKIP_END>, or vice versa) are fatal errors.
87  
88 =head2 Numeric Boundary Mutants
89  
90 Kill Numeric Boundary Mutants first,
91 these are the easiest wins.
92 For example,
93 C<NUM_BOUNDARY_1295>
94 means something like C<if ($x > 10)> became C<if ($x >= 10)> or C<if ($x > 9)>.
95 If that survived, it means, there is a missing edge value.
96 Numeric mutations are important because they reveal missing edge coverage.
97 This example means line 1295.
98  
99 So if that line contains something like this:
100  
101 if(((scalar keys %input) == 1) && exists($input{'type'}) && !ref($input{'type'})) {
102  
103 You need to add a test where
104  
105 =over 4
106  
107 =item * %input contains more than one key
108  
109 =item * One of them is type
110  
111 =item * And behavior must be different
112  
113 =back
114  
115 For example, if you have a test with
116  
117 %input = ( type => 'string' )
118  
119 add a test which sets
120  
121 %input = (
122 type => 'string',
123 something_else => 'value'
124 )
125  
126 =head2 Conditional Inversions
127  
128 Then kill Conditional Inversions, for example, C<COND_INV_1186>,
129 where C<unless (-f $file)> became C<if (-f $file)>.
130 If that survives,
131 test did not assert the negative case.
132  
133 Focus by file,
134 if one file contributes 200 survivors, that's the weakest module.
135  
136 Frequently re-run.
137 The loop should be: add 5-10 targeted tests, re-run mutation tool, watch score climb, repeat.
138  
139 =head1 DESCRIPTION
140  
141 This command-line tool performs mutation testing on a Perl codebase.
142  
143 It scans one or more C<.pm> files, generates code mutations using
144 L<App::Test::Generator::Mutator>, and runs the project's test suite
145 against each mutated version inside an isolated workspace.
146  
147 For each generated mutant:
148  
149 =over 4
150  
151 =item *
152  
153 The mutant is applied in a temporary workspace.
154  
155 =item *
156  
157 The mutated file is syntax-checked.
158  
159 =item *
160  
161 The test suite is executed using C<prove>.
162  
163 =item *
164  
165 If the tests fail, the mutant is considered I<killed>.
166  
167 =item *
168  
169 If the tests pass, the mutant is considered I<survived>.
170  
171 =back
172  
173 A mutation score is then calculated:
174  
175 (killed / total) * 100
176  
177 Mutation testing measures the effectiveness of a test suite. A higher
178 mutation score indicates that the tests are better at detecting behavioral
179 changes in the code.
180  
181 =head1 OPTIONS
182  
183 =head2 --lib <dir>
184  
185 Directory containing Perl modules to mutate.
186  
187 Defaults to C<lib>.
188  
189 =head2 --file <file>
190  
191 Mutate a single file instead of scanning the entire C<--lib> directory.
192  
193 =head2 --tests <dir>
194  
195 Directory containing test files.
196  
197 Defaults to C<t>.
198  
199 =head2 --changed_only
200  
201 Only mutate files that were changed in the most recent commit, as
202 determined by C<git diff --name-only HEAD~1 HEAD>. Files not changed
203 in the current commit retain their mutation results from the previous
204 dashboard run. This significantly reduces CI runtime while preserving
205 accuracy for the files that actually changed.
206  
207 =head2 --exclude <path>
208  
209 Exclude files matching the given path fragment from mutation testing.
210 May be specified multiple times. For example:
211  
212 --exclude lib/Devel --exclude lib/App/Test/Generator/Sample
213  
214 =head2 --base_sha <sha>
215  
216 The git commit SHA to use as the base when computing which files have
217 changed under C<--changed_only>. Defaults to C<HEAD~1>.
218  
219 Use this when automated commits (such as coverage snapshots or generated
220 test stubs) have landed on top of your last code commit, causing
221 C<HEAD~1> to point to an automated commit rather than a real change.
222 Typically set by the CI workflow using the output of C<git log> filtered
223 to exclude automated commits.
224  
225 =head2 --min-score <int>
226  
227 Minimum acceptable mutation score (percentage).
228  
229 If the final score is below this value, the program exits with a
230 non-zero status.
231  
232 =head2 --json <file>
233  
234 Write mutation results to the specified JSON file.
235  
236 The output structure:
237  
238 {
239 score => "85.32",
240 total => 120,
241 killed => 102,
242 survived => [ ... mutant IDs ... ]
243 }
244  
245 =head2 --cover_json <file>
246  
247 The location of the file generated by C<cover -report json>.
248 That file is used to generate an approximation for an LCSAJ table.
249  
250 =head2 --fail-fast
251  
252 (Reserved for future use.)
253  
254 =head2 --mutation_level <full|fast>
255  
256 Setting to C<fast> removes redundant mutations and dedups mutations before running.
257 The default is C<full>.
258  
259 =head2 --timeout <seconds>
260  
261 (Reserved for future use.)
262  
263 =head2 --verbose
264  
265 Print progress information.
266  
267 =head2 --quiet
268  
269 Suppress final summary output.
270  
271 =head1 EXIT CODES
272  
273 =over 4
274  
275 =item = 0
276  
277 Success and mutation score meets minimum threshold.
278  
279 =item = 1
280  
281 Mutation score below C<--min-score>.
282  
283 =item = 2
284  
285 Baseline test suite failed before mutation testing began.
286  
287 =item = 3
288  
289 Invalid command-line options.
290  
291 =back
292  
293 =head1 WORKFLOW
294  
295 The tool performs the following steps:
296  
297 =over 4
298  
299 =item 1.
300  
301 Collect target files (either a single file or all C<.pm> files under C<--lib>).
302  
303 =item 2.
304  
305 Run baseline tests to ensure the suite passes before mutation.
306  
307 =item 3.
308  
309 Generate mutants for each file.
310  
311 =item 4.
312  
313 Apply each mutant in isolation and re-run the test suite.
314  
315 =item 5.
316  
317 Calculate and report mutation statistics.
318  
319 =back
320  
321 =encoding utf-8
322  
323 =head1 WORKFLOW DIAGRAM
324  
325 The mutation testing process follows this execution flow:
326  
327 ┌───────────────────────────────┐
328 │ Start │
329 └───────────────┬───────────────┘
330 │
331 ▼
332 ┌───────────────────────────────┐
333 │ Collect Target Files │
334 │ --file OR scan --lib/*.pm │
335 └───────────────┬───────────────┘
336 │
337 ▼
338 ┌───────────────────────────────┐
339 │ Run Baseline Tests │
340 │ prove -l t │
341 └───────────────┬───────────────┘
342 │
343 Baseline OK? ── No ──► Exit (code 2)
344 │
345 Yes
346 │
347 ▼
348 ┌───────────────────────────────┐
349 │ For Each File │
350 └───────────────┬───────────────┘
351 │
352 ▼
353 ┌───────────────────────────────┐
354 │ Generate Mutants │
355 │ (conditional flips, etc.) │
356 └───────────────┬───────────────┘
357 │
358 ▼
359 ┌───────────────────────────────┐
360 │ For Each Mutant │
361 └───────────────┬───────────────┘
362 │
363 ▼
364 ┌───────────────────────────────┐
365 │ Prepare Workspace │
366 │ (isolated temp directory) │
367 └───────────────┬───────────────┘
368 │
369 ▼
370 ┌───────────────────────────────┐
371 │ Apply Mutant │
372 └───────────────┬───────────────┘
373 │
374 ▼
375 ┌───────────────────────────────┐
376 │ Syntax Check │
377 │ perl -c mutated_file.pm │
378 └───────────────┬───────────────┘
379 │
380 Compiles? ── No ──► Skip Mutant
381 │
382 Yes
383 │
384 ▼
385 ┌───────────────────────────────┐
386 │ Run Test Suite │
387 │ prove t │
388 └───────────────┬───────────────┘
389 │
390 Tests Fail? ── Yes ──► Killed++
391 │
392 No
393 │
394 ▼
395 Survived++
396 │
397 ▼
398 ┌───────────────────────────────┐
399 │ Repeat for Next Mutant │
400 └───────────────┬───────────────┘
401 │
402 ▼
403 ┌───────────────────────────────┐
404 │ Calculate Mutation Score │
405 │ (killed / total) * 100 │
406 └───────────────┬───────────────┘
407 │
408 ▼
409 ┌───────────────────────────────┐
410 │ Print Report / Write JSON │
411 └───────────────┬───────────────┘
412 │
413 ▼
414 Finish
415  
416 =cut
417  
418 # --------------------------------------------------
419 # Difficulty label strings and priority scores.
420 # TODO: move to App::Test::Generator::Constants once
421 # that module is created, so they can be shared with
422 # SchemaExtractor and other consumers.
423 # --------------------------------------------------
424 4 Readonly my $DIFFICULTY_HIGH_LABEL => 'HIGH';
425 4 Readonly my $DIFFICULTY_MEDIUM_LABEL => 'MEDIUM';
426 4 Readonly my $DIFFICULTY_LOW_LABEL => 'LOW';
427  
428 4 Readonly my $DIFFICULTY_HIGH_SCORE => 3;
429 4 Readonly my $DIFFICULTY_MEDIUM_SCORE => 2;
430 4 Readonly my $DIFFICULTY_LOW_SCORE => 1;
431  
432 # --------------------------------------------------
433 # Status strings written into the JSON output and
434 # used by test-generator-index to colour-code the
435 # mutation heatmap.
436 # TODO: move to App::Test::Generator::Constants.
437 # --------------------------------------------------
438 4 Readonly my $STATUS_SURVIVED => 'Survived';
439 4 Readonly my $STATUS_KILLED => 'Killed';
440  
441 # --------------------------------------------------
442 # Mutation level strings accepted by --mutation_level.
443 # 'full' runs every mutant independently.
444 # 'fast' deduplicates mutants by site before running.
445 # --------------------------------------------------
446 4 Readonly my $MUTATION_LEVEL_FULL => 'full';
447 4 Readonly my $MUTATION_LEVEL_FAST => 'fast';
448  
449 4 my @excludes;
450  
451 4 my %opt = (
452 changed_only => 0,
453 lib => 'lib',
454 tests => 't',
455 min_score => 0,
456 fail_fast => 0,
457 verbose => 0,
458 quiet => 0,
459 man => 0,
460 help => 0,
461 mutation_level => $MUTATION_LEVEL_FULL,
462 lcsaj_root => 'lcsaj',
463 lcsaj_hits => 'cover_html/lcsaj_hits.json',
464 );
465  
466 # Exit code 3 is "invalid command-line options" per the EXIT CODES POD section
467 GetOptions(
468 'base_sha=s' => \$opt{base_sha},
469 'changed_only' => \$opt{changed_only},
470 'lib=s' => \$opt{lib},
471 'file=s' => \$opt{file},
472 'tests=s' => \$opt{tests},
473 'min-score=i' => \$opt{min_score},
474 'cover_json=s' => \$opt{cover_json},
475 'exclude=s' => \@excludes,
476 'json=s' => \$opt{json},
477 'fail-fast' => \$opt{fail_fast},
478 'timeout=i' => \$opt{timeout},
479 'verbose' => \$opt{verbose},
480 'quiet' => \$opt{quiet},
481 'help|h' => \$opt{help},
482 'mutation_level=s' => \$opt{mutation_level},
483 'man|m' => \$opt{man},
484 'lcsaj_root=s' => \$opt{lcsaj_root},
485 'lcsaj_hits=s' => \$opt{lcsaj_hits},
486 4 ▶) or pod2usage(3);
Branch: 1 branch
BranchTrueFalseTotal
&GetOptions("base_sha=s", \$opt{"base_sha"}, "changed_only", \$opt{"changed_only"}, "lib=s", \$opt{"lib"}, "file=s", \$opt{"file"}, "tests=s", \$opt{"tests"}, "min-score=i", \$opt{"min_score"}, "cover_json=s", \$opt{"cover_json"}, "exclude=s", \@excludes, "json=s", \$opt{"json"}, "fail-fast", \$opt{"fail_fast"}, "timeout=i", \$opt{"timeout"}, "verbose", \$opt{"verbose"}, "quiet", \$opt{"quiet"}, "help|h", \$opt{"help"}, "mutation_level=s", \$opt{"mutation_level"}, "man|m", \$opt{"man"}, "lcsaj_root=s", \$opt{"lcsaj_root"}, "lcsaj_hits=s", \$opt{"lcsaj_hits"}) or pod2usage(3) 4 0 4
487  
488 4 ▶pod2usage(-exitval => 0, -verbose => 1) if $opt{help};
Branch: 1 branch
BranchTrueFalseTotal
if $opt{"help"} 0 4 4
489 4 ▶pod2usage(-exitval => 0, -verbose => 2) if $opt{man};
Branch: 1 branch
BranchTrueFalseTotal
if $opt{"man"} 0 4 4
490  
491 # warn "DEBUG options: changed_only=$opt{changed_only} base_sha=", $opt{base_sha} // 'undef';
492  
493 # Warn when reserved-for-future-use options are supplied so the
494 # caller knows they have no effect yet rather than silently ignoring
495 4 ▶warn "--fail-fast is reserved for future use and has no effect" if $opt{fail_fast};
Branch: 1 branch
BranchTrueFalseTotal
if $opt{"fail_fast"} 0 4 4
496 4 ▶warn "--timeout is reserved for future use and has no effect" if $opt{timeout};
Branch: 1 branch
BranchTrueFalseTotal
if $opt{"timeout"} 0 4 4
497  
498 # -------------------------
499 # Collect Files
500 # -------------------------
501 4 my @files;
502  
503 4 ▶if($opt{file}) {
Branch: 2 branches
BranchTrueFalseTotal
if ($opt{'file'}) { } 2 2 4
elsif ($opt{'changed_only'}) { } 2 0 2
504 2 push @files, $opt{file};
505 } elsif($opt{changed_only}) {
506 # Only mutate files changed since the base commit
507 croak("Invalid base_sha: $opt{base_sha}")
508 2 ▶ if defined($opt{base_sha}) && $opt{base_sha} !~ /^[0-9A-Za-z._\/-]+\z/;
Branch: 1 branch
BranchTrueFalseTotal
if defined $opt{"base_sha"} and not $opt{"base_sha"} =~ m[^[0-9A-Za-z._/-]+\z] 1 1 2
Condition66%2/3
defined $opt{"base_sha"} and not $opt{"base_sha"} =~ m[^[0-9A-Za-z._/-]+\z]
!ll&&!rl&&r
011
MC/DC50%1/2
defined $opt{"base_sha"} and not $opt{"base_sha"} =~ m[^[0-9A-Za-z._/-]+\z]
defined $opt{"base_sha"} not $opt{"base_sha"} =~ m[^[0-9A-Za-z._/-]+\z]
Truth table66%2/3
A and not(B)
A= defined $opt{"base_sha"}B= not $opt{"base_sha"} =~ m[^[0-9A-Za-z._/-]+\z]
ABresult
0X0
100
111
509 1 ▶ my $base = $opt{base_sha} ? "$opt{base_sha}^" : 'HEAD~1';
Branch: 1 branch
BranchTrueFalseTotal
$opt{'base_sha'} ? : 1 0 1
510 1 my $diff = '';
511 {
512 1 ▶ open(my $git_fh, '-|', 'git', 'diff', '--name-only', $base, 'HEAD')
Branch: 1 branch
BranchTrueFalseTotal
open my $git_fh, "-|", "git", "diff", "--name-only", $base, "HEAD" or croak("Could not run git diff: $!") 1 0 1
513 or croak("Could not run git diff: $!");
514 1 local $/;
515 1 ▶ $diff = <$git_fh> // '';
Condition50%1/2
readline $git_fh // ""
l!l
10
MC/DC0%0/1
readline $git_fh // ""
readline $git_fh
Truth table50%1/2
A
A= readline $git_fh
Aresult
11
00
516 1 close($git_fh);
517 }
518 my @changed = grep {
519 # Must be a lib .pm file, must exist, and must not match any exclusion
520 0 my $f = $_;
521 0 ▶ /^lib\/.*\.pm$/ && -f $f && !grep { $f =~ m{\Q$_\E} } @excludes
Condition0%0/6
m[^lib/.*\.pm$] && -f $f && !grep({$f =~ /\Q$_\E/;} @excludes)
!ll&&!rl&&r
000
m[^lib/.*\.pm$] and -f $f
!ll&&!rl&&r
000
MC/DC0%0/3
m[^lib/.*\.pm$] && -f $f && !grep({$f =~ /\Q$_\E/;} @excludes)
m[^lib/.*\.pm$] -f $f !grep({$f =~ /\Q$_\E/;} @excludes)
Truth table0%0/4
A and B && not(C)
A= m[^lib/.*\.pm$]B= -f $fC= !grep({$f =~ /\Q$_\E/;} @excludes)
ABCresult
0XX0
10X0
1100
1111
522 } split /\n/, $diff;
523  
524 1 ▶ if(@changed) {
Branch: 1 branch
BranchTrueFalseTotal
if (@changed) { } 0 1 1
525 0 ▶ print 'Changed-only mode: mutating ', scalar(@changed), " file(s)\n" if $opt{verbose};
Branch: 1 branch
BranchTrueFalseTotal
if $opt{"verbose"} 0 0 0
526 0 @files = @changed;
527 } else {
528 # No lib/*.pm files changed -- fall through to merge and write
529 # existing results back to JSON without running new mutations
530 1 print "No lib/*.pm files changed -- retaining previous mutation results\n";
531 }
532 } else {
533 find(
534 0 ▶ sub {
Subroutine __ANON__: not called CC=3 SCAR=24.8
535 # Skip files matching any exclusion pattern
536 0 ▶ return if grep { $File::Find::name =~ m{\Q$_\E} } @excludes;
Branch: 1 branch
BranchTrueFalseTotal
if grep {$File::Find::name =~ /\Q$_\E/;} @excludes 0 0 0
537 0 ▶ push @files, $File::Find::name if /\.pm$/;
Branch: 1 branch
BranchTrueFalseTotal
if /\.pm$/ 0 0 0
538 },
539 $opt{lib}
540 0 );
541 }
542  
543 # -------------------------
544 # Verify baseline tests
545 # -------------------------
546  
547 3 ▶print "Running baseline tests...\n" if $opt{verbose};
Branch: 1 branch
BranchTrueFalseTotal
if $opt{"verbose"} 0 3 3
548  
549 # Run with whatever LCSAJ_TARGETS (if any) is already in the parent
550 # environment, not the mutation-scoped value set below -- this baseline
551 # pass verifies the unmodified suite, and t/LCSAJ-Runtime.t exercises
552 # DB::DB() directly against its own file, which is never one of @files,
553 # so scoping LCSAJ_TARGETS to @files here makes that test spuriously fail.
554 #
555 # Use list form of system() to avoid shell interpolation of $opt{tests},
556 # which could be a security risk if the path contains special characters
557 3 ▶if(system('prove', '-Mblib', '-l', '-r', $opt{tests}) != 0) {
Branch: 1 branch
BranchTrueFalseTotal
if (system("prove", "-Mblib", "-l", "-r", $opt{"tests"}) != 0) 1 2 3
558 1 print STDERR "Baseline tests failed.\n";
559 1 exit 2;
560 }
561  
562 # Pass absolute target file paths to the LCSAJ runtime debugger
563 # via environment variable so it knows which files to instrument
564 # during the mutation runs below.
565 2 ▶if(@files) {
Branch: 1 branch
BranchTrueFalseTotal
if (@files) { } 2 0 2
566 2 $ENV{LCSAJ_TARGETS} = join($Config{path_sep}, map { abs_path($_) } @files);
567 } else {
568 # No files to mutate -- preserve existing LCSAJ hit data
569 0 delete $ENV{LCSAJ_TARGETS};
570 }
571  
572 # -------------------------
573 # Run Mutation Testing
574 # -------------------------
575  
576 2 my $total = 0;
577 2 my $killed = 0;
578 2 my @survivors;
579 my @killed_mutants;
580 2 my %skipped_by_file; # lines excluded by MUTANT_SKIP annotations, keyed by filename
581  
582 2 for my $file (@files) {
583 my $mutator = App::Test::Generator::Mutator->new(
584 file => $file,
585 lib_dir => $opt{lib},
586 mutation_level => $opt{mutation_level},
587 2 );
588  
589 2 my @mutants = $mutator->generate_mutants();
590  
591 # Collect skip count from mutator
592 2 ▶ my $skipped = scalar keys %{$mutator->{skip_lines} // {}};
Condition50%1/2
$mutator->{"skip_lines"} // {}
l!l
20
MC/DC0%0/1
$mutator->{"skip_lines"} // {}
$mutator->{"skip_lines"}
Truth table50%1/2
A
A= $mutator->{"skip_lines"}
Aresult
11
00
593  
594 # Add to per-file tracking (needs a new %skipped_by_file hash)
595 2 ▶ $skipped_by_file{$file} = $skipped if $skipped;
Branch: 1 branch
BranchTrueFalseTotal
if $skipped 0 2 2
596  
597 2 my $groups;
598  
599 2 ▶ if($opt{mutation_level} eq $MUTATION_LEVEL_FAST) {
Branch: 1 branch
BranchTrueFalseTotal
if ($opt{'mutation_level'} eq $MUTATION_LEVEL_FAST) { } 0 2 2
600 # Fast mode: deduplicate mutants by mutation site before
601 # running to avoid redundant test runs
602 0 $groups = group_mutants($file, \@mutants);
603 } else {
604 # Full mode: run every mutant independently
605 2 $groups = { map { $_->id => [$_] } @mutants };
606 }
607  
608 # warn "DEBUG file=$file groups=", scalar(keys %{$groups});
609  
610 2 for my $group (values %{$groups}) {
611 10 my $mutant = representative_mutant($group);
612  
613 print 'Testing representative mutant ',
614 $mutant->id,
615 " ($file line ",
616 $mutant->line,
617 ') representing ',
618 0 scalar(@{$group}),
619 10 ▶ " mutants\n" if $opt{verbose};
Branch: 1 branch
BranchTrueFalseTotal
if $opt{"verbose"} 0 10 10
620  
621 10 my $workspace = $mutator->prepare_workspace();
622  
623 10 $mutator->apply_mutant($mutant);
624  
625 10 my $workspace_file = File::Spec->catfile($workspace, $file);
626  
627 10 my $compile = system($^X, '-c', $workspace_file);
628 10 ▶ next if $compile != 0;
Branch: 1 branch
BranchTrueFalseTotal
if $compile != 0 0 10 10
629  
630 # Apply mutation in-place so test files that use 'use lib "$Bin/../lib"'
631 # (a common CPAN idiom) load the mutated code. Those files do
632 # 'unshift @INC' at compile time, which prepends the project's own lib/
633 # BEFORE any PERL5LIB-injected workspace path, making the workspace
634 # approach invisible to them and causing all mutants to survive.
635 # In-place mutation writes to the actual project file, so every @INC
636 # strategy (PERL5LIB, -I, 'use lib', and blib) sees the mutant.
637 10 ▶ my $orig_content = do { open(my $fh, '<', $file) or croak "Cannot read $file: $!"; local $/; <$fh> };
Branch: 1 branch
BranchTrueFalseTotal
open my $fh, "<", $file or croak("Cannot read ${file}: $!") 10 0 10
638 10 ▶ my $mut_content = do { open(my $fh, '<', $workspace_file) or croak "Cannot read $workspace_file: $!"; local $/; <$fh> };
Branch: 1 branch
BranchTrueFalseTotal
open my $fh, "<", $workspace_file or croak("Cannot read ${workspace_file}: $!") 10 0 10
639  
640 10 ▶ { open(my $fh, '>', $file) or croak "Cannot write $file: $!"; print $fh $mut_content; }
Branch: 1 branch
BranchTrueFalseTotal
open my $fh, ">", $file or croak("Cannot write ${file}: $!") 10 0 10
641  
642 # Clear PERL5OPT so no stray -Ilib or -M flags interfere with prove.
643 10 local $ENV{PERL5OPT} = '';
644  
645 # -l adds the project's lib/ to @INC for tests that don't use 'use lib'.
646 10 my $survived = (system('prove', '-l', '-r', $opt{tests}) == 0);
647  
648 # Always restore the original, even if prove crashes.
649 10 ▶ { open(my $fh, '>', $file) or croak "Cannot restore $file: $!"; print $fh $orig_content; }
Branch: 1 branch
BranchTrueFalseTotal
open my $fh, ">", $file or croak("Cannot restore ${file}: $!") 10 0 10
650  
651 10 ▶ if($survived) {
Branch: 1 branch
BranchTrueFalseTotal
if ($survived) { } 5 5 10
652 # Representative survived — mark the entire group as survived
653 5 for my $m (@{$group}) {
654 5 my $difficulty = mutation_difficulty($m);
655  
656 push @survivors, {
657 id => $m->id(),
658 line => $m->line(),
659 file => $file,
660 description => $m->description(),
661 status => $STATUS_SURVIVED,
662 difficulty => $difficulty->{label},
663 priority => $difficulty->{score},
664 hint => $difficulty->{hint},
665 5 };
666  
667 5 $total++;
668 }
669 } else {
670 # Representative killed — assume the whole group is killed
671 5 for my $m (@{$group}) {
672 5 push @killed_mutants, {
673 id => $m->id(),
674 line => $m->line(),
675 file => $file,
676 description => $m->description(),
677 status => $STATUS_KILLED,
678 };
679  
680 5 $total++;
681 5 $killed++;
682 }
683 }
684  
685 # workspace auto-destroyed when $workspace goes out of scope
686 }
687 }
688  
689 # -------------------------
690 # Report
691 # -------------------------
692  
693 2 ▶my $score = $total ? sprintf('%.2f', ($killed / $total) * 100) : 100;
Branch: 1 branch
BranchTrueFalseTotal
$total ? : 2 0 2
694  
695 2 ▶unless($opt{quiet}) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($opt{"quiet"}) 2 0 2
696 2 print "\nMutation Score: $score%\n",
697 "Total: $total\n",
698 "Killed: $killed\n",
699 'Survived: ', scalar(@survivors), "\n";
700 }
701  
702 # printf "DEBUG argv: json=%s changed_only=%s\n",
703 # $opt{json} // 'undef',
704 # $opt{changed_only} // 'undef';
705  
706 2 ▶if($opt{json}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($opt{"json"}) 0 2 2
707 0 ▶ if($opt{changed_only} && -f $opt{json}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($opt{"changed_only"} and -f $opt{"json"}) 0 0 0
Condition0%0/3
$opt{"changed_only"} and -f $opt{"json"}
!ll&&!rl&&r
000
MC/DC0%0/2
$opt{"changed_only"} and -f $opt{"json"}
$opt{"changed_only"} -f $opt{"json"}
Truth table0%0/3
A and B
A= $opt{"changed_only"}B= -f $opt{"json"}
ABresult
0X0
100
111
708 # printf "DEBUG merge: file=%s size=%d\n", $opt{json}, -s $opt{json};
709 0 my $raw = do {
710 0 local $/;
711 0 ▶ open(my $fh, '<', $opt{json}) or die "Cannot open $opt{json}: $!";
Branch: 1 branch
BranchTrueFalseTotal
open my $fh, "<", $opt{"json"} or die qq[Cannot open $opt{"json"}: $!] 0 0 0
712 <$fh>
713 0 };
714 # printf "DEBUG merge: raw_length=%d first_char=%s\n", length($raw // ''), substr($raw // '', 0, 1);
715 0 my $prev = eval { decode_json($raw) };
716 0 ▶ if($@) {
Branch: 1 branch
BranchTrueFalseTotal
if ($@) { } 0 0 0
717 # printf "DEBUG merge: PARSE ERROR: %s\n", substr($@, 0, 200);
718 0 $prev = {};
719 } else {
720 0 ▶ $prev //= {};
Condition0%0/2
$prev //= {}
l!l
00
MC/DC0%0/1
$prev //= {}
$prev
Truth table0%0/2
A
A= $prev
Aresult
11
00
721 }
722  
723 # printf "DEBUG merge: keys=%s survived=%d killed=%d\n",
724 # join(',', sort keys %{$prev // {}}),
725 # scalar(@{ $prev->{survived} || [] }),
726 # scalar(@{ $prev->{killed} || [] });
727  
728 # Keep survived/killed entries for files NOT in this run
729 0 my %mutated_files = map { $_ => 1 } @files;
730 0 ▶ my @prev_survived = grep { !$mutated_files{ $_->{file} // '' } } @{ $prev->{survived} || [] };
Condition0%0/4
$_->{"file"} // ""
l!l
00
$prev->{'survived'} || []
l!l
00
MC/DC0%0/1
$_->{"file"} // ""
$_->{"file"}
MC/DC0%0/1
$prev->{'survived'} || []
$prev->{'survived'}
Truth table0%0/2
A
A= $_->{"file"}
Aresult
11
00
Truth table0%0/2
A
A= $prev->{'survived'}
Aresult
11
00
731 0 ▶ my @prev_killed = grep { !$mutated_files{ $_->{file} // '' } } @{ $prev->{killed} || [] };
Condition0%0/4
$_->{"file"} // ""
l!l
00
$prev->{'killed'} || []
l!l
00
MC/DC0%0/1
$_->{"file"} // ""
$_->{"file"}
MC/DC0%0/1
$prev->{'killed'} || []
$prev->{'killed'}
Truth table0%0/2
A
A= $_->{"file"}
Aresult
11
00
Truth table0%0/2
A
A= $prev->{'killed'}
Aresult
11
00
732  
733 # Merge previous data with current run
734 0 unshift @survivors, @prev_survived;
735 0 unshift @killed_mutants, @prev_killed;
736  
737 # Recalculate totals to include previous data
738 0 $total += scalar(@prev_survived) + scalar(@prev_killed);
739 0 $killed += scalar(@prev_killed);
740 0 ▶ $score = $total ? sprintf('%.2f', ($killed / $total) * 100) : 100;
Branch: 1 branch
BranchTrueFalseTotal
$total ? : 0 0 0
741 }
742  
743 # Sort survivors worst-first so the most impactful gaps
744 # appear at the top of the generated test stub file
745 0 @survivors = sort { $b->{priority} <=> $a->{priority} } @survivors;
746 0 ▶ open(my $fh, '>', $opt{json}) or die "Cannot write $opt{json}: $!";
Branch: 1 branch
BranchTrueFalseTotal
open my $fh, ">", $opt{"json"} or die qq[Cannot write $opt{"json"}: $!] 0 0 0
747 0 print $fh encode_json({
748 score => $score,
749 total => $total,
750 killed_count => $killed,
751 survived_count => scalar(@survivors),
752 killed => \@killed_mutants,
753 survived => \@survivors,
754 skipped => \%skipped_by_file,
755 });
756 0 close $fh;
757 }
758  
759 2 ▶if($score < $opt{min_score}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($score < $opt{"min_score"}) 0 2 2
760 0 print STDERR "score $score is less than min_score $opt{min_score}\n";
761 0 exit 1;
762 }
763  
764 2 exit 0;
765  
766 # --------------------------------------------------
767 # group_mutants
768 #
769 # Purpose: Group mutants that affect the same
770 # mutation site (file, line, and original
771 # operator) so that only one representative
772 # mutant needs to be tested per site in
773 # fast mode.
774 #
775 # Entry: $file - path to the source file being
776 # mutated.
777 # $mutants - arrayref of mutant objects as
778 # returned by generate_mutants().
779 #
780 # Exit: Returns a hashref keyed by site key
781 # (file:line:operator), where each value
782 # is an arrayref of mutant objects at that
783 # site.
784 #
785 # Side effects: None.
786 #
787 # Notes: The site key uses the original token
788 # (operator or keyword being mutated) with
789 # whitespace stripped, so that formatting
790 # differences do not create spurious groups.
791 # Used only when --mutation_level=fast.
792 # --------------------------------------------------
793 0 ▶sub group_mutants {
Subroutine group_mutants: not called CC=3 SCAR=24.8
794 0 my ($file, $mutants) = @_;
795  
796 0 my %groups;
797  
798 0 for my $m (@{$mutants}) {
799 0 my $line = $m->line;
800  
801 # Original token being mutated (e.g. >, ==, !)
802 0 ▶ my $orig = defined($m->original) ? $m->original : '';
Branch: 1 branch
BranchTrueFalseTotal
defined $m->original ? : 0 0 0
803  
804 # Strip whitespace so formatting differences don't
805 # create spurious distinct groups for the same site
806 0 $orig =~ s/\s+//g;
807  
808 # Group key identifies a unique mutation site
809 0 my $key = join(':', $file, $line, $orig);
810  
811 0 push @{$groups{$key}}, $m;
812 }
813  
814 0 return \%groups;
815 }
816  
817 # --------------------------------------------------
818 # representative_mutant
819 #
820 # Purpose: Select a single mutant from a group to
821 # stand in for all mutants at that site
822 # when running in fast mode. If the
823 # representative is killed, the whole
824 # group is assumed killed; if it survives,
825 # the whole group is assumed to survive.
826 #
827 # Entry: $group - arrayref of mutant objects at
828 # a single mutation site.
829 #
830 # Exit: Returns a single mutant object. Never
831 # returns undef (falls back to the first
832 # element of the group).
833 #
834 # Side effects: None.
835 #
836 # Notes: Prefers boundary-type mutations when
837 # available since they probe the most
838 # meaningful behavioural edge. Falls back
839 # to the first mutant in the group if no
840 # boundary mutant is present.
841 # --------------------------------------------------
842 10 ▶sub representative_mutant {
Subroutine representative_mutant: called CC=4 SCAR=13.9
843 10 my $group = $_[0];
844  
845 10 for my $m (@{$group}) {
846 # Prefer boundary-type mutations — they are the most
847 # informative representatives since they probe numeric
848 # edge cases that other mutation types may not exercise
849 10 ▶ if($m->id() && $m->id() =~ /BOUNDARY|NUM/) {
Branch: 1 branch
BranchTrueFalseTotal
if ($m->id and $m->id =~ /BOUNDARY|NUM/) 6 4 10
Condition66%2/3
$m->id and $m->id =~ /BOUNDARY|NUM/
!ll&&!rl&&r
046
MC/DC50%1/2
$m->id and $m->id =~ /BOUNDARY|NUM/
$m->id $m->id =~ /BOUNDARY|NUM/
Truth table66%2/3
A and B
A= $m->idB= $m->id =~ /BOUNDARY|NUM/
ABresult
0X0
100
111
850 6 return $m;
851 }
852 }
853  
854 # Fallback: use the first mutant in the group
855 4 return $group->[0];
856 }
857  
858 # --------------------------------------------------
859 # mutation_difficulty
860 #
861 # Purpose: Assign a difficulty label, priority
862 # score, and hint string to a surviving
863 # mutant, reflecting how meaningful it
864 # would be to kill it and what kind of
865 # test is likely to do so.
866 #
867 # Entry: $mutant - a mutant object with an id()
868 # method returning the mutant ID string.
869 #
870 # Exit: Returns a hashref with three keys:
871 # score - numeric priority (higher is
872 # more important to kill)
873 # label - one of HIGH, MEDIUM, LOW
874 # hint - human-readable advice on what
875 # test to write
876 #
877 # Side effects: None.
878 #
879 # Notes: Classification is based on the mutant
880 # ID prefix, which encodes the mutation
881 # type. This is a heuristic — the same
882 # type of mutation may be more or less
883 # impactful depending on context.
884 #
885 # The label and score constants are
886 # candidates for extraction to
887 # App::Test::Generator::Constants once
888 # that module is created.
889 # --------------------------------------------------
890 5 ▶sub mutation_difficulty {
Subroutine mutation_difficulty: called CC=5 SCAR=16.4
891 5 my $mutant = $_[0];
892  
893 5 ▶ my $id = $mutant->id // '';
Condition50%1/2
$mutant->id // ""
l!l
50
MC/DC0%0/1
$mutant->id // ""
$mutant->id
Truth table50%1/2
A
A= $mutant->id
Aresult
11
00
894  
895 # Numeric boundary mutants expose missing edge-case tests —
896 # the highest priority since they reveal concrete value gaps
897 5 ▶ if($id =~ /BOUNDARY|NUM/) {
Branch: 1 branch
BranchTrueFalseTotal
if ($id =~ /BOUNDARY|NUM/) 3 2 5
898 return {
899 3 score => $DIFFICULTY_HIGH_SCORE,
900 label => $DIFFICULTY_HIGH_LABEL,
901 hint => 'Likely missing edge-case test (boundary value)',
902 };
903 }
904  
905 # Conditional inversions reveal missing negative assertions —
906 # medium priority since one branch was left untested
907 2 ▶ if($id =~ /COND|INV|NEG/) {
Branch: 1 branch
BranchTrueFalseTotal
if ($id =~ /COND|INV|NEG/) 1 1 2
908 return {
909 1 score => $DIFFICULTY_MEDIUM_SCORE,
910 label => $DIFFICULTY_MEDIUM_LABEL,
911 hint => 'Add tests asserting both true and false outcomes',
912 };
913 }
914  
915 # Logical operator changes reveal missing combination tests
916 1 ▶ if($id =~ /LOGIC|AND|OR/) {
Branch: 1 branch
BranchTrueFalseTotal
if ($id =~ /LOGIC|AND|OR/) 0 1 1
917 return {
918 0 score => $DIFFICULTY_MEDIUM_SCORE,
919 label => $DIFFICULTY_MEDIUM_LABEL,
920 hint => 'Test combinations where only one logical operand is true',
921 };
922 }
923  
924 # Default: minor mutation with limited behavioural impact
925 return {
926 1 score => $DIFFICULTY_LOW_SCORE,
927 label => $DIFFICULTY_LOW_LABEL,
928 hint => 'Mutation survived, but impact may be minor',
929 };
930 }
931  
932 =head1 AUTHOR
933  
934 Nigel Horne
935  
936 =cut
« lib/App/Test/Generator/Sample/Module.pm ↑ Summary lib/App/Test/Generator/Planner.pm »