lib/Log/Munger/RulesTest.pm

filter: Statementstmt 70.2% 364 / 518 Branchbran 69.4% 164 / 236 Conditioncond 59.6% 37 / 62 MC/DCmcdc 28.2% 11 / 39 Subroutinesub 100% 25 / 25 Podpod 100% 1 / 1 totaltotal 68.3% 602 / 881 CC 164 SCAR 34.4
CC
164
Cov
69%
test
65.2
_test_var_positive
51.9
_test_decompose
31.8
CRAP 31.3
« lib/Log/Munger/App/Command/list_fields.pm ↑ Summary lib/Log/Munger/LogProcessor.pm »
1 package Log::Munger::RulesTest;
2  
3 19 ▶use 5.006;
Subroutine BEGIN: called CC=1 SCAR=0.0
4 19 ▶use strict;
Subroutine BEGIN: called CC=1 SCAR=0.0
5 19 ▶use warnings;
Subroutine BEGIN: called CC=1 SCAR=0.0
6 19 ▶use Template;
Subroutine BEGIN: called CC=1 SCAR=0.0
7 19 ▶use Log::Munger::RuleFileParser;
Subroutine BEGIN: called CC=1 SCAR=0.0
8 19 ▶use Log::Munger::LogProcessor ();
Subroutine BEGIN: called CC=1 SCAR=0.0
9 19 ▶use B ();
Subroutine BEGIN: called CC=1 SCAR=0.0
10  
11 =head1 NAME
12  
13 Log::Munger::RulesTest - Runs a rule file's own tests and lints it for the usual mistakes.
14  
15 =head1 VERSION
16  
17 Version 0.0.1
18  
19 =cut
20  
21 19 our $VERSION = '0.0.1';
22  
23 =head1 SYNOPSIS
24  
25 use Log::Munger::RulesTest;
26  
27 my $results = Log::Munger::RulesTest->test( 'file' => 'postfix' );
28  
29 if ( defined( $results->{'fatal'} ) ) {
30 print 'File could not be loaded... ' . $results->{'fatal'} . "\n";
31 } else {
32 print 'Errors:' . "\n" . join( "\n", @{ $results->{'errors'} } ) . "\n\n";
33 print 'Warnings:' . "\n" . join( "\n", @{ $results->{'warnings'} } ) . "\n\n";
34 }
35  
36 A rule file carries its own tests. Each primitive under C<vars> has positive and
37 negative cases under C<vars_tests>, each rule has a C<tests> block naming strings
38 that should and should not match, and each C<decompose> entry has its own
39 C<tests>. This module is what runs all of that, along with a lint pass over every
40 resolved var.
41  
42 A rule's test case may also name the program the line arrived under, which is
43 what puts the rule's C<gate> under test alongside its patterns:
44  
45 tests:
46 positive:
47 # the gate has to let dnsmasq-dhcp through before the patterns are tried
48 - string: 'DHCPACK(eth0) 192.0.2.50 aa:bb:cc:dd:ee:ff myhost'
49 program: 'dnsmasq-dhcp'
50 result:
51 dnsmasq_dhcp_type: 'DHCPACK'
52 dnsmasq_dhcp_iface: 'eth0'
53 dnsmasq_dhcp_ip: '192.0.2.50'
54 dnsmasq_dhcp_mac: 'aa:bb:cc:dd:ee:ff'
55 dnsmasq_dhcp_hostname: 'myhost'
56 negative:
57 # a plain string is a line no pattern may match, whatever the program
58 - 'this is not a dnsmasq line at all'
59 # naming a program instead says the gate must not let this daemon in,
60 # even though the line itself is one the patterns handle
61 - string: 'DHCPACK(eth0) 192.0.2.50 aa:bb:cc:dd:ee:ff myhost'
62 program: 'dhcpd'
63  
64 Naming a program is worth the line it costs. A wrong pattern loses one message
65 type; a wrong gate loses the daemon, and every pattern test still passes while it
66 does. Gates are checked against a record holding nothing but C<PROGRAM>, which is
67 what every gate any rule file has needed keys on.
68  
69 A case may also carry a C<numeric> list, naming the fields the file's C<convert>
70 has to have turned into numbers:
71  
72 - string: 'Maximum number of concurrent DNS queries reached (max: 150)'
73 program: 'dnsmasq'
74 numeric:
75 - dnsmasq_max_queries
76 result:
77 # captures are compared as the strings they are, before any conversion
78 dnsmasq_max_queries: '150'
79  
80 C<result> and C<numeric> are asking different questions of the same case, which
81 is why both can name the same field. C<result> compares the raw capture, which is
82 the string C<'150'>. C<numeric> runs decompose and convert over a copy of those
83 captures and asks what C<'150'> became -- so a field a decompose produced can be
84 listed too, which is the case worth covering most: a kv blob split into fields,
85 one of which a convert then coerces.
86  
87 What C<numeric> checks is how perl is holding the value, not what it looks like.
88 Asking C<looks_like_number> would pass on the captured string whether the convert
89 ran or not, which is the one thing the case was written to find out.
90  
91 An C<enriched> map says what the whole field set looks like at the same point,
92 which is what puts a C<decompose> under test as the rule really uses it:
93  
94 - string: 'neti : TTY=pts/0 ; PWD=/home/neti ; USER=root ; COMMAND=/bin/ls -la'
95 program: 'sudo'
96 result:
97 # the raw captures: one blob, not yet split
98 sudo_user: 'neti'
99 sudo_kv: 'TTY=pts/0 ; PWD=/home/neti ; USER=root ; COMMAND=/bin/ls -la'
100 enriched:
101 # and what a consumer receives
102 sudo_user: 'neti'
103 sudo_TTY: 'pts/0'
104 sudo_PWD: '/home/neti'
105 sudo_USER: 'root'
106 sudo_COMMAND: '/bin/ls -la'
107  
108 A decompose entry's own C<tests> feed it a hand-written input, which says the
109 entry works but not that anything is wired to it: rename the capture it reads and
110 those tests still pass while the rule quietly stops splitting. C<enriched> is
111 compared exactly in both directions, so the missing fields are caught.
112  
113 Nothing here needs live log data or a running system, so it is cheap enough to
114 wire into CI. That is what C<log_munger test_all> does, and it exits non-zero if
115 any file reports an error. C<log_munger test_rule_file -f E<lt>fileE<gt>> does the
116 same thing for one file and dumps the whole result as YAML.
117  
118 =head1 METHODS
119  
120 =head2 test
121  
122 Runs a rule file's tests and lints it, gathering everything that went wrong rather
123 than stopping at the first problem.
124  
125 Give it either C<file> or C<hash>, not both and not neither. Beyond that, it
126 reports rather than dies: a rule file bad enough that it will not even load comes
127 back as a C<fatal>, not an exception.
128  
129 The checks are:
130  
131 =over 4
132  
133 =item * Every C<vars_tests> entry. The var is spliced into the entry's
134 C<test_template>, then each positive case has to match with C<TEST> capturing the
135 expected result, and no negative case may match with C<TEST> captured. A negative
136 that matches without capturing C<TEST> is a warning, not an error.
137  
138 =item * Every resolved var, linted for leftover grok C<%{...}>, named captures
139 Perl will not accept, embedded newlines, and anything that will not compile.
140  
141 =item * Every rule, compiled exactly as L<Log::Munger::LogProcessor> compiles it,
142 then run against its C<tests>. Positive strings have to match with the expected
143 captures and negative strings have to match nothing. A pattern that looks like a
144 var reference but names no existing var is flagged, since the engine would
145 silently treat it as an inline regexp.
146  
147 =item * Every rule's C<gate>, for the test cases that name a C<program>. A
148 positive case naming one requires the gate to accept it before the patterns are
149 tried; a negative case naming one passes if the gate refuses it or no pattern
150 matches. A gated rule no case names a program for is a warning.
151  
152 =item * Every rule's C<convert>, for the test cases carrying a C<numeric> list.
153 The case's captures are run through decompose and convert on a copy, and each
154 field named has to come out held as a number rather than as the string the
155 pattern captured. A rule that converts a field to a number and lists none is a
156 warning.
157  
158 =item * Every rule's C<decompose>, for the test cases carrying an C<enriched> map.
159 The case's captures are run through decompose and convert on a copy, and the
160 whole field set is compared exactly. A rule whose decompose fires for one of its
161 own tests and says nothing about the result is a warning.
162  
163 =item * Every C<decompose> entry, rule level and file level, applied on its own to
164 its C<tests> input and checked against the expected output.
165  
166 =item * A file level C<convert> map, for types that are not recognised.
167  
168 =back
169  
170 Anything that would produce wrong output is an error. Anything merely worth
171 knowing, such as a var or rule with no tests at all, is a warning.
172  
173 - file :: The file to load and test. Either a bare name resolved through the
174 search path, such as "postfix", or a path.
175 Default :: undef
176  
177 - hash :: An already loaded rules hash ref to test instead of a file.
178 Default :: undef
179  
180 Returns a hash ref:
181  
182 - fatal :: The reason the file could not be loaded, or undef if it loaded.
183 When this is set the other two are empty, since nothing could be tested.
184  
185 - errors :: Array ref of strings, each naming a problem that would produce
186 wrong output. Empty if there were none.
187  
188 - warnings :: Array ref of strings, each naming something worth knowing that
189 is not itself a failure. Empty if there were none.
190  
191 Every message is prefixed with where the problem is, written as a path into the
192 rule file, so C<.rules.3.tests.positive.0> is the first positive test of the
193 fourth rule.
194  
195 Dies only if neither C<file> nor C<hash> was given, if both were, or if either is
196 of the wrong type.
197  
198 my $results = Log::Munger::RulesTest->test( 'file' => 'postfix' );
199 my $results = Log::Munger::RulesTest->test( 'hash' => $rules_hash );
200  
201 =cut
202  
203 206 ▶sub test {
Subroutine test: called CC=46 SCAR=41.8
Pod documented
204 206 my ( $blank, %opts ) = @_;
205  
206 206 ▶ if ( !defined( $opts{'hash'} ) && !defined( $opts{'file'} ) ) {
Branch: 4 branches
BranchTrueFalseTotal
if (not defined $opts{'hash'} || defined $opts{'file'}) { } 0 206 206
elsif (defined $opts{'hash'} and defined $opts{'file'}) { } 0 206 206
elsif (defined $opts{'hash'} and ref $opts{'hash'} ne 'HASH') { } 0 206 206
elsif (defined $opts{'file'} and ref $opts{'file'} ne '') { } 0 229 229
Condition66%2/3
defined $opts{'hash'} || defined $opts{'file'}
l!l&&r!l&&!r
231830
MC/DC0%0/2
defined $opts{'hash'} || defined $opts{'file'}
defined $opts{'hash'} defined $opts{'file'}
Truth table66%2/3
A || B
A= defined $opts{'hash'}B= defined $opts{'file'}
ABresult
1X1
011
000
207 0 die('$opts{hash} and $opts{file} are both undef and one needs to be defined');
208 ▶ } elsif ( defined( $opts{'hash'} ) && defined( $opts{'file'} ) ) {
Condition66%2/3
defined $opts{"hash"} and defined $opts{"file"}
!ll&&!rl&&r
183230
MC/DC0%0/2
defined $opts{"hash"} and defined $opts{"file"}
defined $opts{"hash"} defined $opts{"file"}
Truth table66%2/3
A and B
A= defined $opts{"hash"}B= defined $opts{"file"}
ABresult
0X0
100
111
209 0 die('$opts{hash} and $opts{file} are both defined and only one should be');
210 ▶ } elsif ( defined( $opts{'hash'} ) && ref( $opts{'hash'} ) ne 'HASH' ) {
Condition66%2/3
defined $opts{"hash"} and ref $opts{"hash"} ne "HASH"
!ll&&!rl&&r
183230
MC/DC0%0/2
defined $opts{"hash"} and ref $opts{"hash"} ne "HASH"
defined $opts{"hash"} ref $opts{"hash"} ne "HASH"
Truth table66%2/3
A and B
A= defined $opts{"hash"}B= ref $opts{"hash"} ne "HASH"
ABresult
0X0
100
111
211 0 die( '$opts{hash} has a ref of "' . ref( $opts{'hash'} ) . '" and not "HASH"' );
212 ▶ } elsif ( defined( $opts{'file'} ) && ref( $opts{'file'} ) ne '' ) {
Condition66%2/3
defined $opts{"file"} and ref $opts{"file"} ne ""
!ll&&!rl&&r
231830
MC/DC0%0/2
defined $opts{"file"} and ref $opts{"file"} ne ""
defined $opts{"file"} ref $opts{"file"} ne ""
Truth table66%2/3
A and B
A= defined $opts{"file"}B= ref $opts{"file"} ne ""
ABresult
0X0
100
111
213 0 die( '$opts{file} has a ref of "' . ref( $opts{'file'} ) . '" and not ""' );
214 }
215  
216 206 ▶ if ( defined( $opts{'file'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $opts{"file"}) 183 23 206
217 183 eval {
218 183 my $parser = Log::Munger::RuleFileParser->new;
219 183 $opts{'hash'} = $parser->load( 'file' => $opts{'file'} );
220 };
221 183 ▶ if ($@) {
Branch: 1 branch
BranchTrueFalseTotal
if ($@) 1 182 183
222 return {
223 1 'fatal' => '"' . $opts{'file'} . '" could not be loaded... ' . $@,
224 'errors' => [],
225 'warnings' => [],
226 };
227 }
228 } ## end if ( defined( $opts{'file'} ) )
229  
230 205 my $rules = $opts{'hash'};
231  
232 205 my @errors;
233 my @warnings;
234  
235 205 my $vars_testable = 0;
236 205 ▶ if ( defined( $rules->{'vars'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $rules->{'vars'}) { } 203 2 205
237 203 ▶ if ( ref( $rules->{'vars'} ) eq 'HASH' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $rules->{'vars'} eq 'HASH') { } 203 0 203
238 203 $vars_testable = 1;
239 } else {
240 0 push( @errors, '.vars has a ref of "' . ref( $rules->{'vars'} ) . '" and not "HASH"' );
241 }
242 } else {
243 2 push( @warnings, '.vars is undef' );
244 }
245  
246 205 my $has_var_tests = 0;
247 205 ▶ if ( defined( $rules->{'vars_tests'} ) ) {
Branch: 2 branches
BranchTrueFalseTotal
if (defined $rules->{'vars_tests'}) { } 182 23 205
elsif (defined $rules->{'vars'}) { } 21 2 23
248 182 ▶ if ( ref( $rules->{'vars_tests'} ) eq 'HASH' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $rules->{'vars_tests'} eq 'HASH') { } 182 0 182
249 182 $has_var_tests = 1;
250 } else {
251 0 push( @errors, '.vars_tests has a ref of "' . ref( $rules->{'vars_tests'} ) . '" and not "HASH"' );
252 }
253 } elsif ( defined( $rules->{'vars'} ) ) {
254 21 push( @warnings, '.vars_tests is undef even though .vars exists, meaning there are no tests for it' );
255 }
256  
257 ##
258 ## process everything under .vars_tests
259 ##
260 205 ▶ if ( $has_var_tests && $vars_testable ) {
Branch: 1 branch
BranchTrueFalseTotal
if ($has_var_tests and $vars_testable) 182 23 205
Condition66%2/3
$has_var_tests and $vars_testable
!ll&&!rl&&r
230182
MC/DC50%1/2
$has_var_tests and $vars_testable
$has_var_tests $vars_testable
Truth table66%2/3
A and B
A= $has_var_testsB= $vars_testable
ABresult
0X0
100
111
261 # tracked so that, once the tests are done, any var that never showed up
262 # here can be reported as untested
263 182 my %tested_vars;
264 182 my $tt = Template->new();
265 182 foreach my $var ( keys( %{ $rules->{'vars_tests'} } ) ) {
266 14846 $tested_vars{$var} = 1;
267 14846 _test_var( $var, $rules, $tt, \@errors, \@warnings );
268 }
269  
270 #
271 # since we are done with var tests, look for any vars we've not done any tests for
272 #
273 182 foreach my $var ( keys( %{ $rules->{'vars'} } ) ) {
274 14847 ▶ if ( !$tested_vars{$var} ) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($tested_vars{$var}) 1 14846 14847
275 1 push( @warnings, '.vars.' . $var . ' lacks any tests' );
276 }
277 }
278 } ## end if ( $has_var_tests && $vars_testable )
279  
280 ##
281 ## lint every resolved var value for problems that break matching
282 ##
283 205 ▶ if ($vars_testable) {
Branch: 1 branch
BranchTrueFalseTotal
if ($vars_testable) 203 2 205
284 203 foreach my $var ( sort keys( %{ $rules->{'vars'} } ) ) {
285 14868 my $value = $rules->{'vars'}{$var};
286 14868 ▶ next if ( ref($value) ne '' ); # ref problems already reported above
Branch: 1 branch
BranchTrueFalseTotal
if ref $value ne "" 0 14868 14868
287 14868 push( @errors, _lint_regexp_string( '.vars.' . $var, $value ) );
288 }
289 }
290  
291 ##
292 ## process everything under .rules
293 ##
294 205 ▶ if ( defined( $rules->{'rules'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $rules->{"rules"}) 201 4 205
295 201 ▶ if ( ref( $rules->{'rules'} ) ne 'ARRAY' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $rules->{'rules'} ne 'ARRAY') { } 0 201 201
296 0 push( @errors, '.rules has a ref of "' . ref( $rules->{'rules'} ) . '" and not "ARRAY"' );
297 } else {
298 201 ▶ my $vars = ( ref( $rules->{'vars'} ) eq 'HASH' ) ? $rules->{'vars'} : {};
Branch: 1 branch
BranchTrueFalseTotal
ref $rules->{'vars'} eq 'HASH' ? : 201 0 201
299  
300 201 my $rule_int = 0;
301 201 foreach my $rule ( @{ $rules->{'rules'} } ) {
302 276 my $where = '.rules.' . $rule_int;
303 276 ▶ if ( ref($rule) ne 'HASH' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $rule ne "HASH") 0 276 276
304 0 push( @errors, $where . ' has a ref of "' . ref($rule) . '" and not "HASH"' );
305 0 $rule_int++;
306 0 next;
307 }
308  
309 # a pattern that references a name absent from vars is silently
310 # treated as an inline regexp by the engine; an all-caps name is
311 # almost certainly a typo'd var reference, so flag it loudly
312 276 ▶ if ( ref( $rule->{'patterns'} ) eq 'ARRAY' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $rule->{"patterns"} eq "ARRAY") 276 0 276
313 276 my $pattern_int = 0;
314 276 foreach my $pattern ( @{ $rule->{'patterns'} } ) {
315 1489 ▶ if ( ( ref($pattern) eq '' )
Branch: 1 branch
BranchTrueFalseTotal
if (ref $pattern eq "" and $pattern =~ /^[A-Z][A-Z0-9_]*\z/ and not exists $vars->{$pattern}) 1 1488 1489
Condition50%3/6
ref $pattern eq "" and $pattern =~ /^[A-Z][A-Z0-9_]*\z/ and not exists $vars->{$pattern}
!ll&&!rl&&r
014881
ref $pattern eq "" and $pattern =~ /^[A-Z][A-Z0-9_]*\z/
!ll&&!rl&&r
001489
MC/DC33%1/3
ref $pattern eq "" and $pattern =~ /^[A-Z][A-Z0-9_]*\z/ and not exists $vars->{$pattern}
ref $pattern eq "" $pattern =~ /^[A-Z][A-Z0-9_]*\z/ not exists $vars->{$pattern}
Truth table50%2/4
A and B and not(C)
A= ref $pattern eq ""B= $pattern =~ /^[A-Z][A-Z0-9_]*\z/C= not exists $vars->{$pattern}
ABCresult
0XX0
10X0
1100
1111
316 && ( $pattern =~ /^[A-Z][A-Z0-9_]*\z/ )
317 && ( !exists( $vars->{$pattern} ) ) )
318 {
319 1 push( @errors,
320 $where
321 . '.patterns.'
322 . $pattern_int
323 . ' looks like a var reference ("'
324 . $pattern
325 . '") but no such var exists' );
326 } ## end if ( ( ref($pattern) eq '' ) && ( $pattern...))
327 1489 $pattern_int++;
328 } ## end foreach my $pattern ( @{ $rule->{'patterns'} } )
329 } ## end if ( ref( $rule->{'patterns'} ) eq 'ARRAY')
330  
331 # compile the rule exactly as the engine does; a failure here is
332 # what surfaces un-degrokked grok and illegal capture names
333 276 my $compiled;
334 # compiled with the file-level defaults the engine passes, so a
335 # rule that declares no convert or decompose of its own is
336 # tested holding the ones it actually runs with
337 276 eval {
338 $compiled = Log::Munger::LogProcessor->_compile_rule(
339 'rule' => $rule,
340 'vars' => $vars,
341 'default_geoip' => $rules->{'geoip'},
342 'default_decompose' => $rules->{'decompose'},
343 276 'default_convert' => $rules->{'convert'},
344 );
345 };
346 276 ▶ if ($@) {
Branch: 1 branch
BranchTrueFalseTotal
if ($@) 0 276 276
347 0 push( @errors, $where . ' failed to compile... ' . $@ );
348 0 $rule_int++;
349 0 next;
350 }
351  
352 #
353 # run the rule's embedded tests
354 #
355 276 ▶ if ( !defined( $rule->{'tests'} ) ) {
Branch: 2 branches
BranchTrueFalseTotal
if (not defined $rule->{'tests'}) { } 0 276 276
elsif (ref $rule->{'tests'} ne 'HASH') { } 0 276 276
356 0 push( @warnings, $where . ' lacks any tests' );
357 } elsif ( ref( $rule->{'tests'} ) ne 'HASH' ) {
358 0 push( @errors, $where . '.tests has a ref of "' . ref( $rule->{'tests'} ) . '" and not "HASH"' );
359 } else {
360 276 my $gated = 0;
361 276 my $converted = 0;
362 276 my $enriched = 0;
363 276 _test_rule_positive( $where, $rule, $compiled, \@errors, \$gated, \$converted, \$enriched );
364 276 _test_rule_negative( $where, $rule, $compiled, \@errors, \$gated );
365  
366 # a gate nothing ever names is a gate nothing has checked,
367 # and a wrong one costs the whole daemon rather than one
368 # message type
369 276 ▶ if ( !$gated && @{ $compiled->{'gate'} } ) {
Branch: 1 branch
BranchTrueFalseTotal
if (not $gated and @{$compiled->{"gate"};}) 3 273 276
Condition100%3/3
not $gated and @{$compiled->{"gate"};}
!ll&&!rl&&r
239343
MC/DC100%2/2
not $gated and @{$compiled->{"gate"};}
not $gated @{$compiled->{"gate"};}
Truth table100%3/3
not(A) and B
A= not $gatedB= @{$compiled->{"gate"};}
ABresult
0X0
100
111
370 3 push( @warnings, $where . ' is gated but no test names a program' );
371 }
372  
373 # a convert nothing ever lists is a convert nothing has
374 # checked. It is the quietest thing in a rule file to get
375 # wrong: the field is captured, the value is right, and it
376 # arrives as a string where a consumer expected a number
377 276 my @unchecked = _unchecked_converts( $rule, $compiled );
378 276 ▶ if (@unchecked) {
Branch: 1 branch
BranchTrueFalseTotal
if (@unchecked) 6 270 276
379 6 push( @warnings,
380 $where
381 . ' converts to a number but no test lists as numeric: '
382 . join( ', ', @unchecked ) );
383 }
384  
385 # a decompose entry's own tests say it works, not that the rule is
386 # wired to it. Rename a capture and the entry's tests still pass
387 # while the rule quietly stops splitting anything
388 276 my @unwired = _unchecked_decomposes( $rule, $compiled );
389 276 ▶ if (@unwired) {
Branch: 1 branch
BranchTrueFalseTotal
if (@unwired) 9 267 276
390 9 push( @warnings,
391 $where
392 . ' decomposes into fields no test says it produces: '
393 . join( ', ', @unwired ) );
394 }
395 } ## end else [ if ( !defined( $rule->{'tests'} ) ) ]
396  
397 # a rule may carry its own decompose entries
398 276 ▶ if ( defined( $rule->{'decompose'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $rule->{"decompose"}) 0 276 276
399 0 _test_decompose( $rule->{'decompose'}, $vars, \@errors, \@warnings, $where . '.decompose' );
400 }
401  
402 276 $rule_int++;
403 } ## end foreach my $rule ( @{ $rules->{'rules'} } )
404 } ## end else [ if ( ref( $rules->{'rules'} ) ne 'ARRAY' )]
405 } ## end if ( defined( $rules->{'rules'} ) )
406  
407 ##
408 ## a file-level decompose is shared by every rule, so test it once here
409 ##
410 205 ▶ if ( defined( $rules->{'decompose'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $rules->{"decompose"}) 64 141 205
411 64 ▶ my $vars = ( ref( $rules->{'vars'} ) eq 'HASH' ) ? $rules->{'vars'} : {};
Branch: 1 branch
BranchTrueFalseTotal
ref $rules->{'vars'} eq 'HASH' ? : 62 2 64
412 64 _test_decompose( $rules->{'decompose'}, $vars, \@errors, \@warnings, '.decompose' );
413 }
414  
415 ##
416 ## validate a file-level convert map (bad types are a load-time error)
417 ##
418 205 ▶ if ( defined( $rules->{'convert'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $rules->{"convert"}) 156 49 205
419 156 eval { Log::Munger::LogProcessor->_compile_convert( $rules->{'convert'} ); };
420 156 ▶ if ($@) {
Branch: 1 branch
BranchTrueFalseTotal
if ($@) 1 155 156
421 1 push( @errors, '.convert is invalid... ' . $@ );
422 }
423 }
424  
425 205 my $results = {
426 'fatal' => undef,
427 'errors' => \@errors,
428 'warnings' => \@warnings,
429 };
430  
431 205 return $results;
432 } ## end sub test
433  
434 # Runs one var's vars_tests entry: validates the shapes involved, splices the
435 # var into its test_template, and hands the resulting regexp to
436 # _test_var_positive and _test_var_negative.
437 #
438 # Args:
439 #
440 # - $var :: The name of the var under test, such as "SSH_FAILED". Both the
441 # var itself and its vars_tests entry are read out of $rules.
442 #
443 # - $rules :: The loaded rules hash ref, read for .vars.$var and
444 # .vars_tests.$var.
445 #
446 # - $tt :: The Template object used to process the test_template.
447 #
448 # - $errors :: Array ref to push error strings onto. Appended to in place.
449 #
450 # - $warnings :: Array ref to push warning strings onto. Appended to in
451 # place.
452 #
453 # Returns nothing. Everything it finds goes onto $errors or $warnings.
454 #
455 # _test_var( 'SSH_FAILED', $rules, $tt, \@errors, \@warnings );
456 14846 ▶sub _test_var {
Subroutine _test_var: called CC=8 SCAR=28.4
457 14846 my ( $var, $rules, $tt, $errors, $warnings ) = @_;
458  
459 14846 my $var_tests = $rules->{'vars_tests'}{$var};
460 14846 my $value = $rules->{'vars'}{$var};
461  
462 14846 ▶ if ( !defined($value) ) {
Branch: 1 branch
BranchTrueFalseTotal
unless (defined $value) 0 14846 14846
463 0 push( @{$errors}, '.vars.' . $var . ' has tests for it but it is undefined' );
464 0 return;
465 }
466 14846 ▶ if ( ref($var_tests) ne 'HASH' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $var_tests ne "HASH") 0 14846 14846
467 0 push( @{$errors}, '.vars_tests.' . $var . ' has a ref of "' . ref($var_tests) . '" and not "HASH"' );
468 0 return;
469 }
470 14846 ▶ if ( ref($value) ne '' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $value ne "") 0 14846 14846
471 0 push( @{$errors}, '.vars.' . $var . ' has a ref of "' . ref($value) . '" and not ""' );
472 0 return;
473 }
474  
475 14846 my $test_template = $var_tests->{'test_template'};
476 14846 ▶ if ( !defined($test_template) ) {
Branch: 1 branch
BranchTrueFalseTotal
unless (defined $test_template) 0 14846 14846
477 0 push( @{$errors}, '.vars_tests.' . $var . '.test_template is undef' );
478 0 return;
479 }
480 14846 ▶ if ( ref($test_template) ne '' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $test_template ne "") 0 14846 14846
481 push(
482 0 @{$errors},
483 '.vars_tests.' . $var . '.test_template has a ref of "' . ref($test_template) . '" and not ""'
484 );
485 0 return;
486 }
487  
488 #
489 # splice the var into its test_template. everything below needs the
490 # resulting regexp, so a template that will not process ends the testing of
491 # this var here.
492 #
493 14846 my $test_regex;
494 14846 ▶ eval { $tt->process( \$test_template, { 'TEST_VAR' => $value }, \$test_regex ) || die( $tt->error() ); };
Branch: 1 branch
BranchTrueFalseTotal
$tt->process(\$test_template, {"TEST_VAR", $value}, \$test_regex) or die $tt->error 14846 0 14846
495 14846 ▶ if ($@) {
Branch: 1 branch
BranchTrueFalseTotal
if ($@) 0 14846 14846
496 0 push( @{$errors}, '.vars_tests.' . $var . '.test_template could not be templated...' . $@ );
497 0 return;
498 }
499  
500 14846 _test_var_positive( $var, $var_tests->{'positive'}, $test_regex, $errors );
501 14846 _test_var_negative( $var, $var_tests->{'negative'}, $test_regex, $errors, $warnings );
502  
503 14846 return;
504 } ## end sub _test_var
505  
506 # Runs one var's positive vars_tests cases. Each case is { string, result },
507 # and the string has to match the templated test regexp with TEST capturing the
508 # expected result.
509 #
510 # Having no positive tests is an error rather than a warning. A var with only
511 # negative tests has never been shown to match anything, which is the same as
512 # not being tested at all.
513 #
514 # Args:
515 #
516 # - $var :: The name of the var under test, used to prefix messages.
517 #
518 # - $positive :: The positive list out of the var's vars_tests entry. An
519 # array ref of { string, result } hash refs.
520 #
521 # - $test_regex :: The regexp string the test_template produced.
522 #
523 # - $errors :: Array ref to push error strings onto. Appended to in place.
524 #
525 # Returns nothing. Everything it finds goes onto $errors.
526 #
527 # _test_var_positive( $var, $var_tests->{'positive'}, $test_regex, \@errors );
528 14846 ▶sub _test_var_positive {
Subroutine _test_var_positive: called CC=14 SCAR=39.5
529 14846 my ( $var, $positive, $test_regex, $errors ) = @_;
530  
531 14846 my $pwhere = '.vars_tests.' . $var . '.positive';
532 14846 ▶ if ( !defined($positive) ) {
Branch: 3 branches
BranchTrueFalseTotal
if (not defined $positive) { } 0 14846 14846
elsif (ref $positive ne 'ARRAY') { } 0 14846 14846
elsif (not defined $positive->[0]) { } 0 14846 14846
533 0 push( @{$errors}, $pwhere . ' is undef' );
534 0 return;
535 } elsif ( ref($positive) ne 'ARRAY' ) {
536 0 push( @{$errors}, $pwhere . ' has a ref of "' . ref($positive) . '" and not "ARRAY"' );
537 0 return;
538 } elsif ( !defined( $positive->[0] ) ) {
539 0 push( @{$errors}, $pwhere . ' is empty and has no tests' );
540 0 return;
541 }
542  
543 14846 my $test_int = 0;
544 14846 foreach my $test ( @{$positive} ) {
545 # an undef entry ends the list, matching the while loop this replaced
546 80971 ▶ last if ( !defined($test) );
Branch: 1 branch
BranchTrueFalseTotal
unless defined $test 0 80971 80971
547  
548 80971 my $twhere = $pwhere . '.' . $test_int;
549 80971 ▶ if ( ref($test) ne 'HASH' ) {
Branch: 5 branches
BranchTrueFalseTotal
if (ref $test ne 'HASH') { } 0 80971 80971
elsif (not defined $test->{'string'}) { } 0 80971 80971
elsif (ref $test->{'string'} ne '') { } 0 80971 80971
elsif (not defined $test->{'result'}) { } 0 80971 80971
elsif (ref $test->{'result'} ne '') { } 0 80971 80971
550 0 push( @{$errors}, $twhere . ' has a ref of "' . ref($test) . '" and not "HASH"' );
551 } elsif ( !defined( $test->{'string'} ) ) {
552 0 push( @{$errors}, $twhere . '.string is undef' );
553 } elsif ( ref( $test->{'string'} ) ne '' ) {
554 0 push( @{$errors}, $twhere . '.string has a ref of "' . ref( $test->{'string'} ) . '" and not ""' );
555 } elsif ( !defined( $test->{'result'} ) ) {
556 0 push( @{$errors}, $twhere . '.result is undef' );
557 } elsif ( ref( $test->{'result'} ) ne '' ) {
558 0 push( @{$errors}, $twhere . '.result has a ref of "' . ref( $test->{'result'} ) . '" and not ""' );
559 } else {
560 my $detail
561 = '... test_regex="'
562 . $test_regex
563 . '" string="'
564 . $test->{'string'}
565 . '" expected result="'
566 80971 . $test->{'result'} . '"';
567 80971 ▶ if ( $test->{'string'} =~ /$test_regex/ ) {
Branch: 1 branch
BranchTrueFalseTotal
if ($test->{'string'} =~ /$test_regex/) { } 80971 0 80971
568 80971 my %found_items = %+;
569 80971 ▶ if ( !defined( $found_items{'TEST'} ) ) {
Branch: 2 branches
BranchTrueFalseTotal
if (not defined $found_items{'TEST'}) { } 0 80971 80971
elsif ($found_items{'TEST'} ne $test->{'result'}) { } 0 80971 80971
570 0 push( @{$errors}, $twhere . ' matched but "TEST" was not captured' . $detail );
571 } elsif ( $found_items{'TEST'} ne $test->{'result'} ) {
572 push(
573 0 @{$errors},
574 $twhere
575 . ' matched but "TEST" captured the wrong result, "'
576 0 . $found_items{'TEST'} . '"'
577 . $detail
578 );
579 }
580 } else {
581 0 push( @{$errors}, $twhere . ' did not match but was expected to' . $detail );
582 }
583 } ## end else [ if ( ref($test) ne 'HASH' ) ]
584  
585 80971 $test_int++;
586 } ## end foreach my $test ( @{$positive} )
587  
588 14846 return;
589 } ## end sub _test_var_positive
590  
591 # Runs one var's negative vars_tests cases. Each case is a plain string that
592 # has to match nothing, or at least not capture TEST: a negative that matches
593 # without capturing TEST is a warning rather than an error.
594 #
595 # Args:
596 #
597 # - $var :: The name of the var under test, used to prefix messages.
598 #
599 # - $negative :: The negative list out of the var's vars_tests entry. An
600 # array ref of plain strings.
601 #
602 # - $test_regex :: The regexp string the test_template produced.
603 #
604 # - $errors :: Array ref to push error strings onto. Appended to in place.
605 #
606 # - $warnings :: Array ref to push warning strings onto. Appended to in
607 # place.
608 #
609 # Returns nothing. Everything it finds goes onto $errors or $warnings.
610 #
611 # _test_var_negative( $var, $var_tests->{'negative'}, $test_regex, \@errors, \@warnings );
612 14846 ▶sub _test_var_negative {
Subroutine _test_var_negative: called CC=8 SCAR=30.2
613 14846 my ( $var, $negative, $test_regex, $errors, $warnings ) = @_;
614  
615 14846 my $nwhere = '.vars_tests.' . $var . '.negative';
616 14846 ▶ if ( !defined($negative) ) {
Branch: 2 branches
BranchTrueFalseTotal
if (not defined $negative) { } 0 14846 14846
elsif (ref $negative ne 'ARRAY') { } 0 14846 14846
617 0 push( @{$errors}, $nwhere . ' is undef' );
618 0 return;
619 } elsif ( ref($negative) ne 'ARRAY' ) {
620 0 push( @{$errors}, $nwhere . ' has a ref of "' . ref($negative) . '" and not "ARRAY"' );
621 0 return;
622 }
623  
624 14846 my $test_int = 0;
625 14846 foreach my $test ( @{$negative} ) {
626 # an undef entry ends the list, matching the while loop this replaced
627 39812 ▶ last if ( !defined($test) );
Branch: 1 branch
BranchTrueFalseTotal
unless defined $test 0 39812 39812
628  
629 39812 my $twhere = $nwhere . '.' . $test_int;
630 39812 ▶ if ( ref($test) ne '' ) {
Branch: 2 branches
BranchTrueFalseTotal
if (ref $test ne '') { } 0 39812 39812
elsif ($test =~ /$test_regex/) { } 0 39812 39812
631 0 push( @{$errors}, $twhere . ' has a ref of "' . ref($test) . '" and not ""' );
632 } elsif ( $test =~ /$test_regex/ ) {
633 0 my %found_items = %+;
634 0 my $detail = '... test_regex="' . $test_regex . '" string="' . $test . '"';
635 0 ▶ if ( !defined( $found_items{'TEST'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (not defined $found_items{'TEST'}) { } 0 0 0
636 0 push( @{$warnings}, $twhere . ' matched but TEST was not found... possible error' . $detail );
637 } else {
638 push(
639 0 @{$errors},
640 0 $twhere . ' matched with TEST having a value of "' . $found_items{'TEST'} . '"' . $detail
641 );
642 }
643 } ## end elsif ( $test =~ /$test_regex/ )
644  
645 39812 $test_int++;
646 } ## end foreach my $test ( @{$negative} )
647  
648 14846 return;
649 } ## end sub _test_var_negative
650  
651 # Lints one fully resolved regexp string for the four things that go wrong often
652 # enough to be worth checking for by name.
653 #
654 # Leftover grok "%{...}" means a pattern was copied in from logstash and never run
655 # through Log::Munger::Degrok. An illegal named capture means a grok capture name
656 # came across as-is when Perl will not take it; grok is happy with "src-ip" and
657 # Perl only accepts [A-Za-z_]\w*. An embedded newline usually means a YAML "|"
658 # block scalar leaked its terminator into the middle of a pattern, which leaves it
659 # unable to match a single line log while looking perfectly fine in the file. And
660 # a string that will not compile at all is caught here rather than at match time.
661 #
662 # The capture name scan skips (?<= and (?<! since those are lookbehind assertions
663 # rather than named captures.
664 #
665 # Args:
666 #
667 # - $where :: Where this string lives in the rule file, written as a path,
668 # such as ".vars.SSH_FAILED". Prefixed onto every message so the caller
669 # knows what to go look at.
670 #
671 # - $value :: The resolved regexp string to lint. This wants to be the value
672 # after templating, not the vars_templated source, since the whole point
673 # is to check what the engine will actually compile.
674 #
675 # Returns a list of error strings, each already prefixed with $where. An empty
676 # list means the string is clean. Nothing is ever pushed to warnings from here;
677 # everything it looks for would produce wrong output.
678 #
679 # my @errors = _lint_regexp_string( '.vars.SSH_FAILED', $rules->{'vars'}{'SSH_FAILED'} );
680 # # ( '.vars.SSH_FAILED has an illegal named-capture "src-ip" (must match [A-Za-z_]\w*)' )
681 14868 ▶sub _lint_regexp_string {
Subroutine _lint_regexp_string: called CC=6 SCAR=17.9
682 14868 my ( $where, $value ) = @_;
683  
684 14868 my @errors;
685  
686 14868 ▶ if ( $value =~ /%\{[^}]*\}/ ) {
Branch: 1 branch
BranchTrueFalseTotal
if ($value =~ /%\{[^}]*\}/) 1 14867 14868
687 1 push( @errors, $where . ' contains un-degrokked grok "%{...}"' );
688 }
689  
690 # validate every (?<name> ... capture name; Perl allows [A-Za-z_]\w* only.
691 # the (?![=!]) guard skips lookbehind assertions (?<= and (?<!
692 14868 ▶ while ( $value =~ /\(\?<(?![=!])([^>]*)>/g ) {
Branch: 1 branch
BranchTrueFalseTotal
if ($value =~ /\(\?<(?![=!])([^>]*)>/g) 7918 14868 22786
693 7918 my $name = $1;
694 7918 ▶ if ( $name !~ /^[A-Za-z_]\w*\z/ ) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($name =~ /^[A-Za-z_]\w*\z/) 1 7917 7918
695 1 push( @errors, $where . ' has an illegal named-capture "' . $name . '" (must match [A-Za-z_]\\w*)' );
696 }
697 }
698  
699 14868 ▶ if ( $value =~ /\n/ ) {
Branch: 1 branch
BranchTrueFalseTotal
if ($value =~ /\n/) 0 14868 14868
700 0 push( @errors, $where . ' contains an embedded newline and cannot match a single-line log' );
701 }
702  
703 14868 my $compiled;
704 14868 eval {
705 # this is a deliberate trial compile of possibly-bad input; keep any
706 # regexp warnings out of the caller's stderr
707 14868 ▶ local $SIG{'__WARN__'} = sub { };
Subroutine __ANON__: called CC=1 SCAR=0.0
708 14868 $compiled = qr/$value/;
709 };
710 14868 ▶ if ($@) {
Branch: 1 branch
BranchTrueFalseTotal
if ($@) 1 14867 14868
711 1 push( @errors, $where . ' does not compile as a regexp... ' . $@ );
712 }
713  
714 14868 return @errors;
715 } ## end sub _lint_regexp_string
716  
717 # Matches a string against a compiled rule's patterns, the way the engine does:
718 # tried in order, first match wins.
719 #
720 # Args:
721 #
722 # - $compiled :: The rule after Log::Munger::LogProcessor->_compile_rule,
723 # read for its qr// patterns list.
724 #
725 # - $string :: The string to match, such as a test case's string.
726 #
727 # Returns a hash ref of the winning pattern's named captures, which may be an
728 # empty hash ref for a pattern with no named captures, or undef when no pattern
729 # matched.
730 #
731 # my $captures = _first_match( $compiled, $test->{'string'} );
732 4304 ▶sub _first_match {
Subroutine _first_match: called CC=3 SCAR=11.0
733 4304 my ( $compiled, $string ) = @_;
734  
735 4304 foreach my $pattern ( @{ $compiled->{'patterns'} } ) {
736 29658 ▶ if ( $string =~ $pattern ) {
Branch: 1 branch
BranchTrueFalseTotal
if ($string =~ /$pattern/) 4303 25355 29658
737 4303 my %captures = %+;
738 4303 return \%captures;
739 }
740 }
741  
742 1 return undef;
743 } ## end sub _first_match
744  
745 # Runs a rule's positive tests. Each case names a string that has to match one of
746 # the rule's patterns, along with the captures that match is expected to produce.
747 #
748 # The patterns are tried in order and the first to match wins, exactly as the
749 # engine does it, so a test also pins down which pattern is supposed to be
750 # handling a given line.
751 #
752 # A case may also name a program, and then the rule's gate has to accept it
753 # before the patterns are tried. This is what covers the gate itself. A gate is
754 # not a detail of the pattern below it -- it is the reason the rule is consulted
755 # at all, and getting it wrong means a daemon's whole log goes unmatched while
756 # every pattern test still passes. Naming a program on at least one case is what
757 # turns "these patterns work" into "this rule fires for this daemon".
758 #
759 # What is compared against a case's result is the raw named captures and nothing
760 # else, which is why a positive test's expected result lists a port as the string
761 # '54321' even though a convert: turns it into a number at runtime. A case that
762 # also carries numeric or enriched gets a second look via _test_numeric /
763 # _test_enriched, which run decompose and convert over a copy of the captures.
764 # Geoip never runs here; it is a runtime concern.
765 #
766 # Args:
767 #
768 # - $where :: Where this rule lives, written as a path, such as ".rules.3".
769 # The per-test messages extend it, giving ".rules.3.tests.positive.0".
770 #
771 # - $rule :: The raw rule hash ref, straight out of the rule file. Only its
772 # tests.positive is read here. Each entry is
773 # { string, result => {}, program => optional }.
774 #
775 # - $compiled :: The same rule after Log::Munger::LogProcessor->_compile_rule,
776 # which is where the qr// patterns and the compiled gates come from.
777 #
778 # - $errors :: Array ref to push error strings onto. Appended to in place.
779 #
780 # - $gated :: Scalar ref, set to 1 if any case named a program. The caller
781 # uses it to warn about a gated rule whose tests never exercise the gate.
782 #
783 # - $converted :: Scalar ref, set to 1 if any case carried a numeric list.
784 # The caller uses it the same way for a rule whose converts go untested.
785 #
786 # - $enriched :: Scalar ref, set to 1 if any case carried an enriched map.
787 # The caller uses it the same way for a rule whose decompose goes
788 # untested.
789 #
790 # Returns nothing. Everything it finds goes onto $errors, which is left untouched
791 # if all the positive tests pass.
792 #
793 # _test_rule_positive( '.rules.0', $rule, $compiled, \@errors, \$gated, \$converted, \$enriched );
794 276 ▶sub _test_rule_positive {
Subroutine _test_rule_positive: called CC=16 SCAR=33.9
795 276 my ( $where, $rule, $compiled, $errors, $gated, $converted, $enriched ) = @_;
796  
797 276 my $positive = $rule->{'tests'}{'positive'};
798 276 ▶ if ( !defined($positive) ) {
Branch: 2 branches
BranchTrueFalseTotal
if (not defined $positive) { } 0 276 276
elsif (ref $positive ne 'ARRAY') { } 0 276 276
799 0 push( @{$errors}, $where . '.tests.positive is undef' );
800 0 return;
801 } elsif ( ref($positive) ne 'ARRAY' ) {
802 0 push( @{$errors}, $where . '.tests.positive has a ref of "' . ref($positive) . '" and not "ARRAY"' );
803 0 return;
804 }
805  
806 276 my $test_int = 0;
807 276 foreach my $test ( @{$positive} ) {
808 2003 my $twhere = $where . '.tests.positive.' . $test_int;
809 2003 ▶ if ( ref($test) ne 'HASH' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $test ne "HASH") 0 2003 2003
810 0 push( @{$errors}, $twhere . ' has a ref of "' . ref($test) . '" and not "HASH"' );
811 0 $test_int++;
812 0 next;
813 }
814 2003 ▶ if ( !defined( $test->{'string'} ) || ref( $test->{'string'} ) ne '' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (not defined $test->{"string"} or ref $test->{"string"} ne "") 0 2003 2003
Condition33%1/3
not defined $test->{"string"} or ref $test->{"string"} ne ""
l!l&&r!l&&!r
002003
MC/DC0%0/2
not defined $test->{"string"} or ref $test->{"string"} ne ""
not defined $test->{"string"} ref $test->{"string"} ne ""
Truth table33%1/3
not(A) or B
A= not defined $test->{"string"}B= ref $test->{"string"} ne ""
ABresult
1X1
011
000
815 0 push( @{$errors}, $twhere . '.string is undef or not a string' );
816 0 $test_int++;
817 0 next;
818 }
819 2003 ▶ my $expected = defined( $test->{'result'} ) ? $test->{'result'} : {};
Branch: 1 branch
BranchTrueFalseTotal
defined $test->{'result'} ? : 2003 0 2003
820 2003 ▶ if ( ref($expected) ne 'HASH' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $expected ne "HASH") 0 2003 2003
821 0 push( @{$errors}, $twhere . '.result has a ref of "' . ref($expected) . '" and not "HASH"' );
822 0 $test_int++;
823 0 next;
824 }
825  
826 # a named program has to get past the rule's gate before the patterns
827 # are worth trying
828 2003 ▶ if ( defined( $test->{'program'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $test->{"program"}) 462 1541 2003
829 462 ▶ if ( ref( $test->{'program'} ) ne '' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $test->{"program"} ne "") 0 462 462
830 0 push( @{$errors}, $twhere . '.program has a ref of "' . ref( $test->{'program'} ) . '" and not ""' );
831 0 $test_int++;
832 0 next;
833 }
834 462 ${$gated} = 1;
835 462 my $refusal = _gate_refusal( $compiled, $test->{'program'} );
836 462 ▶ if ( defined($refusal) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $refusal) 1 461 462
837 1 push( @{$errors}, $twhere . ' ' . $refusal );
838 1 $test_int++;
839 1 next;
840 }
841 } ## end if ( defined( $test->{'program'} ) )
842  
843 2002 my $got = _first_match( $compiled, $test->{'string'} );
844  
845 2002 ▶ if ( !defined($got) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (not defined $got) { } 1 2001 2002
846 1 push( @{$errors}, $twhere . ' did not match any pattern... string="' . $test->{'string'} . '"' );
847 } else {
848 2001 my $diff = _capture_diff( $expected, $got );
849 2001 ▶ if ( defined($diff) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $diff) 0 2001 2001
850 push(
851 0 @{$errors},
852 0 $twhere . ' captures differ from expected: ' . $diff . ' string="' . $test->{'string'} . '"'
853 );
854 }
855 2001 ▶ if ( defined( $test->{'numeric'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $test->{"numeric"}) 338 1663 2001
856 338 ${$converted} = 1;
857 338 _test_numeric( $twhere, $test->{'numeric'}, $compiled, $got, $errors );
858 }
859 2001 ▶ if ( defined( $test->{'enriched'} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $test->{"enriched"}) 160 1841 2001
860 160 ${$enriched} = 1;
861 160 _test_enriched( $twhere, $test->{'enriched'}, $compiled, $got, $errors );
862 }
863 } ## end else [ if ( !defined($got) ) ]
864  
865 2002 $test_int++;
866 } ## end foreach my $test ( @{$positive} )
867  
868 276 return;
869 } ## end sub _test_rule_positive
870  
871 # Runs a rule's negative tests. Each case is a string that has to match none of
872 # the rule's patterns.
873 #
874 # This is what keeps a loose pattern honest. A pattern written as "a word, a
875 # space, then anything" will happily match most of a log file, and its positive
876 # tests will not notice. A good negative case is one that is genuinely outside
877 # what the pattern was written for, not merely unrelated in subject matter.
878 #
879 # A case may instead be written as { string, program }, and then it passes if
880 # either the gate refuses the program or no pattern matches. That is the shape
881 # of a gate written too wide: the line is one this rule handles perfectly well,
882 # and the whole point is that it arrived under a program this rule has no
883 # business claiming.
884 #
885 # Args:
886 #
887 # - $where :: Where this rule lives, written as a path, such as ".rules.3".
888 # The per-test messages extend it, giving ".rules.3.tests.negative.0".
889 #
890 # - $rule :: The raw rule hash ref, straight out of the rule file. Only its
891 # tests.negative is read here. Each entry is either a plain string or a
892 # { string, program } hash ref.
893 #
894 # - $compiled :: The same rule after Log::Munger::LogProcessor->_compile_rule,
895 # which is where the qr// patterns and the compiled gates come from.
896 #
897 # - $errors :: Array ref to push error strings onto. Appended to in place.
898 #
899 # - $gated :: Scalar ref, set to 1 if any case named a program. Shared with
900 # _test_rule_positive, since either kind of case exercises the gate.
901 #
902 # Returns nothing. Everything it finds goes onto $errors, which is left untouched
903 # if all the negative tests pass.
904 #
905 # _test_rule_negative( '.rules.0', $rule, $compiled, \@errors, \$gated );
906 276 ▶sub _test_rule_negative {
Subroutine _test_rule_negative: called CC=15 SCAR=33.4
907 276 my ( $where, $rule, $compiled, $errors, $gated ) = @_;
908  
909 276 my $negative = $rule->{'tests'}{'negative'};
910 276 ▶ if ( !defined($negative) ) {
Branch: 2 branches
BranchTrueFalseTotal
if (not defined $negative) { } 0 276 276
elsif (ref $negative ne 'ARRAY') { } 0 276 276
911 0 push( @{$errors}, $where . '.tests.negative is undef' );
912 0 return;
913 } elsif ( ref($negative) ne 'ARRAY' ) {
914 0 push( @{$errors}, $where . '.tests.negative has a ref of "' . ref($negative) . '" and not "ARRAY"' );
915 0 return;
916 }
917  
918 276 my $test_int = 0;
919 276 foreach my $test ( @{$negative} ) {
920 618 my $twhere = $where . '.tests.negative.' . $test_int;
921  
922 618 my $string = $test;
923 618 my $program = undef;
924 618 ▶ if ( ref($test) eq 'HASH' ) {
Branch: 2 branches
BranchTrueFalseTotal
if (ref $test eq 'HASH') { } 247 371 618
elsif (ref $test ne '') { } 0 371 371
925 247 $string = $test->{'string'};
926 247 $program = $test->{'program'};
927 247 ▶ if ( !defined($string) || ref($string) ne '' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (not defined $string or ref $string ne "") 0 247 247
Condition33%1/3
not defined $string or ref $string ne ""
l!l&&r!l&&!r
00247
MC/DC0%0/2
not defined $string or ref $string ne ""
not defined $string ref $string ne ""
Truth table33%1/3
not(A) or B
A= not defined $stringB= ref $string ne ""
ABresult
1X1
011
000
928 0 push( @{$errors}, $twhere . '.string is undef or not a string' );
929 0 $test_int++;
930 0 next;
931 }
932 247 ▶ if ( !defined($program) || ref($program) ne '' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (not defined $program or ref $program ne "") 0 247 247
Condition33%1/3
not defined $program or ref $program ne ""
l!l&&r!l&&!r
00247
MC/DC0%0/2
not defined $program or ref $program ne ""
not defined $program ref $program ne ""
Truth table33%1/3
not(A) or B
A= not defined $programB= ref $program ne ""
ABresult
1X1
011
000
933 0 push( @{$errors}, $twhere . '.program is undef or not a string' );
934 0 $test_int++;
935 0 next;
936 }
937 247 ${$gated} = 1;
938 } elsif ( ref($test) ne '' ) {
939 0 push( @{$errors}, $twhere . ' has a ref of "' . ref($test) . '" and not "" or "HASH"' );
940 0 $test_int++;
941 0 next;
942 }
943  
944 # a refused program is the whole answer; the patterns are never reached
945 618 ▶ if ( defined($program) && defined( _gate_refusal( $compiled, $program ) ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $program and defined _gate_refusal($compiled, $program)) 246 372 618
Condition100%3/3
defined $program and defined _gate_refusal($compiled, $program)
!ll&&!rl&&r
3711246
MC/DC100%2/2
defined $program and defined _gate_refusal($compiled, $program)
defined $program defined _gate_refusal($compiled, $program)
Truth table100%3/3
A and B
A= defined $programB= defined _gate_refusal($compiled, $program)
ABresult
0X0
100
111
946 246 $test_int++;
947 246 next;
948 }
949  
950 372 foreach my $pattern ( @{ $compiled->{'patterns'} } ) {
951 2537 ▶ if ( $string =~ $pattern ) {
Branch: 1 branch
BranchTrueFalseTotal
if ($string =~ /$pattern/) 1 2536 2537
952 push(
953 1 ▶ @{$errors},
Branch: 1 branch
BranchTrueFalseTotal
defined $program ? : 1 0 1
954 $twhere
955 . ' matched a pattern but should not have'
956 . ( defined($program) ? ' (gate accepted "' . $program . '")' : '' )
957 . '... string="'
958 . $string . '"'
959 );
960 1 last;
961 } ## end if ( $string =~ $pattern )
962 } ## end foreach my $pattern ( @{ $compiled->{'patterns'...}})
963 372 $test_int++;
964 } ## end foreach my $test ( @{$negative} )
965  
966 276 return;
967 } ## end sub _test_rule_negative
968  
969 # Reports whether perl is holding a scalar as a number rather than as the string
970 # a capture produced.
971 #
972 # This is what a "numeric" test case is asking, and asking it any other way does
973 # not work. Scalar::Util::looks_like_number answers "could this be read as a
974 # number", which is true of the captured string before any conversion happens, so
975 # a check built on it passes whether the convert ran or not.
976 #
977 # A capture is POK and nothing else. Once _convert has put an int() or a numeric
978 # addition through it the scalar carries IOK or NOK as well, and that is the only
979 # difference between "44444" fresh out of a pattern and 44444 after a convert:
980 # the two compare equal both as strings and as numbers.
981 #
982 # The value is read through a reference and never used in numeric context here,
983 # since doing so would set the very flag being looked for.
984 #
985 # Args:
986 #
987 # - $value :: The scalar to inspect, normally a field out of a captures hash
988 # that decompose and convert have been run over. undef is allowed and
989 # reported as not converted.
990 #
991 # Returns 1 when the scalar carries a numeric slot, otherwise 0.
992 #
993 # _holds_a_number( 44444 ) # 1
994 # _holds_a_number( '44444' ) # 0
995 585 ▶sub _holds_a_number {
Subroutine _holds_a_number: called CC=3 SCAR=11.3
996 585 my ($value) = @_;
997  
998 585 ▶ if ( !defined($value) ) {
Branch: 1 branch
BranchTrueFalseTotal
unless (defined $value) 0 585 585
999 0 return 0;
1000 }
1001  
1002 585 my $sv = B::svref_2object( \$value );
1003  
1004 585 ▶ return ( $sv->FLAGS & ( B::SVf_IOK() | B::SVf_NOK() ) ) ? 1 : 0;
Branch: 1 branch
BranchTrueFalseTotal
$sv->FLAGS & 768 ? : 584 1 585
1005 } ## end sub _holds_a_number
1006  
1007 # Runs a copy of a case's captures through decompose and then convert, the way
1008 # the engine does, and hands back what a consumer would receive.
1009 #
1010 # Copied rather than modified, because .result compares the raw capture set and
1011 # that does not change. Everything a case says about the stage after -- what a
1012 # blob was split into, what a string was coerced to -- is asked of this copy.
1013 #
1014 # geoip is not run. It needs a database, and it adds a key rather than changing
1015 # one, so it is a runtime concern rather than a property of the rule file.
1016 #
1017 # Args:
1018 #
1019 # - $compiled :: The rule after Log::Munger::LogProcessor->_compile_rule,
1020 # read for its decompose list and convert map.
1021 #
1022 # - $captures :: Hash ref of the raw named captures the winning pattern
1023 # produced. Left untouched.
1024 #
1025 # Returns a hash ref of the fields as they would reach a consumer.
1026 #
1027 # my $fields = _enrich( $compiled, \%got );
1028 498 ▶sub _enrich {
Subroutine _enrich: called CC=3 SCAR=11.0
1029 498 my ( $compiled, $captures ) = @_;
1030  
1031 498 my %fields = %{$captures};
1032 498 ▶ if ( @{ $compiled->{'decompose'} } ) {
Branch: 1 branch
BranchTrueFalseTotal
if (@{$compiled->{"decompose"};}) 298 200 498
1033 298 Log::Munger::LogProcessor->_decompose( $compiled, \%fields );
1034 }
1035 498 ▶ if ( keys( %{ $compiled->{'convert'} } ) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (keys %{$$compiled{"convert"};}) 431 67 498
1036 431 Log::Munger::LogProcessor->_convert( $compiled, \%fields );
1037 }
1038  
1039 498 return \%fields;
1040 } ## end sub _enrich
1041  
1042 # Runs a positive case's enriched expectation: the field set a consumer receives
1043 # once decompose and convert have run.
1044 #
1045 # This is what puts a decompose under test as the rule really uses it. An entry's
1046 # own tests feed it a hand-written input in isolation, which says the entry works
1047 # but not that it is wired to anything -- nothing there notices when a pattern's
1048 # capture is renamed out from under it, or when the field one entry produces
1049 # stops being the field the next one consumes.
1050 #
1051 # Compared exactly in both directions, as .result is, so a field that quietly
1052 # stops being produced is caught and so is one that turns up unasked for.
1053 #
1054 # Args:
1055 #
1056 # - $twhere :: Where this case lives, written as a path, such as
1057 # ".rules.3.tests.positive.0". Used to prefix any error.
1058 #
1059 # - $expected :: The case's enriched map, as written in the rule file.
1060 #
1061 # - $compiled :: The rule after Log::Munger::LogProcessor->_compile_rule.
1062 #
1063 # - $captures :: Hash ref of the raw named captures. Copied, not modified.
1064 #
1065 # - $errors :: Array ref to push error strings onto. Appended to in place.
1066 #
1067 # Returns nothing. Everything it finds goes onto $errors.
1068 #
1069 # _test_enriched( '.rules.0.tests.positive.1', $expected, $compiled, \%got, \@errors );
1070 160 ▶sub _test_enriched {
Subroutine _test_enriched: called CC=3 SCAR=11.7
1071 160 my ( $twhere, $expected, $compiled, $captures, $errors ) = @_;
1072  
1073 160 ▶ if ( ref($expected) ne 'HASH' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $expected ne "HASH") 0 160 160
1074 0 push( @{$errors}, $twhere . '.enriched has a ref of "' . ref($expected) . '" and not "HASH"' );
1075 0 return;
1076 }
1077  
1078 160 my $diff = _capture_diff( $expected, _enrich( $compiled, $captures ) );
1079 160 ▶ if ( defined($diff) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $diff) 3 157 160
1080 3 push( @{$errors}, $twhere . '.enriched differs from what decompose and convert produced: ' . $diff );
1081 }
1082  
1083 160 return;
1084 } ## end sub _test_enriched
1085  
1086 # Runs a positive test case's captures through decompose and convert, the way the
1087 # engine does (via _enrich), and checks the fields the case listed as numeric.
1088 #
1089 # The captures are copied first. What .result compares is the raw capture set and
1090 # that does not change -- which is why a positive test's expected result still
1091 # lists a port as the string '54321'. This runs on a copy of the same captures so
1092 # a case can say what that string is supposed to become.
1093 #
1094 # Decompose runs first, as it does at runtime, so a field a decompose produced can
1095 # be listed too. That covers the case worth covering most: a kv blob split into
1096 # fields, one of which a convert then coerces.
1097 #
1098 # Args:
1099 #
1100 # - $twhere :: Where this case lives, written as a path, such as
1101 # ".rules.3.tests.positive.0". Used to prefix any error.
1102 #
1103 # - $numeric :: The case's numeric list, as written in the rule file. An array
1104 # ref of capture names, such as [ 'dnsmasq_max_queries' ].
1105 #
1106 # - $compiled :: The rule after Log::Munger::LogProcessor->_compile_rule,
1107 # read for its decompose list and convert map.
1108 #
1109 # - $captures :: Hash ref of the raw named captures the winning pattern
1110 # produced. Copied rather than modified.
1111 #
1112 # - $errors :: Array ref to push error strings onto. Appended to in place.
1113 #
1114 # Returns nothing. Everything it finds goes onto $errors.
1115 #
1116 # _test_numeric( '.rules.0.tests.positive.1', ['ssh_src_port'], $compiled, \%got, \@errors );
1117 338 ▶sub _test_numeric {
Subroutine _test_numeric: called CC=7 SCAR=21.0
1118 338 my ( $twhere, $numeric, $compiled, $captures, $errors ) = @_;
1119  
1120 338 ▶ if ( ref($numeric) ne 'ARRAY' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $numeric ne "ARRAY") 0 338 338
1121 0 push( @{$errors}, $twhere . '.numeric has a ref of "' . ref($numeric) . '" and not "ARRAY"' );
1122 0 return;
1123 }
1124  
1125 338 my %converted = %{ _enrich( $compiled, $captures ) };
1126  
1127 338 foreach my $field ( @{$numeric} ) {
1128 586 ▶ if ( ref($field) ne '' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $field ne "") 0 586 586
1129 0 push( @{$errors}, $twhere . '.numeric entry has a ref of "' . ref($field) . '" and not ""' );
1130 0 next;
1131 }
1132 586 ▶ if ( !exists( $converted{$field} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
unless (exists $converted{$field}) 1 585 586
1133 push(
1134 1 @{$errors},
1135 $twhere
1136 . '.numeric names "'
1137 . $field
1138 . '", which nothing produced; neither the pattern nor a decompose sets it'
1139 );
1140 1 next;
1141 } ## end if ( !exists( $converted{$field} ) )
1142 585 ▶ if ( !_holds_a_number( $converted{$field} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
unless (_holds_a_number($converted{$field})) 1 584 585
1143 push(
1144 1 ▶ @{$errors},
Branch: 1 branch
BranchTrueFalseTotal
defined $converted{$field} ? : 1 0 1
1145 $twhere
1146 . '.numeric names "'
1147 . $field
1148 . '", which came out as the string "'
1149 1 . ( defined( $converted{$field} ) ? $converted{$field} : '' )
1150 . '"; no convert turned it into a number'
1151 );
1152 } ## end if ( !_holds_a_number( $converted{$field} ...))
1153 } ## end foreach my $field ( @{$numeric} )
1154  
1155 338 return;
1156 } ## end sub _test_numeric
1157  
1158 # Lists the numeric conversions a rule's own tests do not check.
1159 #
1160 # Asking this per field rather than per rule matters: a rule that lists one
1161 # converted field satisfies a per-rule check while its other converts go
1162 # unwatched, which is exactly how http_access_logs kept a status code and a byte
1163 # count as strings while its port conversion was tested.
1164 #
1165 # Only fields the rule's own positive tests really produce are considered, and
1166 # only int and float entries -- lc, uc and mac turn a string into another string,
1167 # and a rule that inherits the file level map often produces none of the fields in
1168 # it. Warning about either would be noise, and noise in a lint teaches people to
1169 # skip the output.
1170 #
1171 # Args:
1172 #
1173 # - $rule :: The raw rule hash ref, read for its positive tests: their
1174 # strings, and the numeric lists they already carry.
1175 #
1176 # - $compiled :: The rule after Log::Munger::LogProcessor->_compile_rule.
1177 #
1178 # Returns a sorted list of field names, empty when everything convertible is
1179 # already listed.
1180 #
1181 # my @unchecked = _unchecked_converts( $rule, $compiled );
1182 276 ▶sub _unchecked_converts {
Subroutine _unchecked_converts: called CC=14 SCAR=26.6
1183 276 my ( $rule, $compiled ) = @_;
1184  
1185 648 my %numeric_type = map { $_ => 1 }
1186 667 ▶ grep { $compiled->{'convert'}{$_} eq 'int' || $compiled->{'convert'}{$_} eq 'float' }
Condition100%3/3
$compiled->{'convert'}{$_} eq 'int' || $compiled->{'convert'}{$_} eq 'float'
l!l&&r!l&&!r
6153319
MC/DC100%2/2
$compiled->{'convert'}{$_} eq 'int' || $compiled->{'convert'}{$_} eq 'float'
$compiled->{'convert'}{$_} eq 'int' $compiled->{'convert'}{$_} eq 'float'
Truth table100%3/3
A || B
A= $compiled->{'convert'}{$_} eq 'int'B= $compiled->{'convert'}{$_} eq 'float'
ABresult
1X1
011
000
1187 276 keys( %{ $compiled->{'convert'} } );
1188 276 ▶ if ( !keys(%numeric_type) ) {
Branch: 1 branch
BranchTrueFalseTotal
unless (keys %numeric_type) 123 153 276
1189 123 return ();
1190 }
1191  
1192 153 my %listed;
1193 my %produced;
1194 153 ▶ foreach my $test ( @{ $rule->{'tests'}{'positive'} || [] } ) {
Condition50%1/2
$rule->{'tests'}{'positive'} || []
l!l
1530
MC/DC0%0/1
$rule->{'tests'}{'positive'} || []
$rule->{'tests'}{'positive'}
Truth table50%1/2
A
A= $rule->{'tests'}{'positive'}
Aresult
11
00
1195 1585 ▶ next if ( ref($test) ne 'HASH' || !defined( $test->{'string'} ) );
Branch: 1 branch
BranchTrueFalseTotal
if ref $test ne "HASH" or not defined $test->{"string"} 0 1585 1585
Condition33%1/3
ref $test ne "HASH" or not defined $test->{"string"}
l!l&&r!l&&!r
001585
MC/DC0%0/2
ref $test ne "HASH" or not defined $test->{"string"}
ref $test ne "HASH" not defined $test->{"string"}
Truth table33%1/3
A or not(B)
A= ref $test ne "HASH"B= not defined $test->{"string"}
ABresult
1X1
011
000
1196 1585 ▶ $listed{$_} = 1 for @{ $test->{'numeric'} || [] };
Condition100%2/2
$test->{'numeric'} || []
l!l
3371248
MC/DC100%1/1
$test->{'numeric'} || []
$test->{'numeric'}
Truth table100%2/2
A
A= $test->{'numeric'}
Aresult
11
00
1197  
1198 1585 my $captures = _first_match( $compiled, $test->{'string'} );
1199 1585 ▶ next if ( !defined($captures) || !keys( %{$captures} ) );
Branch: 1 branch
BranchTrueFalseTotal
unless defined $captures and keys %{$captures;} 6 1579 1585
Condition66%2/3
defined $captures and keys %{$captures;}
!ll&&!rl&&r
061579
MC/DC50%1/2
defined $captures and keys %{$captures;}
defined $captures keys %{$captures;}
Truth table66%2/3
A and B
A= defined $capturesB= keys %{$captures;}
ABresult
0X0
100
111
1200  
1201 1579 ▶ if ( @{ $compiled->{'decompose'} } ) {
Branch: 1 branch
BranchTrueFalseTotal
if (@{$compiled->{"decompose"};}) 491 1088 1579
1202 491 Log::Munger::LogProcessor->_decompose( $compiled, $captures );
1203 }
1204 1579 foreach my $field ( keys( %{$captures} ) ) {
1205 6200 ▶ $produced{$field} = 1 if ( $numeric_type{$field} );
Branch: 1 branch
BranchTrueFalseTotal
if $numeric_type{$field} 1275 4925 6200
1206 }
1207 } ## end foreach my $test ( @{ $rule->{'tests'}{'positive'...}})
1208  
1209 153 return sort grep { !$listed{$_} } keys(%produced);
1210 } ## end sub _unchecked_converts
1211  
1212 # Lists the fields a rule's decompose produces that no enriched map names.
1213 #
1214 # Per field for the same reason as the converts above: one enriched case covers
1215 # the entries that fired for that line, and a rule whose other tests reach a
1216 # different entry would otherwise look covered.
1217 #
1218 # A file level decompose is shared by every rule in the file and most of them
1219 # capture none of the fields it names, so a rule whose decompose never fires for
1220 # any of its own tests is not asked about.
1221 #
1222 # Args:
1223 #
1224 # - $rule :: The raw rule hash ref, read for its positive tests and the
1225 # enriched maps they already carry.
1226 #
1227 # - $compiled :: The rule after Log::Munger::LogProcessor->_compile_rule.
1228 #
1229 # Returns a sorted list of field names, empty when nothing is missing.
1230 #
1231 # my @unwired = _unchecked_decomposes( $rule, $compiled );
1232 276 ▶sub _unchecked_decomposes {
Subroutine _unchecked_decomposes: called CC=12 SCAR=25.6
1233 276 my ( $rule, $compiled ) = @_;
1234  
1235 276 ▶ if ( !@{ $compiled->{'decompose'} } ) {
Branch: 1 branch
BranchTrueFalseTotal
unless (@{$compiled->{"decompose"};}) 140 136 276
1236 140 return ();
1237 }
1238  
1239 136 my %named;
1240 my %produced;
1241 136 ▶ foreach my $test ( @{ $rule->{'tests'}{'positive'} || [] } ) {
Condition50%1/2
$rule->{'tests'}{'positive'} || []
l!l
1360
MC/DC0%0/1
$rule->{'tests'}{'positive'} || []
$rule->{'tests'}{'positive'}
Truth table50%1/2
A
A= $rule->{'tests'}{'positive'}
Aresult
11
00
1242 717 ▶ next if ( ref($test) ne 'HASH' || !defined( $test->{'string'} ) );
Branch: 1 branch
BranchTrueFalseTotal
if ref $test ne "HASH" or not defined $test->{"string"} 0 717 717
Condition33%1/3
ref $test ne "HASH" or not defined $test->{"string"}
l!l&&r!l&&!r
00717
MC/DC0%0/2
ref $test ne "HASH" or not defined $test->{"string"}
ref $test ne "HASH" not defined $test->{"string"}
Truth table33%1/3
A or not(B)
A= ref $test ne "HASH"B= not defined $test->{"string"}
ABresult
1X1
011
000
1243 717 ▶ $named{$_} = 1 for keys( %{ $test->{'enriched'} || {} } );
Condition100%2/2
$test->{'enriched'} || {}
l!l
160557
MC/DC100%1/1
$test->{'enriched'} || {}
$test->{'enriched'}
Truth table100%2/2
A
A= $test->{'enriched'}
Aresult
11
00
1244  
1245 717 my $captures = _first_match( $compiled, $test->{'string'} );
1246 717 ▶ next if ( !defined($captures) || !keys( %{$captures} ) );
Branch: 1 branch
BranchTrueFalseTotal
unless defined $captures and keys %{$captures;} 0 717 717
Condition33%1/3
defined $captures and keys %{$captures;}
!ll&&!rl&&r
00717
MC/DC0%0/2
defined $captures and keys %{$captures;}
defined $captures keys %{$captures;}
Truth table33%1/3
A and B
A= defined $capturesB= keys %{$captures;}
ABresult
0X0
100
111
1247  
1248 717 my %split = %{$captures};
1249 717 Log::Munger::LogProcessor->_decompose( $compiled, \%split );
1250 717 foreach my $field ( keys(%split) ) {
1251 3329 ▶ $produced{$field} = 1 if ( !exists( $captures->{$field} ) );
Branch: 1 branch
BranchTrueFalseTotal
unless exists $captures->{$field} 1378 1951 3329
1252 }
1253 } ## end foreach my $test ( @{ $rule->{'tests'}{'positive'...}})
1254  
1255 136 return sort grep { !$named{$_} } keys(%produced);
1256 } ## end sub _unchecked_decomposes
1257  
1258 # Runs a compiled rule's gates against a program name. Each gate is evaluated
1259 # by Log::Munger::LogProcessor->_gate_passes, so this is the engine's own gate
1260 # check rather than a parallel copy of it.
1261 #
1262 # The engine gates on a whole record, but every gate any rule file has ever
1263 # needed keys on PROGRAM, so a test names a program rather than building a
1264 # record. A gate gets a synthetic { PROGRAM => $program } to look at; one keyed
1265 # on some other field finds nothing there and refuses, which is the right answer
1266 # for a test that has not said what that field holds.
1267 #
1268 # Args:
1269 #
1270 # - $compiled :: The rule after Log::Munger::LogProcessor->_compile_rule.
1271 # Only its gate list is read, each entry being
1272 # { field, literals => {}, regexps => [] }.
1273 #
1274 # - $program :: The program name to gate, as a plain string, such as
1275 # "dnsmasq-dhcp" or "/usr/sbin/cron".
1276 #
1277 # Returns undef when every gate passes, otherwise a string naming the gate that
1278 # refused and what it wanted, ready to be dropped into an error message.
1279 #
1280 # my $why = _gate_refusal( $compiled, 'cron' );
1281 # # 'gate 0 on PROGRAM does not accept "cron"'
1282 709 ▶sub _gate_refusal {
Subroutine _gate_refusal: called CC=3 SCAR=11.0
1283 709 my ( $compiled, $program ) = @_;
1284  
1285 709 my $item = { 'PROGRAM' => $program };
1286  
1287 709 my $gate_int = 0;
1288 709 foreach my $gate ( @{ $compiled->{'gate'} } ) {
1289 709 ▶ if ( !Log::Munger::LogProcessor->_gate_passes( $gate, $item->{ $gate->{'field'} } ) ) {
Branch: 1 branch
BranchTrueFalseTotal
unless ("Log::Munger::LogProcessor"->_gate_passes($gate, $$item{$gate->{"field"}})) 247 462 709
1290 247 return 'gate ' . $gate_int . ' on ' . $gate->{'field'} . ' does not accept "' . $program . '"';
1291 }
1292 462 $gate_int++;
1293 }
1294  
1295 462 return undef;
1296 } ## end sub _gate_refusal
1297  
1298 # Compares two flat capture hashes and describes the first way they differ.
1299 #
1300 # Both directions are checked, so a capture that was expected and never appeared
1301 # is caught, and so is one that appeared without being asked for. That second case
1302 # matters more than it looks: an unexpected capture usually means a pattern picked
1303 # up a group nobody meant to add, and without checking for it a test would pass
1304 # while the rule quietly emitted a field consumers were not written for.
1305 #
1306 # Comparison is string comparison, which is what the rule files want. Everything
1307 # coming out of a regexp capture is a string, and the expected results in YAML are
1308 # written to match.
1309 #
1310 # Args:
1311 #
1312 # - $expected :: Hash ref of the captures the test says there should be, as
1313 # written in the rule file. Values are compared as strings.
1314 #
1315 # - $got :: Hash ref of the captures that actually came out, a copy of %+.
1316 #
1317 # Returns undef if the two agree, otherwise a string describing the first
1318 # difference found, ready to be dropped into an error message. Only the first is
1319 # reported, since a single wrong pattern usually throws off several keys at once
1320 # and listing them all says no more than listing one.
1321 #
1322 # my $diff = _capture_diff( { ssh_user => 'alice' }, { ssh_user => 'bob' } );
1323 # # 'key "ssh_user" expected "alice" but got "bob"'
1324 2398 ▶sub _capture_diff {
Subroutine _capture_diff: called CC=6 SCAR=17.9
1325 2398 my ( $expected, $got ) = @_;
1326  
1327 2398 foreach my $key ( sort keys( %{$expected} ) ) {
1328 8906 ▶ if ( !exists( $got->{$key} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
unless (exists $got->{$key}) 2 8904 8906
1329 2 return 'expected key "' . $key . '" not captured';
1330 }
1331 8904 ▶ if ( $expected->{$key} ne $got->{$key} ) {
Branch: 1 branch
BranchTrueFalseTotal
if ($expected->{$key} ne $got->{$key}) 1 8903 8904
1332 1 return 'key "' . $key . '" expected "' . $expected->{$key} . '" but got "' . $got->{$key} . '"';
1333 }
1334 }
1335 2395 foreach my $key ( sort keys( %{$got} ) ) {
1336 8903 ▶ if ( !exists( $expected->{$key} ) ) {
Branch: 1 branch
BranchTrueFalseTotal
unless (exists $expected->{$key}) 1 8902 8903
1337 1 return 'unexpected key "' . $key . '" captured as "' . $got->{$key} . '"';
1338 }
1339 }
1340  
1341 2394 return undef;
1342 } ## end sub _capture_diff
1343  
1344 # Tests a decompose list, either a rule's own or the file level default.
1345 #
1346 # Every entry is compiled on its own first, which is what surfaces an unknown
1347 # type, leftover grok in a pattern entry, or a pattern that will not compile.
1348 #
1349 # An entry that carries tests is then exercised one entry at a time. The captures
1350 # hash is built as nothing but { field => input }, the single entry is applied to
1351 # it, and what comes out has to equal the expected result. Testing entries in
1352 # isolation like this means a list of several does not have to be reasoned about
1353 # as a whole, and it keeps a change to one entry from breaking the tests of the
1354 # others.
1355 #
1356 # The expected result has to account for the entry's remove: setting. With remove
1357 # set the source field is gone by the time the comparison happens, so listing it
1358 # would fail; without it the source field is still there and leaving it out would
1359 # fail as an unexpected capture.
1360 #
1361 # Args:
1362 #
1363 # - $list :: The decompose list, as written in the rule file. An array ref of
1364 # { field, type, ... } hash refs.
1365 #
1366 # - $vars :: The resolved vars hash ref, which a "type: pattern" entry
1367 # resolves its pattern name against. An empty hash ref is fine for a list
1368 # that only uses kv or json entries.
1369 #
1370 # - $errors :: Array ref to push error strings onto. Appended to in place.
1371 #
1372 # - $warnings :: Array ref to push warning strings onto. Appended to in
1373 # place. An entry with no tests at all lands here rather than in errors.
1374 #
1375 # - $where :: Where this list lives, written as a path, such as ".decompose"
1376 # for the file level one or ".rules.3.decompose" for a rule's own. The
1377 # per-entry and per-test messages extend it.
1378 #
1379 # Returns nothing. Everything it finds goes onto $errors or $warnings.
1380 #
1381 # _test_decompose( $rules->{'decompose'}, $vars, \@errors, \@warnings, '.decompose' );
1382 64 ▶sub _test_decompose {
Subroutine _test_decompose: called CC=13 SCAR=34.6
1383 64 my ( $list, $vars, $errors, $warnings, $where ) = @_;
1384  
1385 64 ▶ if ( ref($list) ne 'ARRAY' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $list ne "ARRAY") 0 64 64
1386 0 push( @{$errors}, $where . ' has a ref of "' . ref($list) . '" and not "ARRAY"' );
1387 0 return;
1388 }
1389  
1390 64 my $int = 0;
1391 64 foreach my $entry ( @{$list} ) {
1392 151 my $ewhere = $where . '.' . $int;
1393  
1394 # compile just this entry (surfaces bad type / grok / regexp)
1395 151 my $compiled;
1396 151 eval { $compiled = Log::Munger::LogProcessor->_compile_decompose( [$entry], $vars ); };
1397 151 ▶ if ($@) {
Branch: 1 branch
BranchTrueFalseTotal
if ($@) 0 151 151
1398 0 push( @{$errors}, $ewhere . ' failed to compile... ' . $@ );
1399 0 $int++;
1400 0 next;
1401 }
1402  
1403 151 ▶ if ( !defined( $entry->{'tests'} ) ) {
Branch: 2 branches
BranchTrueFalseTotal
if (not defined $entry->{'tests'}) { } 0 151 151
elsif (ref $entry->{'tests'} ne 'ARRAY') { } 0 151 151
1404 0 push( @{$warnings}, $ewhere . ' lacks any tests' );
1405 0 $int++;
1406 0 next;
1407 } elsif ( ref( $entry->{'tests'} ) ne 'ARRAY' ) {
1408 0 push( @{$errors}, $ewhere . '.tests has a ref of "' . ref( $entry->{'tests'} ) . '" and not "ARRAY"' );
1409 0 $int++;
1410 0 next;
1411 }
1412  
1413 151 my $rule = { 'decompose' => $compiled };
1414 151 my $test_int = 0;
1415 151 foreach my $test ( @{ $entry->{'tests'} } ) {
1416 237 my $twhere = $ewhere . '.tests.' . $test_int;
1417 237 ▶ if ( ref($test) ne 'HASH' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $test ne "HASH") 0 237 237
1418 0 push( @{$errors}, $twhere . ' has a ref of "' . ref($test) . '" and not "HASH"' );
1419 0 $test_int++;
1420 0 next;
1421 }
1422 237 ▶ if ( !defined( $test->{'input'} ) || ref( $test->{'input'} ) ne '' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (not defined $test->{"input"} or ref $test->{"input"} ne "") 0 237 237
Condition33%1/3
not defined $test->{"input"} or ref $test->{"input"} ne ""
l!l&&r!l&&!r
00237
MC/DC0%0/2
not defined $test->{"input"} or ref $test->{"input"} ne ""
not defined $test->{"input"} ref $test->{"input"} ne ""
Truth table33%1/3
not(A) or B
A= not defined $test->{"input"}B= ref $test->{"input"} ne ""
ABresult
1X1
011
000
1423 0 push( @{$errors}, $twhere . '.input is undef or not a string' );
1424 0 $test_int++;
1425 0 next;
1426 }
1427 237 ▶ my $expected = defined( $test->{'result'} ) ? $test->{'result'} : {};
Branch: 1 branch
BranchTrueFalseTotal
defined $test->{'result'} ? : 237 0 237
1428 237 ▶ if ( ref($expected) ne 'HASH' ) {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $expected ne "HASH") 0 237 237
1429 0 push( @{$errors}, $twhere . '.result has a ref of "' . ref($expected) . '" and not "HASH"' );
1430 0 $test_int++;
1431 0 next;
1432 }
1433  
1434 237 my %captures = ( $entry->{'field'} => $test->{'input'} );
1435 237 Log::Munger::LogProcessor->_decompose( $rule, \%captures );
1436  
1437 237 my $diff = _capture_diff( $expected, \%captures );
1438 237 ▶ if ( defined($diff) ) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $diff) 1 236 237
1439 push(
1440 1 @{$errors},
1441 1 $twhere . ' decompose output differs: ' . $diff . ' input="' . $test->{'input'} . '"'
1442 );
1443 }
1444  
1445 237 $test_int++;
1446 } ## end foreach my $test ( @{ $entry->{'tests'} } )
1447  
1448 151 $int++;
1449 } ## end foreach my $entry ( @{$list} )
1450  
1451 64 return;
1452 } ## end sub _test_decompose
1453  
1454 19 1; # End of Log::Munger::RulesTest
« lib/Log/Munger/App/Command/list_fields.pm ↑ Summary lib/Log/Munger/LogProcessor.pm »