| line |
stmt |
bran |
cond |
sub |
pod |
time |
code |
|
1
|
|
|
|
|
|
|
=head1 NAME |
|
2
|
|
|
|
|
|
|
|
|
3
|
|
|
|
|
|
|
Synapse::Monitor::Listener |
|
4
|
|
|
|
|
|
|
|
|
5
|
|
|
|
|
|
|
|
|
6
|
|
|
|
|
|
|
=head1 About Synapse's Open Telephony Toolkit |
|
7
|
|
|
|
|
|
|
|
|
8
|
|
|
|
|
|
|
L is a part of Synapse's Wholesale Open Telephony Toolkit. |
|
9
|
|
|
|
|
|
|
|
|
10
|
|
|
|
|
|
|
As we are refactoring our codebase, we at Synapse have decided to release a |
|
11
|
|
|
|
|
|
|
substantial portion of our code under the GPL. We hope that as a developer, you |
|
12
|
|
|
|
|
|
|
will find it as useful as we have found open source and CPAN to be of use to |
|
13
|
|
|
|
|
|
|
us. |
|
14
|
|
|
|
|
|
|
|
|
15
|
|
|
|
|
|
|
|
|
16
|
|
|
|
|
|
|
=head1 What is L all about |
|
17
|
|
|
|
|
|
|
|
|
18
|
|
|
|
|
|
|
In the wholesale telecom business, you need to monitor a lot of stuff. For |
|
19
|
|
|
|
|
|
|
example, for each egress and ingress route (and there's hundreds of them and |
|
20
|
|
|
|
|
|
|
thousands of dialcodes), you need to permaently check answer seizure ration, |
|
21
|
|
|
|
|
|
|
average length of call, MOS, customer credit limit usage, make sure your |
|
22
|
|
|
|
|
|
|
systems are up and ping-eable, CPU load, etc. etc. |
|
23
|
|
|
|
|
|
|
|
|
24
|
|
|
|
|
|
|
The idea is that you have a collection of scripts / daemons / cron jobs / etc |
|
25
|
|
|
|
|
|
|
which product Ip. Events have an associated type (e.g. 'ping'), |
|
26
|
|
|
|
|
|
|
I (e.g. UNKNOWN, OK, WARNING, DOWN), and indentifier (e.g. ping |
|
27
|
|
|
|
|
|
|
example-dot-com). |
|
28
|
|
|
|
|
|
|
|
|
29
|
|
|
|
|
|
|
What L does is that it picks up newly created |
|
30
|
|
|
|
|
|
|
events and can be configured to perform a set of predefined actions when it |
|
31
|
|
|
|
|
|
|
detects an I, such as firing an email, suspending a service, |
|
32
|
|
|
|
|
|
|
blocking an IP address, etc. |
|
33
|
|
|
|
|
|
|
|
|
34
|
|
|
|
|
|
|
|
|
35
|
|
|
|
|
|
|
=head1 L overview and installation |
|
36
|
|
|
|
|
|
|
|
|
37
|
|
|
|
|
|
|
The library is split as follows: |
|
38
|
|
|
|
|
|
|
|
|
39
|
|
|
|
|
|
|
=over 4 |
|
40
|
|
|
|
|
|
|
|
|
41
|
|
|
|
|
|
|
=item script synapse-monitor-listener-cli, used to create and set configuration objects. |
|
42
|
|
|
|
|
|
|
|
|
43
|
|
|
|
|
|
|
=item script synapse-monitor-listener-service, which is a daemon desgined to be |
|
44
|
|
|
|
|
|
|
running in the background |
|
45
|
|
|
|
|
|
|
|
|
46
|
|
|
|
|
|
|
=item L, listener object, which is designed to |
|
47
|
|
|
|
|
|
|
choose what action(s) to do with .evt.yml objects / files. |
|
48
|
|
|
|
|
|
|
|
|
49
|
|
|
|
|
|
|
=item L, listener action object, which is |
|
50
|
|
|
|
|
|
|
designed to define and execute arbitrary actions. |
|
51
|
|
|
|
|
|
|
|
|
52
|
|
|
|
|
|
|
=back |
|
53
|
|
|
|
|
|
|
|
|
54
|
|
|
|
|
|
|
You install the package as follows: |
|
55
|
|
|
|
|
|
|
|
|
56
|
|
|
|
|
|
|
perl Makefile.PL |
|
57
|
|
|
|
|
|
|
make |
|
58
|
|
|
|
|
|
|
make test |
|
59
|
|
|
|
|
|
|
make install |
|
60
|
|
|
|
|
|
|
synapse-monitor-listener-cli --create-configdir |
|
61
|
|
|
|
|
|
|
synapse-monitor-listener-service install |
|
62
|
|
|
|
|
|
|
synapse-monitor-listener-service start |
|
63
|
|
|
|
|
|
|
|
|
64
|
|
|
|
|
|
|
synapse-monitor-listener-service should then start on each reboot, runlevels |
|
65
|
|
|
|
|
|
|
3-5. You can check that this is the case and that the daemon has been correctly |
|
66
|
|
|
|
|
|
|
installed using the excellent chkconfig linux tool. |
|
67
|
|
|
|
|
|
|
|
|
68
|
|
|
|
|
|
|
|
|
69
|
|
|
|
|
|
|
=head1 L configuration |
|
70
|
|
|
|
|
|
|
|
|
71
|
|
|
|
|
|
|
Note: L does NOT need restarting once you have |
|
72
|
|
|
|
|
|
|
changed the configuration. |
|
73
|
|
|
|
|
|
|
|
|
74
|
|
|
|
|
|
|
For the sake of the example - and because this package is designed to be a |
|
75
|
|
|
|
|
|
|
telephony package after all - let's say we have a bunch of call detail record |
|
76
|
|
|
|
|
|
|
(or "cdr") files on the filesystem. Each file contains a list of calls, wether |
|
77
|
|
|
|
|
|
|
they were answered or not, the call duration, etc. |
|
78
|
|
|
|
|
|
|
|
|
79
|
|
|
|
|
|
|
Say we want to check and monitor ASR values, i.e. how many calls were answered |
|
80
|
|
|
|
|
|
|
on the last 100 calls. We have written a script, called asrcheck.pl, which runs |
|
81
|
|
|
|
|
|
|
in the background, and which produces notification files looking like this: |
|
82
|
|
|
|
|
|
|
|
|
83
|
|
|
|
|
|
|
--- |
|
84
|
|
|
|
|
|
|
id: asr-myvoipsupplier-moroccomobilemeditel |
|
85
|
|
|
|
|
|
|
state: OK |
|
86
|
|
|
|
|
|
|
listener: asr |
|
87
|
|
|
|
|
|
|
destination: Morocco-Mobile-Meditel |
|
88
|
|
|
|
|
|
|
vendor: myvoipsupplier |
|
89
|
|
|
|
|
|
|
notification-email: customer.service@example.com |
|
90
|
|
|
|
|
|
|
head-wc: 100 |
|
91
|
|
|
|
|
|
|
asr: 62 |
|
92
|
|
|
|
|
|
|
|
|
93
|
|
|
|
|
|
|
Notification files should be placed in /tmp/ and end with .evt.yml. |
|
94
|
|
|
|
|
|
|
synapse-monitor-listener-service will process these files as they appear and |
|
95
|
|
|
|
|
|
|
delete them after they have been processed. |
|
96
|
|
|
|
|
|
|
|
|
97
|
|
|
|
|
|
|
The only required fields are "listener", "state", and "id". All the rest is |
|
98
|
|
|
|
|
|
|
optional but your underlying "action" scpripts could use the extra information. |
|
99
|
|
|
|
|
|
|
A copy of the file will be passed to them as $ENV{YAML_FILE}. |
|
100
|
|
|
|
|
|
|
|
|
101
|
|
|
|
|
|
|
=over 4 |
|
102
|
|
|
|
|
|
|
|
|
103
|
|
|
|
|
|
|
=item id - should be unique across all listeners. i.e. there shouldn't be an |
|
104
|
|
|
|
|
|
|
"asr" listener called "foo" and an "acd" listener called "foo". Set up your |
|
105
|
|
|
|
|
|
|
scripts to call your checks "asr-foo" and "acd-foo" instead. |
|
106
|
|
|
|
|
|
|
|
|
107
|
|
|
|
|
|
|
=item state - can be any string without spaces. The number of states should be |
|
108
|
|
|
|
|
|
|
discrete and finite. |
|
109
|
|
|
|
|
|
|
|
|
110
|
|
|
|
|
|
|
=item listener - which listener configuration to use on this type of check. |
|
111
|
|
|
|
|
|
|
|
|
112
|
|
|
|
|
|
|
=back |
|
113
|
|
|
|
|
|
|
|
|
114
|
|
|
|
|
|
|
For the sake of the example, say we have 3 possible states: |
|
115
|
|
|
|
|
|
|
|
|
116
|
|
|
|
|
|
|
|
|
117
|
|
|
|
|
|
|
=over 4 |
|
118
|
|
|
|
|
|
|
|
|
119
|
|
|
|
|
|
|
=item OK : when the ASR is > 30 |
|
120
|
|
|
|
|
|
|
|
|
121
|
|
|
|
|
|
|
=item WARNING : when the ASR is < 30 but > 10 |
|
122
|
|
|
|
|
|
|
|
|
123
|
|
|
|
|
|
|
=item DOWN : when the ASR is < 10 |
|
124
|
|
|
|
|
|
|
|
|
125
|
|
|
|
|
|
|
=back |
|
126
|
|
|
|
|
|
|
|
|
127
|
|
|
|
|
|
|
|
|
128
|
|
|
|
|
|
|
We need to let L know what to do when there is a |
|
129
|
|
|
|
|
|
|
I on this type of notification. |
|
130
|
|
|
|
|
|
|
|
|
131
|
|
|
|
|
|
|
|
|
132
|
|
|
|
|
|
|
First, let's configure an "asr" listener: |
|
133
|
|
|
|
|
|
|
|
|
134
|
|
|
|
|
|
|
# first of all, create our "ASR listener" object |
|
135
|
|
|
|
|
|
|
synapse-monitor-listener-cli type listener create asr "ASR listener" |
|
136
|
|
|
|
|
|
|
|
|
137
|
|
|
|
|
|
|
# DOWN -> OK, or WARN -> OK : cool... |
|
138
|
|
|
|
|
|
|
synapse-monitor-listener-cli listener asr action WARN.DOWN OK asr-email-goodjob |
|
139
|
|
|
|
|
|
|
|
|
140
|
|
|
|
|
|
|
# OK -> WARN : send warning email |
|
141
|
|
|
|
|
|
|
synapse-monitor-listener-cli listener asr action OK WARN asr-email-warning |
|
142
|
|
|
|
|
|
|
|
|
143
|
|
|
|
|
|
|
# OK -> DOWN or WARN -> DOWN = send "down" email + suspend route |
|
144
|
|
|
|
|
|
|
synapse-monitor-listener-cli listener asr action OK.WARN DOWN asr-email-down suspend-route |
|
145
|
|
|
|
|
|
|
|
|
146
|
|
|
|
|
|
|
|
|
147
|
|
|
|
|
|
|
Now that's done, let's configure matching actions: |
|
148
|
|
|
|
|
|
|
|
|
149
|
|
|
|
|
|
|
# a copy of the YAML notification file will be passed as $ENV{YAML_FILE}, set up your scripts accordingly... |
|
150
|
|
|
|
|
|
|
# format is |
|
151
|
|
|
|
|
|
|
# synapse-monitor-listener-cli type action create useless-action echo I_AM_USELESS >/dev/null |
|
152
|
|
|
|
|
|
|
|
|
153
|
|
|
|
|
|
|
# to delete an action: |
|
154
|
|
|
|
|
|
|
# synapse-monitor-listener-cli action useless-action remove |
|
155
|
|
|
|
|
|
|
|
|
156
|
|
|
|
|
|
|
synapse-monitor-listener-cli type action create asr-email-goodjob synapse-email-notification /etc/synapse-monitor/emails/goodjob.xml |
|
157
|
|
|
|
|
|
|
synapse-monitor-listener-cli type action create asr-email-warning synapse-email-notification /etc/synapse-monitor/emails/warning.xml |
|
158
|
|
|
|
|
|
|
synapse-monitor-listener-cli type action create asr-email-down synapse-email-notification /etc/synapse-monitor/emails/down.xml |
|
159
|
|
|
|
|
|
|
synapse-monitor-listener-cli type action create email-restored synapse-email-notification /etc/synapse-monitor/emails/restored.xml |
|
160
|
|
|
|
|
|
|
synapse-monitor-listener-cli type action create suspend-route synapse-suspend-route |
|
161
|
|
|
|
|
|
|
|
|
162
|
|
|
|
|
|
|
|
|
163
|
|
|
|
|
|
|
That's it. You don't need to restart the daemon: the configuration changes are |
|
164
|
|
|
|
|
|
|
picked up immediately. |
|
165
|
|
|
|
|
|
|
|
|
166
|
|
|
|
|
|
|
=cut |
|
167
|
|
|
|
|
|
|
package Synapse::Monitor::Listener; |
|
168
|
1
|
|
|
1
|
|
933
|
use base qw /Synapse::CLI::Config::Object/; |
|
|
1
|
|
|
|
|
2
|
|
|
|
1
|
|
|
|
|
1216
|
|
|
169
|
1
|
|
|
1
|
|
91483
|
use Synapse::Logger; |
|
|
1
|
|
|
|
|
622
|
|
|
|
1
|
|
|
|
|
68
|
|
|
170
|
1
|
|
|
1
|
|
7
|
use YAML::XS; |
|
|
1
|
|
|
|
|
14
|
|
|
|
1
|
|
|
|
|
54
|
|
|
171
|
1
|
|
|
1
|
|
5
|
use warnings; |
|
|
1
|
|
|
|
|
1
|
|
|
|
1
|
|
|
|
|
30
|
|
|
172
|
1
|
|
|
1
|
|
5
|
use strict; |
|
|
1
|
|
|
|
|
2
|
|
|
|
1
|
|
|
|
|
1136
|
|
|
173
|
|
|
|
|
|
|
|
|
174
|
|
|
|
|
|
|
|
|
175
|
|
|
|
|
|
|
our $VERSION = 0.3; |
|
176
|
|
|
|
|
|
|
our $EVTDIR = '/tmp'; |
|
177
|
|
|
|
|
|
|
our $EVTEXT = '.evt.yml'; |
|
178
|
|
|
|
|
|
|
|
|
179
|
|
|
|
|
|
|
|
|
180
|
|
|
|
|
|
|
sub action { |
|
181
|
42
|
|
|
42
|
1
|
30565
|
my $self = shift; |
|
182
|
42
|
|
|
|
|
68
|
my $before = shift; |
|
183
|
42
|
|
|
|
|
53
|
my $after = shift; |
|
184
|
42
|
|
|
|
|
84
|
my @actions = @_; |
|
185
|
42
|
100
|
|
|
|
123
|
if ($before =~ /\./) { |
|
186
|
12
|
|
|
|
|
38
|
for my $before (split /\./, $before) { |
|
187
|
24
|
|
|
|
|
67
|
$self->action ($before, $after, @actions); |
|
188
|
|
|
|
|
|
|
} |
|
189
|
12
|
|
|
|
|
81
|
return $self; |
|
190
|
|
|
|
|
|
|
} |
|
191
|
30
|
50
|
|
|
|
74
|
if ($after =~ /\./) { |
|
192
|
0
|
|
|
|
|
0
|
for my $after (split /\./, $after) { |
|
193
|
0
|
|
|
|
|
0
|
$self->action ($before, $after, @actions); |
|
194
|
|
|
|
|
|
|
} |
|
195
|
0
|
|
|
|
|
0
|
return $self; |
|
196
|
|
|
|
|
|
|
} |
|
197
|
|
|
|
|
|
|
|
|
198
|
30
|
|
100
|
|
|
100
|
$self->{action} ||= {}; |
|
199
|
30
|
|
100
|
|
|
121
|
$self->{action}->{$before} ||= {}; |
|
200
|
30
|
|
|
|
|
83
|
$self->{action}->{$before}->{$after} = \@_; |
|
201
|
30
|
|
|
|
|
121
|
return $self; |
|
202
|
|
|
|
|
|
|
} |
|
203
|
|
|
|
|
|
|
|
|
204
|
|
|
|
|
|
|
|
|
205
|
|
|
|
|
|
|
sub process { |
|
206
|
5
|
|
|
5
|
0
|
10334
|
my $self = shift; |
|
207
|
5
|
|
|
|
|
10
|
my $oldState = shift; |
|
208
|
5
|
|
|
|
|
7
|
my $newState = shift; |
|
209
|
5
|
|
|
|
|
10
|
my $event = shift; |
|
210
|
5
|
100
|
|
|
|
19
|
$self->{action}->{$oldState} || do { |
|
211
|
1
|
|
|
|
|
5
|
logger ($self->name() . ": no action specified for old state $oldState"); |
|
212
|
1
|
|
|
|
|
106
|
return; |
|
213
|
|
|
|
|
|
|
}; |
|
214
|
4
|
50
|
|
|
|
15
|
$self->{action}->{$oldState}->{$newState} || do { |
|
215
|
0
|
|
|
|
|
0
|
logger ($self->name() . ": no action specified for old state $oldState to new state $newState"); |
|
216
|
0
|
|
|
|
|
0
|
return; |
|
217
|
|
|
|
|
|
|
}; |
|
218
|
|
|
|
|
|
|
|
|
219
|
4
|
|
|
|
|
15
|
logger ($self->name() . ": iterating actions for $oldState -> $newState"); |
|
220
|
4
|
|
|
|
|
495
|
for my $action (@{$self->{action}->{$oldState}->{$newState}}) { |
|
|
4
|
|
|
|
|
16
|
|
|
221
|
5
|
|
|
|
|
22
|
logger ("action: $action"); |
|
222
|
5
|
|
|
|
|
449
|
my $action_obj = Synapse::Monitor::Listener::Action->new ($action); |
|
223
|
5
|
50
|
|
|
|
826
|
if ($action_obj) { $action_obj->process ($event) } |
|
|
5
|
|
|
|
|
20
|
|
|
224
|
|
|
|
|
|
|
else { |
|
225
|
0
|
|
|
|
|
0
|
logger ("$action: cannot instantiate object - skipping"); |
|
226
|
0
|
|
|
|
|
0
|
next; |
|
227
|
|
|
|
|
|
|
} |
|
228
|
|
|
|
|
|
|
}; |
|
229
|
|
|
|
|
|
|
} |
|
230
|
|
|
|
|
|
|
|
|
231
|
|
|
|
|
|
|
|
|
232
|
|
|
|
|
|
|
sub __evtfiles__() { |
|
233
|
3
|
|
|
3
|
|
141
|
opendir EVTDIR, $EVTDIR; |
|
234
|
3
|
|
|
|
|
184
|
my @files = readdir (EVTDIR); |
|
235
|
3
|
|
|
|
|
58
|
closedir EVTDIR; |
|
236
|
3
|
|
|
|
|
7
|
my @res = (); |
|
237
|
3
|
|
|
|
|
8
|
for my $file (@files) { |
|
238
|
21
|
100
|
|
|
|
129
|
$file =~ /\Q$EVTEXT\E$/ or next; |
|
239
|
3
|
50
|
|
|
|
63
|
-e "$EVTDIR/$file" or next; |
|
240
|
3
|
50
|
|
|
|
48
|
-e "$EVTDIR/$file.lock" and next; |
|
241
|
3
|
|
|
|
|
10
|
push @res, "$EVTDIR/$file"; |
|
242
|
|
|
|
|
|
|
} |
|
243
|
3
|
|
|
|
|
13
|
return @res; |
|
244
|
|
|
|
|
|
|
} |
|
245
|
|
|
|
|
|
|
|
|
246
|
|
|
|
|
|
|
|
|
247
|
|
|
|
|
|
|
sub __loadfile__($) { |
|
248
|
3
|
|
|
3
|
|
3
|
my $file = shift; |
|
249
|
3
|
50
|
|
|
|
118
|
open YAMLFILE, $file or return; |
|
250
|
3
|
|
|
|
|
70
|
my $data = join '', ; |
|
251
|
3
|
|
|
|
|
33
|
close YAMLFILE; |
|
252
|
3
|
|
|
|
|
153
|
return Load $data; |
|
253
|
|
|
|
|
|
|
} |
|
254
|
|
|
|
|
|
|
|
|
255
|
|
|
|
|
|
|
|
|
256
|
|
|
|
|
|
|
sub runonce { |
|
257
|
3
|
|
|
3
|
0
|
4230
|
my $class = shift; |
|
258
|
3
|
|
|
|
|
11
|
for my $file (__evtfiles__) { |
|
259
|
3
|
|
|
|
|
10
|
my $event = __loadfile__ $file; |
|
260
|
3
|
|
|
|
|
278
|
unlink $file; |
|
261
|
3
|
50
|
|
|
|
11
|
$event || next; |
|
262
|
|
|
|
|
|
|
|
|
263
|
3
|
|
50
|
|
|
10
|
my $id = $event->{id} || next; |
|
264
|
3
|
|
50
|
|
|
8
|
my $newState = $event->{state} || next; |
|
265
|
3
|
|
50
|
|
|
11
|
my $listener = $event->{listener} || next; |
|
266
|
|
|
|
|
|
|
|
|
267
|
3
|
|
66
|
|
|
29
|
my $oldState = Synapse::Monitor::Listener::State->new ($id) || Synapse::Monitor::Listener::State->create ($id => 'UNKNOWN'); |
|
268
|
3
|
|
|
|
|
958
|
$oldState = $oldState->label(); |
|
269
|
|
|
|
|
|
|
|
|
270
|
3
|
50
|
|
|
|
23
|
$oldState eq $newState and next; |
|
271
|
3
|
|
50
|
|
|
18
|
$listener = $class->new ($listener) || next; |
|
272
|
3
|
|
|
|
|
58
|
eval { $listener->process ($oldState, $newState, $event) }; |
|
|
3
|
|
|
|
|
11
|
|
|
273
|
3
|
|
|
|
|
16
|
Synapse::CLI::Config::execute ("Synapse::Monitor::Listener::State", $id, "set", "label", $newState); |
|
274
|
|
|
|
|
|
|
} |
|
275
|
|
|
|
|
|
|
} |
|
276
|
|
|
|
|
|
|
|
|
277
|
|
|
|
|
|
|
|
|
278
|
|
|
|
|
|
|
1; |
|
279
|
|
|
|
|
|
|
|
|
280
|
|
|
|
|
|
|
|
|
281
|
|
|
|
|
|
|
__END__ |