blib/lib/Database/BI/Model/Geocode.pm

filter: Statementstmt 99.1% 460 / 464 Branchbran 95.4% 233 / 244 Conditioncond 84.9% 192 / 226 MC/DCmcdc 70.7% 92 / 130 Subroutinesub 100% 41 / 41 Podpod 100% 12 / 12 totaltotal 92.2% 1030 / 1117 CC 221 SCAR 28.2
CC
221
Cov
95%
_resolve
40.0
_lookup_online
26.2
resolve_column
25.0
CRAP 16.8
« blib/lib/Database/BI/Insights.pm ↑ Summary script/database-bi »
1 package Database::BI::Model::Geocode;
2  
3 # Resolve a location column to coordinates, once per distinct value.
4 #
5 # A dataset with places but no coordinates offers
6 # loc=<column>, and the distinct values of that column are looked up by a
7 # Geo::Coder::List chain: Geo::Coder::Free's local data first, then
8 # OpenStreetMap's Nominatim service (Geo::Coder::OSM) for the places the local
9 # data does not know. This module is that lookup, kept out of the controller so
10 # that the interesting part - which values are worth asking about, which were
11 # asked already, and what happens to the rest - is testable without an HTTP
12 # request, a data file, or a geocoder on the machine.
13 #
14 # The rule it is built around is that a dirty column is normal rather than an
15 # error. A place column holds street names, blank cells, and the odd bare number
16 # that a column-offset mistake put there. Two things stop that from ending a
17 # request:
18 #
19 # 1. Every distinct value is put to the geocoder's own valid_location() before
20 # anything is looked up. A value it refuses is counted, reported, and
21 # dropped: it never reaches a backend, so it cannot croak, and it is not
22 # counted as a lookup.
23 # 2. The batch is made with on_error => 'undef', so a value the geocoder
24 # refuses later is a miss rather than a death. Nothing is wrapped in an
25 # eval: a backend exception still propagates, exactly as it does from
26 # geocode() itself.
27 #
28 # Skipping is remembered rather than repeated. An attempted-and-unresolved value,
29 # so the next request over the same column plots what resolved,
30 # reports the rest, and asks the geocoder nothing at all.
31  
32 3 ▶use strict;
Subroutine BEGIN: called CC=1 SCAR=0.0
33 3 ▶use warnings;
Subroutine BEGIN: called CC=1 SCAR=0.0
34  
35 3 ▶use Carp qw(croak carp);
Subroutine BEGIN: called CC=1 SCAR=0.0
36 3 ▶use Database::BI::I18N ();
Subroutine BEGIN: called CC=1 SCAR=0.0
37 3 ▶use Readonly;
Subroutine BEGIN: called CC=1 SCAR=0.0
38 3 ▶use Scalar::Util qw(blessed);
Subroutine BEGIN: called CC=1 SCAR=0.0
39 3 ▶use Time::HiRes ();
Subroutine BEGIN: called CC=1 SCAR=0.0
40 3 ▶use Params::Get ();
Subroutine BEGIN: called CC=1 SCAR=0.0
41 3 ▶use Params::Validate::Strict qw(validate_strict);
Subroutine BEGIN: called CC=1 SCAR=0.0
42  
43 3 our $VERSION = '0.023.1';
44  
45 # The version of Geo::Coder::Free this module asks for: 0.46 added valid_location()
46 # and on_error => 'undef', 0.47 added geocode_many(), and 0.48 added the
47 # per-lookup read counts the lookups figure below reports. This module asks for
48 # all three rather than reimplementing the rule, the batch loop or the count it
49 # exists to use. The messages that name the floor are given this number rather
50 # than spelling it out, so there is one place to change; a caller that pins
51 # min_geocoder_version for a test gets its own number quoted back, which is the
52 # floor it asked for. t/recommended-versions.t holds the constant and the
53 # Makefile.PL recommends entry to each other.
54 3 Readonly my $MIN_GEOCODER_VERSION => '0.48';
55  
56 # The chain the lookups go through, and the online geocoder at its end.
57 # Geo::Coder::List 0.39 records every geocoder it asks in log(), which is how a
58 # lookup that reached Nominatim is told from one the local data answered, and
59 # how a Nominatim that did not answer is told from a place it does not know.
60 # These are named FLOOR rather than MIN_*VERSION: t/recommended-versions.t
61 # reads the file's one $MIN_*VERSION as Geo::Coder::Free's floor, and the
62 # messages that name these modules spell their numbers out instead.
63 3 Readonly my $LIST_FLOOR => '0.39';
64 3 Readonly my $OSM_FLOOR => '0.03';
65 3 Readonly my $THROTTLED_FLOOR => '0.14';
66  
67 # Nominatim's usage policy: at most one request a second, from a client that
68 # names itself. The gap is kept by LWP::UserAgent::Throttled, which waits
69 # before a request rather than refusing it, and the user agent is shared by
70 # every lookup in the process, so two map pages opened together still keep it.
71 3 Readonly my $NOMINATIM_HOST => 'nominatim.openstreetmap.org';
72 3 Readonly my $NOMINATIM_GAP => 1;
73  
74 # Geo::Coder::OSM 0.03 asks http://nominatim..., which answers with a redirect
75 # to https: two requests a lookup, the second held a second by the throttle, and
76 # the place name sent once in clear. Its documented %SOURCES hook names this
77 # source instead, and it is asked for by name.
78 3 Readonly my $NOMINATIM_SOURCE => 'database-bi-https';
79 3 Readonly my $NOMINATIM_URL => "https://$NOMINATIM_HOST";
80 3 Readonly my $ONLINE_TIMEOUT => 10; # seconds; a service that has stopped answering must not hold a page
81 3 Readonly my $USER_AGENT => "Database-BI/$VERSION (+https://metacpan.org/dist/Database-BI)";
82  
83 # Geo::Coder::List's own cache (its L2), used when the caller gives none: one
84 # CHI Memory cache for the process, so the answers outlive the object, which the
85 # map page makes afresh for every request. A place Nominatim found is then not
86 # asked again while the process lives, whichever request needs it. The map page
87 # and script/bi-geocode pass a CHI File cache under model_dir instead, which
88 # also outlives the process and is shared by every hypnotoad worker.
89 3 Readonly my $CACHE_NAMESPACE => __PACKAGE__;
90 3 Readonly my $CACHE_MAX_SIZE => 4 * 1024 * 1024; # bytes, for the process-wide default
91 3 Readonly my $CACHE_DIR => 'geocode-cache'; # under model_dir, for file_cache()
92  
93 # The default for map_geocode_max (section 6.2): new lookups allowed in one
94 # request. A crafted URL must not be able to turn one request into unbounded CPU.
95 3 Readonly my $DEFAULT_MAX => 500;
96  
97 # The most characters of text place_text() lets be added to each place (the
98 # map page's loc_text=, bi-geocode's --text).
99 3 Readonly our $PLACE_TEXT_MAX => 200;
100  
101 # The model store's shared place cache. A place name means the same place
102 # whichever file's column holds it, so one answer serves every dataset: the
103 # second file that names Ramsgate plots it without asking the geocoder. The
104 # key cannot collide with a per-dataset entry, which always starts with a spec
105 # scheme (path:, table:, url: or redis:).
106 3 Readonly my $SHARED_CACHE_KEY => 'shared';
107  
108 # ---------------------------------------------------------------------------
109 # I18N message dictionary.
110 # Every user-visible string this module produces is keyed here. The keys are
111 # for the conditions it names - geocoder missing, the
112 # location column not being in the view, skipped rows, the cap, and no openaddr
113 # data - so the controller and this module describe the same conditions with the
114 # same words; the rest are this module's own argument and interface errors, which
115 # section 11 does not name because they are not states a page reaches. Pass an
116 # object that responds to maketext($key, @args) as the "i18n" constructor
117 # argument to override them; it is called in preference to this table.
118 # ---------------------------------------------------------------------------
119  
120 3 Readonly our %MESSAGES => (
121 'error_map_geocoder_missing' => 'Geocoding a location column needs Geo::Coder::Free %s or later, and %s',
122 'error_map_geocoder_list_missing' => 'Geocoding a location column needs Geo::Coder::List 0.39 or later, and %s',
123 'error_map_loc_column' => 'Column "%s" is not in this dataset, so there is nothing to geocode',
124 'error_geocoder_interface' => 'Geocode: the geocoder must provide %s; Geo::Coder::Free %s or later does',
125 'error_max_invalid' => 'Geocode: option "max" must be a positive integer, not "%s"',
126 'error_online_max_invalid' => 'Geocode: option "online_max" must be a positive integer, not "%s"',
127 'error_cache_dir' => 'Geocode: the geocode cache needs a directory that exists, not "%s"',
128 'error_cache_interface' => 'Geocode: option "cache" must be a hashref or an object with get() and set(), not a %s',
129 'error_values_arrayref' => 'Geocode: the values must be an arrayref, not "%s"',
130 'error_rows_arrayref' => 'Geocode: the rows must be an arrayref, not "%s"',
131 'error_column_required' => 'Geocode: the column name is required',
132 'warn_map_rows_skipped' => 'Plotted %d of %d rows; %d could not be plotted: %s',
133 # The reasons warn_map_rows_skipped gives, those that apply, joined with ', '.
134 'warn_map_rows_why_empty' => '%d with no place in the row',
135 'warn_map_rows_why_refused' => '%d whose place is a bare number, not a place name',
136 'warn_map_rows_why_not_found' => '%d whose place was not found',
137 'warn_map_rows_why_later' => '%d whose place has not been looked up yet',
138 'warn_map_geocode_capped' => 'Geocoded %d of the %d places that had not been looked up before; the rest are untouched, so open the data source again or run the geocode warm script to finish them',
139 'warn_map_geocoder_unavailable_data' => 'The geocoder has no OpenAddresses data configured (OPENADDR_HOME), so places resolve only as coarsely as the bundled MaxMind data allows',
140 'warn_map_geocoder_local_only' => 'Places the local data does not know were not looked up online, which needs Geo::Coder::OSM 0.03 and LWP::UserAgent::Throttled 0.14 or later',
141 'warn_map_geocoder_online_failed' => 'OpenStreetMap did not answer (%s), so %d places the local data does not know are left to try again later',
142 );
143  
144 # _fmt($key [, @args]) -> $string
145 #
146 # Package-level i18n formatter, for the messages new() raises before the object
147 # exists. Instance methods use _msg() so a caller-supplied i18n object can
148 # override the built-in strings.
149 163 ▶sub _fmt {
Subroutine _fmt: called CC=4 SCAR=13.9
150 163 my ($key, @args) = @_;
151  
152 163 ▶ my $tmpl = Database::BI::I18N::text(__PACKAGE__, \%MESSAGES, $key)
Condition100%4/4
Database::BI::I18N::text(__PACKAGE__, \%Database::BI::Model::Geocode::MESSAGES, $key) // q[Internal error: unknown message key '] . ($key // "undef") . q[']
l!l
1612
$key // "undef"
l!l
11
MC/DC100%1/1
Database::BI::I18N::text(__PACKAGE__, \%Database::BI::Model::Geocode::MESSAGES, $key) // q[Internal error: unknown message key '] . ($key // "undef") . q[']
Database::BI::I18N::text(__PACKAGE__, \%Database::BI::Model::Geocode::MESSAGES, $key)
MC/DC100%1/1
$key // "undef"
$key
Truth table100%2/2
A
A= Database::BI::I18N::text(__PACKAGE__, \%Database::BI::Model::Geocode::MESSAGES, $key)
Aresult
11
00
Truth table100%2/2
A
A= $key
Aresult
11
00
153 // "Internal error: unknown message key '" . ($key // 'undef') . "'";
154  
155 163 ▶ return @args ? sprintf($tmpl, @args) : $tmpl;
Branch: 1 branch
BranchTrueFalseTotal
@args ? : 122 41 163
156 }
157  
158 # _msg($self, $key [, @args]) -> $string
159 120 ▶sub _msg {
Subroutine _msg: called CC=2 SCAR=6.9
160 120 my ($self, $key, @args) = @_;
161  
162 120 ▶ if (my $i18n = $self->{'_i18n'}) {
Branch: 1 branch
BranchTrueFalseTotal
if (my $i18n = $$self{"_i18n"}) 3 117 120
163 3 return $i18n->maketext($key, @args);
164 }
165  
166 117 return _fmt($key, @args);
167 }
168  
169 # _message($self, $key [, @args]) -> { key => $key, text => $text }
170 #
171 # The form a message is reported in: the key lets a caller re-render it through
172 # its own i18n, and the text is what to show when it does not.
173 66 ▶sub _message {
Subroutine _message: called CC=1 SCAR=0.0
174 66 my ($self, $key, @args) = @_;
175  
176 66 return { 'key' => $key, 'text' => $self->_msg($key, @args) };
177 }
178  
179 =encoding utf8
180  
181 =head1 NAME
182  
183 Database::BI::Model::Geocode - Resolve a location column to coordinates, once per distinct value
184  
185 =head1 VERSION
186  
187 0.023.1
188  
189 =head1 SYNOPSIS
190  
191 B<Geocode a table's place column and build map markers:>
192  
193 use Database::BI::Model::Geocode;
194  
195 my $geocode = Database::BI::Model::Geocode->new();
196 my $result = $geocode->resolve_column($rows, 'city');
197  
198 for my $marker (@{ $result->{'markers'} }) {
199 printf " %s: lat=%.4f lon=%.4f (%d rows)\n",
200 $marker->{'value'}, $marker->{'lat'}, $marker->{'lon'},
201 $marker->{'rows'};
202 }
203  
204 my $c = $result->{'counts'};
205 printf "Plotted %d of %d rows (%d skipped, %d unresolved)\n",
206 $c->{'plotted'}, $c->{'rows'}, $c->{'skipped'}, $c->{'unresolved'};
207  
208 B<Re-use remembered answers so the next request costs nothing:>
209  
210 # Load what was stored from a previous run.
211 my $spec = 'table:sales';
212 my $models = decode_json(Mojo::File->new('.bi/models.json')->slurp);
213 my $cache = Database::BI::Model::Geocode->cache_from_models($models, $spec);
214  
215 # Resolve; cached values are used without asking the geocoder.
216 my $result = $geocode->resolve_column($rows, 'city', cache => $cache);
217  
218 # Save the updated cache back to the models file.
219 my $updated = Database::BI::Model::Geocode->models_with_cache(
220 $models, $spec, $result->{'cache'},
221 backend => $result->{'backend'},
222 built => '2026-10-18T12:00:00',
223 );
224 Mojo::File->new('.bi/models.json')->spew(encode_json($updated));
225  
226 B<Warm the cache for a whole column without needing rows:>
227  
228 # Extract unique city values from your own data.
229 my @cities = do { my %s; grep { !$s{$_}++ } map { $_->{'city'} } @{ $rows } };
230  
231 my $result = $geocode->resolve_values(\@cities,
232 cache => $cache,
233 region => 'GB',
234 max => 1000,
235 );
236  
237 printf "Resolved %d / %d new places\n",
238 $result->{'counts'}{'resolved'}, $result->{'counts'}{'attempted'};
239  
240 B<Ask whether the geocoder is available before offering it in the UI:>
241  
242 my $geocode = Database::BI::Model::Geocode->new();
243 my ($ok, $why) = $geocode->geocoder_available();
244  
245 unless ($ok) {
246 carp "Map panel disabled: $why";
247 }
248  
249 B<Inject a test double so geocoding works without a real geocoder:>
250  
251 package MyGeocoderStub;
252 sub new { bless {}, shift }
253 sub valid_location { my ($self, $v) = @_; $v && $v !~ /\A\d+\z/ }
254 sub geocode_many {
255 my ($self, $places, %opts) = @_;
256 return { 'London' => { lat => 51.5, long => -0.12, database => 'stub' } };
257 }
258  
259 my $geocode = Database::BI::Model::Geocode->new(geocoder => MyGeocoderStub->new);
260 my $result = $geocode->resolve_column($rows, 'city');
261  
262 =head1 DESCRIPTION
263  
264 The geocoder is a L<Geo::Coder::List> chain:
265  
266 Geo::Coder::List->new()
267 ->push(Geo::Coder::Free->new()) # the local data
268 ->push(Geo::Coder::OSM->new()); # OpenStreetMap's Nominatim
269  
270 C<Geo::Coder::Free> answers from data on this machine. A place it does not know
271 is asked of Nominatim, over the network, through an L<LWP::UserAgent::Throttled>
272 that keeps Nominatim's usage policy: one request a second, from a client that
273 names itself (C<Database-BI/I<version>>), with a ten-second time-out. The user
274 agent is shared by every lookup in the process, so the gap holds across
275 requests. Nominatim is told the place name and nothing else.
276  
277 A table column holds something less tidy than a place name: values repeated
278 thousands of times, blank cells, numbers that drifted in from an adjacent
279 column. This module does the three things the column then needs.
280  
281 =over 4
282  
283 =item B<Ask once per distinct value.>
284  
285 The column is reduced to a set before any lookup happens. A repeated city name
286 costs one lookup, so a 100,000-row file with a dozen cities costs twelve lookups,
287 not 100,000.
288  
289 =item B<Refuse a value that is not a location before any backend sees it.>
290  
291 C<< $geocoder->valid_location($value) >> is the gate the geocoder's own guard
292 applies before looking anything up. This module asks the same question first, so
293 a pure-digit string (a row number that ended up in the wrong column), an empty
294 cell, or a Perl reference is counted and reported as skipped rather than sent to
295 a backend that would reject it anyway.
296  
297 =item B<Look the rest up without letting one value end the batch.>
298  
299 Each place goes down a C<Geo::Coder::List> on its own: the local data, then
300 Nominatim when it is in the chain. With the local data alone the list holds
301 only C<Geo::Coder::Free>, which is given the C<region> hint (C<Geo::Coder::OSM>
302 would put it in the Nominatim query). Either way the list's cache answers what
303 it has found before. C<Geo::Coder::List> catches what a geocoder throws and
304 asks the next one, so a throw does not end the request; a place whose lookup
305 threw and found nothing is held (put in C<not_attempted>, not cached), because "not found" is
306 remembered for good and an error is not an answer.
307  
308 An injected C<geocoder> is asked for one C<geocode_many()> batch instead,
309 requested with C<< on_error => 'undef' >>. A value it refuses after the
310 pre-filter is a miss; a backend that throws is still a throw, because nothing
311 here wraps it in an C<eval>.
312  
313 =item B<Tell a Nominatim that failed from a place it does not know.>
314  
315 C<Geo::Coder::OSM> answers nothing in both cases. The chain's log says whether
316 Nominatim was asked, and its last response says whether it answered. A place
317 it failed on is held, not remembered as not found, and Nominatim is not asked
318 again in that request: the rest are asked of the local data alone, and the ones
319 it does not know are held too. The page is told Nominatim's status line and how
320 many places wait, never which.
321  
322 =back
323  
324 The counts the module returns add up, always.
325  
326 skipped + cached + attempted + not_attempted == distinct
327 resolved + unresolved == cached + attempted
328  
329 C<lookups> is the number of places that reached a geocoder rather than the
330 chain's own cache. For an injected geocoder it is the number that geocoder
331 reports for the batch, which C<Geo::Coder::Free> 0.48 and later keeps lower than
332 C<attempted> when a value is answered from a warm process cache. It never exceeds
333 C<attempted>. C<online> is how many of them were asked of Nominatim.
334  
335 =head2 The cache
336  
337 A cache entry is a C<[lat, lon]> arrayref for a place that resolved, or the
338 scalar C<0> for a place that was looked up and not found. That C<0> sentinel
339 prevents a column of unresolvable values from being retried on every page
340 request.
341  
342 The cache is passed per call and returned per call. The caller's hashref is
343 never modified: a request that dies part way through cannot leave a half-written
344 cache behind. The controller is responsible for persisting the returned cache,
345 which keeps the C<model_dir>-inside-C<data_dir> guard in the one place that
346 already owns it.
347  
348 A previously stored cache is used even when the geocoder is not available.
349 Places already resolved continue to be plotted; only new lookups are unavailable.
350  
351 =head2 Without the geocoder
352  
353 C<Geo::Coder::Free> and C<Geo::Coder::List> are recommended dependencies, not
354 required ones. They are loaded the first time they are needed, not at compile
355 time. C<Geo::Coder::OSM> and C<LWP::UserAgent::Throttled> are recommended too;
356 without them the chain is the local data alone, and a request with places it
357 did not know says so (C<warn_map_geocoder_local_only>). When
358 C<Geo::Coder::Free> or C<Geo::Coder::List> is absent or older than its minimum
359 version:
360  
361 =over 4
362  
363 =item *
364  
365 C<< $geocode->geocoder_available() >> returns C<(0, $reason)>.
366  
367 =item *
368  
369 C<resolve_column> and C<resolve_values> still return a complete result: all
370 values are placed in C<not_attempted>, cached answers are still plotted, and
371 the C<error_map_geocoder_missing> message explains the situation.
372  
373 =item *
374  
375 Nothing croaks. The page renders with whatever the cache already holds.
376  
377 =back
378  
379 =head1 CONSTRUCTOR
380  
381 =head2 new
382  
383 my $geocode = Database::BI::Model::Geocode->new(%options);
384 my $geocode = Database::BI::Model::Geocode->new(\%options); # hashref also accepted
385  
386 Create a new resolver. Accepts a flat key/value list or a hashref; both forms
387 are handled by C<Params::Get>.
388  
389 =head3 API SPECIFICATION
390  
391 =head4 INPUT
392  
393 geocoder => object, # optional; duck-typed: must provide valid_location() and geocode_many()
394 openaddr => string, # optional; path to OpenAddresses data (default: $ENV{OPENADDR_HOME})
395 min_geocoder_version => string, # optional; minimum version to accept (default: '0.48')
396 i18n => object, # optional; must implement maketext($key, @args)
397 online => boolean, # optional; ask Nominatim about places the local data does not know (default: 1)
398 online_max => positive integer, # optional; places asked of Nominatim per call (default: no limit)
399 cache => hashref|object, # optional; Geo::Coder::List's cache (a hashref, or an object with get() and set(), such as file_cache()); default: files under cache_dir (below)
400 cache_dir => string, # optional; the directory for that cache, read through Object::Configure (below)
401 model_dir => string, # optional; the default cache_dir is <model_dir>/geocode-cache
402 config => hashref, # optional; the application's config, for its cache_dir (default: database_bi.conf is read)
403 home => string, # optional; the folder a relative cache_dir in config is under
404  
405 Without C<cache>, the answers are kept as files (a L<CHI> C<File> cache, with
406 a copy in memory) in the first directory that is set of:
407  
408 =over 4
409  
410 =item 1. C<cache_dir> from L<Object::Configure>: its config files, then the
411 environment variable C<Database__BI__Model__Geocode__cache_dir>; the
412 C<cache_dir> argument is what those fall back on;
413  
414 =item 2. C<$ENV{CACHEDIR}>;
415  
416 =item 3. C<$ENV{CACHE_DIR}>;
417  
418 =item 4. C<cache_dir> in F<database_bi.conf> (C<config>, or the file itself
419 when C<config> is not given);
420  
421 =item 5. F<E<lt>model_dirE<gt>/geocode-cache>, when C<model_dir> is given.
422  
423 =back
424  
425 With none of them, the answers are kept in one CHI C<Memory> cache per process.
426 L<Database::BI::CacheDir> makes the choice; the directory chosen is
427 C<< $geocode->{cache_dir} >>. Several caches may share a directory: each keeps
428 its own namespace in it.
429  
430 C<geocoder> is the injection seam for tests. An injected geocoder is used
431 exactly as given: it is not version-checked, so any object that responds to
432 C<valid_location()> and C<geocode_many()> works. A geocoder that fails either
433 C<can()> check causes an immediate C<croak> (C<error_geocoder_interface>), not
434 a silent degradation.
435  
436 C<openaddr> overrides C<$ENV{OPENADDR_HOME}>. It is forwarded to
437 C<< Geo::Coder::Free->new(openaddr => ...) >>. When it is absent and the
438 environment variable is not set, the geocoder uses its built-in MaxMind data
439 only, which resolves city and country names but not street addresses.
440 C<warn_map_geocoder_unavailable_data> is emitted in that case.
441  
442 C<min_geocoder_version> lets the caller pin the floor, or a test simulate
443 a too-old installation on a machine that has the current version.
444  
445 C<< online => 0 >> keeps every lookup on this machine: a place the local data
446 does not know is then not sent to OpenStreetMap. C<online_max> caps how many
447 places one C<resolve_*()> call asks Nominatim about. At one a second, it is
448 roughly how long the call can wait for Nominatim; the places past it go into
449 C<not_attempted> uncached, so the next call asks them. An injected
450 C<geocoder> is used as it is, and these two do not apply to it.
451  
452 C<i18n> is an optional object whose C<maketext($key, @args)> method is called
453 in place of the built-in C<%MESSAGES> table. The key is one of the constants
454 defined in C<%MESSAGES>; the args are the same positional values C<sprintf>
455 would receive.
456  
457 =head4 OUTPUT
458  
459 Returns C<$self>.
460  
461 Croaks with C<error_geocoder_interface> if the injected C<geocoder> does not
462 provide both C<valid_location()> and C<geocode_many()>, and with
463 C<error_online_max_invalid> if C<online_max> is not a positive integer.
464  
465 Never croaks because the geocoder module is absent or too old: that is a
466 runtime condition reported by C<geocoder_available()>, not a constructor error.
467  
468 =head1 SUBROUTINES/METHODS
469  
470 =head2 resolve_column
471  
472 my $result = $geocode->resolve_column(\@rows, $column_name, %options);
473  
474 Resolve one named column of a table. C<\@rows> is an arrayref of hashrefs
475 (the records from the data pipeline), and C<$column_name> is the key to look
476 up in each row.
477  
478 This is the main entry point for the map controller. It runs the same
479 resolution as L</resolve_values>, then adds per-marker row counts and the
480 C<markers> list that a map rendering loop needs.
481  
482 =head3 API SPECIFICATION
483  
484 =head4 INPUT
485  
486 \@rows => arrayref of hashrefs, required
487 $column_name => non-empty string, required
488 cache => hashref, # optional; { value => [lat, lon] | 0 }
489 region => two-letter string, # optional; country hint for the geocoder (e.g. 'GB')
490 max => positive integer, # optional; max new lookups; default 500
491 lookup => 0 | 1, # optional; 0 asks nothing and lists what it would ask; default 1
492  
493 C<\@rows> is the full record set as returned by the pipeline. Rows that are
494 not hashrefs produce an C<undef> cell, which is classified as skipped.
495  
496 C<$column_name> must be a non-empty string. It is used as a hash key; it is
497 not validated against a schema, so a column name with spaces or special
498 characters is accepted if the data uses one.
499  
500 C<cache> holds the already-known answers for this column. Entries are either
501 a C<[lat, lon]> arrayref (resolved) or C<0> (looked up and not found). The
502 caller's hashref is never modified; the returned C<cache> key holds the
503 updated copy.
504  
505 C<region> is passed straight through to the geocoder as a country hint. It
506 is not derived from the Accept-Language header, because a language is not a
507 country.
508  
509 C<max> is counted in new lookups, not rows. A cached answer costs nothing
510 and does not count toward it. Values held back by the cap go into
511 C<not_attempted> and are not stored in the cache, so the next request picks
512 them up.
513  
514 C<lookup =E<gt> 0> plots from C<cache> alone and asks the geocoder nothing.
515 The places it would have asked (up to C<max>, in first-seen order) are returned
516 in C<pending>, and are counted in C<not_attempted> meanwhile. A caller can then
517 look them up one at a time with L</resolve_values>, saving each answer as it
518 comes, which is how the map page shows its progress and can be cancelled. The
519 geocoder is still loaded, because its rule decides which values are places.
520  
521 =head4 OUTPUT
522  
523 Returns a hashref:
524  
525 {
526 available => 1 | 0,
527 points => { $value => { lat => $n, lon => $n, rows => $n, first_row => $n, backend? => $s } },
528 markers => [ { value => $s, lat => $n, lon => $n, rows => $n, first_row => $n } ],
529 counts => {
530 rows => $n, # total rows in \@rows
531 plotted => $n, # rows whose value resolved to a point
532 unusable => $n, # rows - plotted
533 distinct => $n, # unique values seen
534 skipped => $n, # refused before any lookup (empty, ref, digit-only)
535 cached => $n, # found in the cache (resolved or not)
536 attempted => $n, # sent to the geocoder this request
537 lookups => $n, # lookups the geocoder reports (may be < attempted)
538 online => $n, # of those, places asked of Nominatim
539 resolved => $n, # total with a position (cache + new)
540 unresolved => $n, # total without (cache + new)
541 not_attempted => $n, # held back by a cap, by a failed lookup, or because no geocoder
542 },
543 skipped => [ $value, ... ], # values refused before lookup
544 unresolved => [ $value, ... ], # values tried (now or before) and not found
545 not_attempted => [ $value, ... ], # held back by the cap
546 pending => [ $value, ... ], # with lookup => 0: what would have been asked; else empty
547 messages => [ { key => $k, text => $t }, ... ],
548 cache => { $value => [lat, lon] | 0 }, # updated copy; never the caller's hashref
549 backends => { $backend_name => $count },
550 backend => $backend_name | undef, # undef when nothing resolved
551 capped => 0 | 1,
552 }
553  
554 C<points> holds one entry per value that produced a position, whether the
555 position came from the cache or from this request. C<markers> is the same
556 data in first-seen row order, which reads better on a map than alphabetical
557 order.
558  
559 C<backend> on a point entry is set only for values looked up in this request;
560 cached values do not carry one because the cache stores a position, not which
561 database gave it. The top-level C<backend> is the name of the backend that
562 answered the most lookups this request.
563  
564 The count invariant always holds:
565  
566 skipped + cached + attempted + not_attempted == distinct
567  
568 =head3 MESSAGES
569  
570 error_rows_arrayref -- \@rows is not an arrayref
571 error_column_required -- $column_name is empty or undef
572 error_map_loc_column -- column not found in any row
573 error_map_geocoder_missing -- no geocoder, or version below the floor
574 error_map_geocoder_list_missing -- Geo::Coder::List absent or too old
575 warn_map_rows_skipped -- some rows could not be plotted, and why
576 warn_map_rows_why_* -- the reasons it names (not messages of their own)
577 warn_map_geocode_capped -- max or online_max held some values back
578 warn_map_geocoder_unavailable_data -- local data only, and OPENADDR_HOME not set
579 warn_map_geocoder_local_only -- no Geo::Coder::OSM, and places were not found
580 warn_map_geocoder_online_failed -- Nominatim did not answer; places held
581  
582 =head3 EXAMPLE
583  
584 my $cache = Database::BI::Model::Geocode->cache_from_models($models, $spec);
585 my $result = $geocode->resolve_column($rows, 'city',
586 cache => $cache,
587 region => 'GB',
588 max => 200,
589 );
590  
591 if ($result->{'capped'}) {
592 carp "Some cities were not looked up; load the page again to finish them.";
593 }
594  
595 for my $m (@{ $result->{'markers'} }) {
596 printf "[%.4f, %.4f] %s (%d rows)\n",
597 $m->{'lat'}, $m->{'lon'}, $m->{'value'}, $m->{'rows'};
598 }
599  
600 =cut
601  
602 123 ▶sub new {
Subroutine new: called CC=16 SCAR=27.7
Pod documented
603 # Strategy: normalise the argument list with Params::Get so a caller may pass
604 # a hashref or a flat list interchangeably, then validate before using
605 # anything. An injected geocoder is checked for the methods this module asks
606 # of one, rather than failing half way down a column.
607 123 my $class = shift;
608 123 ▶ my $raw = Params::Get::get_params(undef, \@_) // {};
Condition100%2/2
Params::Get::get_params(undef, \@_) // {}
l!l
9924
MC/DC100%1/1
Params::Get::get_params(undef, \@_) // {}
Params::Get::get_params(undef, \@_)
Truth table100%2/2
A
A= Params::Get::get_params(undef, \@_)
Aresult
11
00
609  
610 123 my $args = validate_strict(
611 schema => {
612 geocoder => { type => 'object', optional => 1, default => undef },
613 openaddr => { type => 'string', optional => 1, default => undef },
614 min_geocoder_version => { type => 'string', optional => 1, default => $MIN_GEOCODER_VERSION },
615 i18n => { type => 'object', optional => 1, default => undef, can => 'maketext' },
616 online => { type => 'scalar', optional => 1, default => 1 },
617 online_max => { type => 'scalar', optional => 1, default => undef },
618 cache => { type => 'object|hashref', optional => 1, default => undef },
619 cache_dir => { type => 'string', optional => 1, default => undef },
620 model_dir => { type => 'string', optional => 1, default => undef },
621 config => { type => 'hashref', optional => 1, default => undef },
622 home => { type => 'string', optional => 1, default => undef },
623 },
624 input => $raw,
625 );
626  
627 # Geo::Coder::List calls get() and set() on its cache inside an eval and
628 # carries on when they fail, so an object without them would cache nothing
629 # and say so only in a warning per place. It is refused here instead.
630 120 my $cache = $args->{'cache'};
631 120 ▶ croak(_fmt('error_cache_interface', ref($cache)))
Branch: 1 branch
BranchTrueFalseTotal
if builtin::blessed($cache) and not $cache->can("get") && $cache->can("set") 1 119 120
Condition83%5/6
builtin::blessed($cache) and not $cache->can("get") && $cache->can("set")
!ll&&!rl&&r
11721
$cache->can('get') && $cache->can('set')
!ll&&!rl&&r
102
MC/DC66%2/3
builtin::blessed($cache) and not $cache->can("get") && $cache->can("set")
builtin::blessed($cache) $cache->can('get') $cache->can('set')
Truth table75%3/4
A and not(B && C)
A= builtin::blessed($cache)B= $cache->can('get')C= $cache->can('set')
ABCresult
0XX0
1110
10X1
1101
632 if blessed($cache) && !($cache->can('get') && $cache->can('set'));
633  
634 # Without a cache object, the answers are kept on disc in the directory
635 # Database::BI::CacheDir chooses (Object::Configure, CACHEDIR, CACHE_DIR,
636 # database_bi.conf, then <model_dir>/geocode-cache). With none of those,
637 # they are kept in this process's memory, as before.
638 119 my $cache_dir;
639 119 ▶ unless (defined $cache) {
Branch: 1 branch
BranchTrueFalseTotal
unless (defined $cache) 108 11 119
640 108 require Database::BI::CacheDir;
641 $cache_dir = Database::BI::CacheDir::resolve(
642 class => $class,
643 params => $args,
644 (defined($args->{'config'}) ? (config => $args->{'config'}, home => $args->{'home'}) : ()),
645 default => (defined($args->{'model_dir'}) && length($args->{'model_dir'})
646 108 ▶ ? _catdir($args->{'model_dir'}, $CACHE_DIR) : undef),
Branch: 2 branches
BranchTrueFalseTotal
defined $$args{'config'} ? : 13 95 108
defined $$args{'model_dir'} && length $$args{'model_dir'} ? : 13 95 108
Condition66%2/3
defined $$args{"model_dir"} and length $$args{"model_dir"}
!ll&&!rl&&r
95013
MC/DC50%1/2
defined $$args{"model_dir"} and length $$args{"model_dir"}
defined $$args{"model_dir"} length $$args{"model_dir"}
Truth table66%2/3
A and B
A= defined $$args{"model_dir"}B= length $$args{"model_dir"}
ABresult
0X0
100
111
647 );
648 108 ▶ $cache = _file_cache_at($cache_dir) if defined $cache_dir;
Branch: 1 branch
BranchTrueFalseTotal
if defined $cache_dir 13 95 108
649 }
650  
651 # A cap that is not a positive integer is refused, as max is in _options:
652 # defaulting it would quietly lift the limit a caller set to keep a page quick.
653 119 my $online_max = $args->{'online_max'};
654 119 ▶ croak(_fmt('error_online_max_invalid', $online_max))
Branch: 1 branch
BranchTrueFalseTotal
if defined $online_max and not $online_max =~ /\A\d+\z/ && $online_max > 0 2 117 119
Condition100%6/6
defined $online_max and not $online_max =~ /\A\d+\z/ && $online_max > 0
!ll&&!rl&&r
101162
$online_max =~ /\A\d+\z/ && $online_max > 0
!ll&&!rl&&r
1116
MC/DC100%3/3
defined $online_max and not $online_max =~ /\A\d+\z/ && $online_max > 0
defined $online_max $online_max =~ /\A\d+\z/ $online_max > 0
Truth table100%4/4
A and not(B && C)
A= defined $online_maxB= $online_max =~ /\A\d+\z/C= $online_max > 0
ABCresult
0XX0
1110
10X1
1101
655 if defined($online_max) && !(($online_max =~ /\A\d+\z/) && ($online_max > 0));
656  
657 my $self = bless {
658 '_i18n' => $args->{'i18n'},
659 'openaddr' => $args->{'openaddr'} // $ENV{'OPENADDR_HOME'},
660 'min_version' => $args->{'min_geocoder_version'},
661 117 ▶ 'online' => $args->{'online'} ? 1 : 0,
Branch: 1 branch
BranchTrueFalseTotal
$$args{'online'} ? : 107 10 117
Condition100%3/3
$$args{"openaddr"} // $ENV{"OPENADDR_HOME"}
l!l&&r!l&&!r
23193
MC/DC100%2/2
$$args{"openaddr"} // $ENV{"OPENADDR_HOME"}
$$args{"openaddr"} $ENV{"OPENADDR_HOME"}
Truth table100%3/3
A // B
A= $$args{"openaddr"}B= $ENV{"OPENADDR_HOME"}
ABresult
1X1
011
000
662 'online_max' => $online_max,
663 'cache' => $cache,
664 'cache_dir' => $cache_dir,
665 }, $class;
666  
667 117 ▶ if (my $geocoder = $args->{'geocoder'}) {
Branch: 1 branch
BranchTrueFalseTotal
if (my $geocoder = $$args{"geocoder"}) 68 49 117
668 68 $self->_check_interface($geocoder);
669 64 $self->{'_geocoder'} = $geocoder;
670 }
671  
672 113 return $self;
673 }
674  
675 # _check_interface($geocoder)
676 #
677 # A geocoder has to answer the two questions this module asks of one. Checked
678 # here rather than half way through a column.
679 71 ▶sub _check_interface {
Subroutine _check_interface: called CC=3 SCAR=11.0
680 71 my ($self, $geocoder) = @_;
681  
682 71 for my $method (qw(valid_location geocode_many)) {
683 139 ▶ next if $geocoder->can($method);
Branch: 1 branch
BranchTrueFalseTotal
if $geocoder->can($method) 133 6 139
684 6 croak(_fmt('error_geocoder_interface', $method . '()', $MIN_GEOCODER_VERSION));
685 }
686  
687 65 return 1;
688 }
689  
690 # _load_geocoder() -> geocoder object, or undef when there is not one to be had
691 #
692 # The real geocoder, loaded on demand so the application installs, starts and
693 # tests without it: a Geo::Coder::List holding Geo::Coder::Free, and then
694 # Geo::Coder::OSM when online lookups are wanted and it can be had. Returns
695 # nothing when Geo::Coder::Free or Geo::Coder::List is absent, older than its
696 # floor, will not load, or builds nothing, and records which it was for the
697 # message. Geo::Coder::OSM is not needed: without it the chain is the local
698 # data alone, and a page whose places it does not know says why.
699 29 ▶sub _load_geocoder {
Subroutine _load_geocoder: called CC=12 SCAR=25.8
700 29 my $self = shift;
701  
702 29 my $ok = eval {
703 29 require Geo::Coder::Free;
704 23 Geo::Coder::Free->VERSION($self->{'min_version'});
705 23 1;
706 };
707  
708 29 ▶ unless ($ok) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($ok) 6 23 29
709 6 my $error = $@;
710  
711 6 ▶ $self->{'_unavailable'} =
Branch: 2 branches
BranchTrueFalseTotal
$error =~ /Can't locate/ ? : 6 0 6
defined $Geo::Coder::Free::VERSION ? : 0 0 0
712 ($error =~ /Can't locate/) ? 'it is not installed'
713 : (defined($Geo::Coder::Free::VERSION) ? "$Geo::Coder::Free::VERSION is installed"
714 : 'it could not be loaded');
715  
716 6 return;
717 }
718  
719 23 my $have_list = eval {
720 23 require Geo::Coder::List;
721 22 Geo::Coder::List->VERSION($LIST_FLOOR);
722 22 1;
723 };
724  
725 23 ▶ unless ($have_list) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($have_list) 1 22 23
726 1 my $error = $@;
727  
728 1 $self->{'_unavailable_key'} = 'error_map_geocoder_list_missing';
729 1 ▶ $self->{'_unavailable'} =
Branch: 2 branches
BranchTrueFalseTotal
$error =~ /Can't locate/ ? : 1 0 1
defined $Geo::Coder::List::VERSION ? : 0 0 0
730 ($error =~ /Can't locate/) ? 'it is not installed'
731 : (defined($Geo::Coder::List::VERSION) ? "$Geo::Coder::List::VERSION is installed"
732 : 'it could not be loaded');
733  
734 1 return;
735 }
736  
737 my $local = Geo::Coder::Free->new(
738 22 ▶ $self->{'openaddr'} ? (openaddr => $self->{'openaddr'}) : ()
Branch: 1 branch
BranchTrueFalseTotal
$$self{'openaddr'} ? : 0 22 22
739 );
740  
741 22 ▶ unless ($local) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($local) 0 22 22
742 # It is installed, so the message must not fall back on saying that it is not.
743 0 $self->{'_unavailable'} = 'it could not be started';
744 0 return;
745 }
746  
747 # Two lists over one cache: the local data alone, which is asked with the
748 # region hint (Geo::Coder::OSM would put it in the Nominatim query), and the
749 # whole chain when Nominatim is wanted. What either finds is in the cache
750 # for the other.
751 22 ▶ my $cache = $self->{'cache'} // _default_cache();
Condition66%2/3
$$self{"cache"} // _default_cache()
l!l&&r!l&&!r
11110
MC/DC0%0/2
$$self{"cache"} // _default_cache()
$$self{"cache"} _default_cache()
Truth table66%2/3
A // B
A= $$self{"cache"}B= _default_cache()
ABresult
1X1
011
000
752 22 my $chain = Geo::Coder::List->new(cache => $cache)->push($local);
753 22 $self->{'_local'} = $local;
754 22 $self->{'_local_chain'} = $chain;
755  
756 22 ▶ if ($self->{'online'}) {
Branch: 1 branch
BranchTrueFalseTotal
if ($$self{"online"}) 16 6 22
757 16 ▶ if (my $online = _online_geocoder()) {
Branch: 1 branch
BranchTrueFalseTotal
if (my $online = _online_geocoder()) { } 15 1 16
758 15 $chain = Geo::Coder::List->new(cache => $cache)->push($local)->push($online);
759 15 $self->{'_online'} = $online;
760 } else {
761 1 $self->{'_online_missing'} = 1;
762 }
763 }
764  
765 22 return $chain;
766 }
767  
768 # _default_cache() -> CHI cache
769 #
770 # The process-wide cache for Geo::Coder::List, made on first use. CHI is a
771 # prerequisite of the distribution, so this is not one more optional module.
772 3 my $default_cache;
773  
774 30 ▶sub _default_cache {
Subroutine _default_cache: called CC=2 SCAR=7.0
775 30 require CHI;
776  
777 30 ▶ $default_cache //= CHI->new(
Condition66%2/3
$default_cache //= "CHI"->new("driver", "Memory", "global", 1, "namespace", $CACHE_NAMESPACE, "max_size", $CACHE_MAX_SIZE)
l!l&&r!l&&!r
2820
MC/DC0%0/2
$default_cache //= "CHI"->new("driver", "Memory", "global", 1, "namespace", $CACHE_NAMESPACE, "max_size", $CACHE_MAX_SIZE)
$default_cache "CHI"->new("driver", "Memory", "global", 1, "namespace", $CACHE_NAMESPACE, "max_size", $CACHE_MAX_SIZE)
Truth table66%2/3
A //= B
A= $default_cacheB= "CHI"->new("driver", "Memory", "global", 1, "namespace", $CACHE_NAMESPACE, "max_size", $CACHE_MAX_SIZE)
ABresult
1X1
011
000
778 driver => 'Memory',
779 global => 1,
780 namespace => $CACHE_NAMESPACE,
781 max_size => $CACHE_MAX_SIZE,
782 );
783  
784 30 return $default_cache;
785 }
786  
787 # _online_geocoder() -> Geo::Coder::OSM object, or undef when it cannot be had
788 #
789 # Nominatim, behind a user agent that keeps its usage policy: one request a
790 # second, a name that says who is asking, and a time-out so a service that has
791 # stopped answering cannot hold a page. The user agent is made once per
792 # process, because the gap it keeps is measured from the last request it made:
793 # one per object would let two pages opened together ask twice in a second.
794 my $online_ua;
795  
796 21 ▶sub _online_geocoder {
Subroutine _online_geocoder: called CC=3 SCAR=11.0
797 21 my $ok = eval {
798 21 require Geo::Coder::OSM;
799 19 Geo::Coder::OSM->VERSION($OSM_FLOOR);
800 19 1;
801 };
802 21 ▶ return unless $ok && nominatim_ua();
Branch: 1 branch
BranchTrueFalseTotal
unless $ok and nominatim_ua() 2 19 21
Condition66%2/3
$ok and nominatim_ua()
!ll&&!rl&&r
2019
MC/DC50%1/2
$ok and nominatim_ua()
$ok nominatim_ua()
Truth table66%2/3
A and B
A= $okB= nominatim_ua()
ABresult
0X0
100
111
803  
804 {
805 3 ▶ no warnings qw(once); # the module's own variable, which only it reads
Subroutine BEGIN: called CC=1 SCAR=0.0
806 19 $Geo::Coder::OSM::SOURCES{$NOMINATIM_SOURCE} = $NOMINATIM_URL;
807 }
808  
809 19 return Geo::Coder::OSM->new(ua => $online_ua, sources => $NOMINATIM_SOURCE);
810 }
811  
812 =head2 nominatim_ua
813  
814 my $ua = Database::BI::Model::Geocode::nominatim_ua()
815 or return; # LWP::UserAgent::Throttled is not installed
816  
817 The user agent every request to Nominatim goes through, from the geocoder and
818 from L<Database::BI::Model::Boundary> alike: one per process, so the gap of a
819 second Nominatim's usage policy asks for holds across both, and across map
820 pages opened together. It names this application, gives up after ten seconds,
821 and follows the environment's proxy settings.
822  
823 =head3 API SPECIFICATION
824  
825 =head4 INPUT
826  
827 None.
828  
829 =head4 OUTPUT
830  
831 An L<LWP::UserAgent::Throttled> (an L<LWP::UserAgent>), the same object on
832 every call; or an empty list when LWP::UserAgent::Throttled 0.14 or later
833 cannot be loaded.
834  
835 =cut
836  
837 19 ▶sub nominatim_ua {
Subroutine nominatim_ua: called CC=3 SCAR=11.0
Pod documented
838 19 ▶ return $online_ua if $online_ua;
Branch: 1 branch
BranchTrueFalseTotal
if $online_ua 17 2 19
839  
840 2 my $ok = eval {
841 2 require LWP::UserAgent::Throttled;
842 2 LWP::UserAgent::Throttled->VERSION($THROTTLED_FLOOR);
843 2 1;
844 };
845 2 ▶ return unless $ok;
Branch: 1 branch
BranchTrueFalseTotal
unless $ok 0 2 2
846  
847 2 my $ua = LWP::UserAgent::Throttled->new(
848 agent => $USER_AGENT,
849 timeout => $ONLINE_TIMEOUT,
850 keep_alive => 1,
851 );
852 2 $ua->env_proxy();
853 2 $ua->throttle({ $NOMINATIM_HOST => $NOMINATIM_GAP });
854 2 $online_ua = $ua;
855  
856 2 return $online_ua;
857 }
858  
859 # _judge() -> the object whose valid_location() decides what is a place
860 #
861 # An injected geocoder answers for itself. Geo::Coder::List has no rule of its
862 # own, so the chain is judged by Geo::Coder::Free's, the module it asks first.
863 118 ▶sub _judge {
Subroutine _judge: called CC=2 SCAR=6.9
864 118 my ($self, $geocoder) = @_;
865  
866 118 ▶ return $geocoder->can('valid_location') ? $geocoder : $self->{'_local'};
Branch: 1 branch
BranchTrueFalseTotal
$geocoder->can('valid_location') ? : 87 31 118
867 }
868  
869 # _geocoder() -> geocoder object or undef
870 #
871 # Loaded once per object, so a missing module is not looked for on every value.
872 185 ▶sub _geocoder {
Subroutine _geocoder: called CC=2 SCAR=6.9
873 185 my $self = shift;
874  
875 185 ▶ if (!exists $self->{'_geocoder'}) {
Branch: 1 branch
BranchTrueFalseTotal
unless (exists $self->{"_geocoder"}) 29 156 185
876 29 $self->{'_geocoder'} = $self->_load_geocoder();
877 }
878  
879 185 return $self->{'_geocoder'};
880 }
881  
882 # _coordinates($result) -> ($lat, $lon)
883 #
884 # The position out of a geocoder result, read through the accessors of a
885 # Geo::Location::Point and the keys of a plain hashref from another geocoder.
886 # 'lon' is only a third fallback, as in HTML::Genealogy::Map: it is not part of
887 # the point interface, and only one of Geo::Coder::Free's MaxMind shortcuts ever
888 # spells longitude that way.
889 99 ▶sub _coordinates {
Subroutine _coordinates: called CC=9 SCAR=22.5
890 99 my $point = shift;
891  
892 99 ▶ return unless ref($point) && (ref($point) ne 'ARRAY');
Branch: 1 branch
BranchTrueFalseTotal
unless ref $point and ref $point ne "ARRAY" 32 67 99
Condition100%3/3
ref $point and ref $point ne "ARRAY"
!ll&&!rl&&r
31167
MC/DC100%2/2
ref $point and ref $point ne "ARRAY"
ref $point ref $point ne "ARRAY"
Truth table100%3/3
A and B
A= ref $pointB= ref $point ne "ARRAY"
ABresult
0X0
100
111
893  
894 67 ▶ if (blessed($point) && $point->can('latitude') && $point->can('longitude')) {
Branch: 1 branch
BranchTrueFalseTotal
if (builtin::blessed($point) and $point->can("latitude") and $point->can("longitude")) 45 22 67
Condition66%4/6
builtin::blessed($point) and $point->can("latitude") and $point->can("longitude")
!ll&&!rl&&r
22045
builtin::blessed($point) and $point->can("latitude")
!ll&&!rl&&r
22045
MC/DC33%1/3
builtin::blessed($point) and $point->can("latitude") and $point->can("longitude")
builtin::blessed($point) $point->can("latitude") $point->can("longitude")
Truth table50%2/4
A and B and C
A= builtin::blessed($point)B= $point->can("latitude")C= $point->can("longitude")
ABCresult
0XX0
10X0
1100
1111
895 45 return ($point->latitude(), $point->longitude());
896 }
897  
898 22 ▶ if (ref($point) eq 'HASH') {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $point eq "HASH") 22 0 22
899 22 ▶ return ($point->{'lat'}, $point->{'long'} // $point->{'lng'} // $point->{'lon'});
Condition83%5/6
($$point{"long"} // $$point{"lng"}) // $$point{"lon"}
l!l&&r!l&&!r
2110
$$point{"long"} // $$point{"lng"}
l!l&&r!l&&!r
7141
MC/DC0%0/3
($$point{"long"} // $$point{"lng"}) // $$point{"lon"}
$$point{"long"} $$point{"lng"} $$point{"lon"}
Truth table75%3/4
A // B // C
A= $$point{"long"}B= $$point{"lng"}C= $$point{"lon"}
ABCresult
1XX1
01X1
0011
0000
900 }
901  
902 0 return;
903 }
904  
905 # _backend_of($result) -> $name or undef
906 #
907 # Which database answered, as section 6.2 wants recorded and shown. 'database'
908 # names the backend and 'geocoder' the distribution; a result carrying neither is
909 # not guessed at. Geo::Coder::List puts the geocoder object that answered in
910 # 'geocoder', which is named by its class (Geo::Coder::OSM), and 'cache' when
911 # its own cache did, which names nothing.
912 66 ▶sub _backend_of {
Subroutine _backend_of: called CC=12 SCAR=24.9
913 66 my $point = shift;
914  
915 66 ▶ return unless ref($point) && (ref($point) ne 'ARRAY');
Branch: 1 branch
BranchTrueFalseTotal
unless ref $point and ref $point ne "ARRAY" 3 63 66
Condition100%3/3
ref $point and ref $point ne "ARRAY"
!ll&&!rl&&r
2163
MC/DC100%2/2
ref $point and ref $point ne "ARRAY"
ref $point ref $point ne "ARRAY"
Truth table100%3/3
A and B
A= ref $pointB= ref $point ne "ARRAY"
ABresult
0X0
100
111
916  
917 63 for my $key (qw(database geocoder)) {
918 78 ▶ my $value = (blessed($point) && $point->can($key)) ? $point->$key() : $point->{$key};
Branch: 1 branch
BranchTrueFalseTotal
builtin::blessed($point) && $point->can($key) ? : 41 37 78
Condition66%2/3
builtin::blessed($point) and $point->can($key)
!ll&&!rl&&r
37041
MC/DC50%1/2
builtin::blessed($point) and $point->can($key)
builtin::blessed($point) $point->can($key)
Truth table66%2/3
A and B
A= builtin::blessed($point)B= $point->can($key)
ABresult
0X0
100
111
919 78 ▶ $value = ref($value) if blessed($value);
Branch: 1 branch
BranchTrueFalseTotal
if builtin::blessed($value) 9 69 78
920 78 ▶ next if defined($value) && $value eq 'cache';
Branch: 1 branch
BranchTrueFalseTotal
if defined $value and $value eq "cache" 4 74 78
Condition100%3/3
defined $value and $value eq "cache"
!ll&&!rl&&r
16584
MC/DC100%2/2
defined $value and $value eq "cache"
defined $value $value eq "cache"
Truth table100%3/3
A and B
A= defined $valueB= $value eq "cache"
ABresult
0X0
100
111
921 74 ▶ return $value if defined($value) && length($value) && !ref($value);
Branch: 1 branch
BranchTrueFalseTotal
if defined $value and length $value and not ref $value 58 16 74
Condition66%4/6
defined $value and length $value and not ref $value
!ll&&!rl&&r
16058
defined $value and length $value
!ll&&!rl&&r
16058
MC/DC33%1/3
defined $value and length $value and not ref $value
defined $value length $value not ref $value
Truth table50%2/4
A and B and not(C)
A= defined $valueB= length $valueC= not ref $value
ABCresult
0XX0
10X0
1100
1111
922 }
923  
924 5 return;
925 }
926  
927 # _has_position($cache_entry)
928 #
929 # True for a remembered [lat, lon]; false for a remembered 0, which says "looked
930 # up and not found" and is a different thing from "never looked up".
931 31 ▶sub _has_position {
Subroutine _has_position: called CC=5 SCAR=16.1
932 31 my $entry = shift;
933  
934 31 ▶ return 0 unless (ref($entry) eq 'ARRAY') && (@{ $entry } >= 2);
Branch: 1 branch
BranchTrueFalseTotal
unless ref $entry eq "ARRAY" and @{$entry;} >= 2 14 17 31
Condition100%3/3
ref $entry eq "ARRAY" and @{$entry;} >= 2
!ll&&!rl&&r
13117
MC/DC100%2/2
ref $entry eq "ARRAY" and @{$entry;} >= 2
ref $entry eq "ARRAY" @{$entry;} >= 2
Truth table100%3/3
A and B
A= ref $entry eq "ARRAY"B= @{$entry;} >= 2
ABresult
0X0
100
111
935  
936 17 ▶ return (defined($entry->[0]) && defined($entry->[1])) ? 1 : 0;
Branch: 1 branch
BranchTrueFalseTotal
defined $$entry[0] && defined $$entry[1] ? : 15 2 17
Condition100%3/3
defined $$entry[0] and defined $$entry[1]
!ll&&!rl&&r
1115
MC/DC100%2/2
defined $$entry[0] and defined $$entry[1]
defined $$entry[0] defined $$entry[1]
Truth table100%3/3
A and B
A= defined $$entry[0]B= defined $$entry[1]
ABresult
0X0
100
111
937 }
938  
939 # _resolve($values, \%opts) -> \%result
940 #
941 # The whole of section 6.2 in one pass: classify every distinct value, apply the
942 # cap, make one batch, and count. No rows and no row-level message.
943 88 ▶sub _resolve {
Subroutine _resolve: called CC=40 SCAR=36.9
944 88 my ($self, $values, $opts) = @_;
945  
946 88 my $max = $opts->{'max'};
947 88 my $region = $opts->{'region'};
948  
949 88 my ($available, $reason) = $self->geocoder_available();
950 88 my $geocoder = $self->_geocoder();
951  
952 # A working copy: a caller's cache is never modified, so a request that dies
953 # half way through cannot leave a half-written one behind.
954 88 ▶ my %cache = %{ $opts->{'cache'} // {} };
Condition100%2/2
$$opts{"cache"} // {}
l!l
2068
MC/DC100%1/1
$$opts{"cache"} // {}
$$opts{"cache"}
Truth table100%2/2
A
A= $$opts{"cache"}
Aresult
11
00
955  
956 88 my %seen;
957 88 my (@skipped, @unresolved, @not_attempted, @pending);
958 88 my ($distinct, $cached_resolved, $cached_unresolved) = (0, 0, 0);
959  
960 # The positions found, whether they were found now or in an earlier request.
961 88 my %points;
962 my %backends;
963  
964 # 1. Which values are worth asking about at all.
965 88 ▶ for my $value (@{ $values // [] }) {
Condition50%1/2
$values // []
l!l
880
MC/DC0%0/1
$values // []
$values
Truth table50%1/2
A
A= $values
Aresult
11
00
966 603 ▶ my $key = (defined($value) && !ref($value)) ? $value : '';
Branch: 1 branch
BranchTrueFalseTotal
defined $value && !ref($value) ? : 587 16 603
Condition100%3/3
defined $value and not ref $value
!ll&&!rl&&r
124587
MC/DC100%2/2
defined $value and not ref $value
defined $value not ref $value
Truth table100%3/3
A and not(B)
A= defined $valueB= not ref $value
ABresult
0X0
100
111
967 603 ▶ next if $seen{$key}++;
Branch: 1 branch
BranchTrueFalseTotal
if $seen{$key}++ 438 165 603
968 165 $distinct++;
969  
970 # A cell with nothing in it is not a location under any reading, so it is
971 # refused whether or not a geocoder is installed.
972 165 ▶ if (!defined($value) || ref($value) || !length($key)) {
Branch: 1 branch
BranchTrueFalseTotal
if (not defined $value or ref $value or not length $key) 16 149 165
Condition83%5/6
not defined $value or ref $value or not length $key
l!l&&r!l&&!r
412149
not defined $value or ref $value
l!l&&r!l&&!r
40161
MC/DC66%2/3
not defined $value or ref $value or not length $key
not defined $value ref $value not length $key
Truth table75%3/4
not(A) or B or not(C)
A= not defined $valueB= ref $valueC= not length $key
ABCresult
1XX1
01X1
0011
0000
973 16 push @skipped, $value;
974 16 next;
975 }
976  
977 # Remembered answers are used even when there is no geocoder: what has
978 # been resolved before is still plottable.
979 149 ▶ if (exists $cache{$key}) {
Branch: 1 branch
BranchTrueFalseTotal
if (exists $cache{$key}) 24 125 149
980 24 ▶ if (_has_position($cache{$key})) {
Branch: 1 branch
BranchTrueFalseTotal
if (_has_position($cache{$key})) { } 14 10 24
981 14 $cached_resolved++;
982  
983 # A remembered place is a place to plot. The point is built here
984 # rather than only for the values looked up below, so that a
985 # request which asks the geocoder nothing - because it has
986 # nothing new to ask it - is not a request that plots nothing.
987 # No backend is claimed for it: the cache remembers a position,
988 # not which dataset gave it.
989 $points{$key} = {
990 'lat' => $cache{$key}[0],
991 14 'lon' => $cache{$key}[1],
992 };
993 } else {
994 10 $cached_unresolved++;
995 10 push @unresolved, $value;
996 }
997 24 next;
998 }
999  
1000 125 ▶ unless ($geocoder) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($geocoder) 9 116 125
1001 # No geocoder: the rule cannot be asked, so nothing is refused here
1002 # and nothing is looked up either.
1003 9 push @not_attempted, $value;
1004 9 next;
1005 }
1006  
1007 # The rule, asked of the module that would do the lookup. A value it
1008 # refuses never reaches a backend, so it cannot croak there.
1009 116 ▶ unless ($self->_judge($geocoder)->valid_location($key)) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($self->_judge($geocoder)->valid_location($key)) 15 101 116
1010 15 push @skipped, $value;
1011 15 next;
1012 }
1013  
1014 101 push @pending, $value;
1015 }
1016  
1017 # 2. The cap, counted in new lookups: map_geocode_max in section 6.2. Values
1018 # held back are left out of the cache, so the next request picks them up.
1019 88 my $wanted = scalar @pending;
1020 88 my @attempt = @pending;
1021 88 my $capped = 0;
1022  
1023 88 ▶ if ($wanted > $max) {
Branch: 1 branch
BranchTrueFalseTotal
if ($wanted > $max) 6 82 88
1024 6 @attempt = @pending[0 .. $max - 1];
1025 6 push @not_attempted, @pending[$max .. $#pending];
1026 6 $capped = 1;
1027 }
1028  
1029 # With lookup => 0 nothing is asked: the places that would have been are
1030 # handed back as 'pending', for a caller that looks them up one at a time
1031 # (the map page's progress bar), and are not_attempted meanwhile. The cap
1032 # still bounds the list; the rest are pending on the next request.
1033 88 my @wait;
1034  
1035 88 ▶ unless ($opts->{'lookup'} // 1) {
Branch: 1 branch
BranchTrueFalseTotal
unless ($$opts{"lookup"} // 1) 1 87 88
Condition100%2/2
$$opts{"lookup"} // 1
l!l
835
MC/DC100%1/1
$$opts{"lookup"} // 1
$$opts{"lookup"}
Truth table100%2/2
A
A= $$opts{"lookup"}
Aresult
11
00
1036 1 @wait = @attempt;
1037 1 unshift @not_attempted, @attempt;
1038 1 @attempt = ();
1039 1 $capped = 0;
1040 }
1041  
1042 # 3. The lookups, each place down a Geo::Coder::List on its own, so that its
1043 # cache answers what it has found before. With an online geocoder a place
1044 # that reaches Nominatim is counted against online_max, and a Nominatim that
1045 # does not answer is not taken to mean the place does not exist. Without
1046 # one the list holds the local data alone, asked with the region hint. An
1047 # injected geocoder is asked for one batch, with the non-croaking mode.
1048 88 my ($lookups, $resolved, $unresolved) = (0, 0, 0);
1049 88 my $lookup = { 'answers' => {}, 'held' => [], 'online' => 0 };
1050  
1051 88 ▶ if (@attempt) {
Branch: 1 branch
BranchTrueFalseTotal
if (@attempt) 61 27 88
1052 $lookup = $self->{'_online'} ? $self->_lookup_online($geocoder, \@attempt, $region)
1053 61 ▶ : $self->{'_local_chain'} ? $self->_lookup_local(\@attempt, $region)
Branch: 2 branches
BranchTrueFalseTotal
$$self{'_online'} ? : 14 47 61
$$self{'_local_chain'} ? : 7 40 47
1054 : _lookup_batch($geocoder, \@attempt, $region);
1055  
1056 60 $lookups = $lookup->{'lookups'};
1057  
1058 # A place held back was not answered either way: it is left out of the
1059 # cache, like a place past the cap, so the next request asks again.
1060 60 my %held = map { $_ => 1 } @{ $lookup->{'held'} };
1061 60 push @not_attempted, @{ $lookup->{'held'} };
1062 60 @attempt = grep { !$held{$_} } @attempt;
1063 60 ▶ $capped = 1 if $lookup->{'spent'} && @{ $lookup->{'held'} };
Branch: 1 branch
BranchTrueFalseTotal
if $$lookup{"spent"} and @{$$lookup{"held"};} 1 59 60
Condition100%3/3
$$lookup{"spent"} and @{$$lookup{"held"};}
!ll&&!rl&&r
5721
MC/DC100%2/2
$$lookup{"spent"} and @{$$lookup{"held"};}
$$lookup{"spent"} @{$$lookup{"held"};}
Truth table100%3/3
A and B
A= $$lookup{"spent"}B= @{$$lookup{"held"};}
ABresult
0X0
100
111
1064  
1065 60 for my $value (@attempt) {
1066 84 my $point = $lookup->{'answers'}{$value};
1067 84 my ($lat, $lon) = _coordinates($point);
1068  
1069 84 ▶ if (defined($lat) && defined($lon)) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $lat and defined $lon) { } 59 25 84
Condition66%2/3
defined $lat and defined $lon
!ll&&!rl&&r
25059
MC/DC50%1/2
defined $lat and defined $lon
defined $lat defined $lon
Truth table66%2/3
A and B
A= defined $latB= defined $lon
ABresult
0X0
100
111
1070 59 my $backend = _backend_of($point);
1071  
1072 # Nominatim answers in strings ("51.33"), the local data in
1073 # numbers; the cache and the map get numbers either way.
1074 59 ($lat, $lon) = ($lat + 0, $lon + 0);
1075  
1076 59 $cache{$value} = [ $lat, $lon ];
1077 59 ▶ $points{$value} = {
Branch: 1 branch
BranchTrueFalseTotal
defined $backend ? : 55 4 59
1078 'lat' => $lat,
1079 'lon' => $lon,
1080 (defined($backend) ? ('backend' => $backend) : ()),
1081 };
1082 59 ▶ $backends{$backend}++ if defined($backend);
Branch: 1 branch
BranchTrueFalseTotal
if defined $backend 55 4 59
1083 59 $resolved++;
1084  
1085 } else {
1086 # Looked up and not found: recorded as tried, so the next request
1087 # over this column does not ask again.
1088 25 $cache{$value} = 0;
1089 25 push @unresolved, $value;
1090 25 $unresolved++;
1091 }
1092 }
1093 }
1094  
1095 # 4. What to tell the user. Counts and keys, never a location value: a place
1096 # column in a personal dataset is personal data (section 7).
1097 87 my @messages;
1098  
1099 # geocoder_available() already returns the finished sentence, so it is used as
1100 # the text; formatting it again repeated the whole message.
1101 push @messages, {
1102 'key' => $self->{'_unavailable_key'} // 'error_map_geocoder_missing',
1103 87 ▶ 'text' => defined($reason) ? $reason : $self->_msg('error_map_geocoder_missing', $self->{'min_version'}, 'it is not installed'),
Branch: 2 branches
BranchTrueFalseTotal
unless $available 5 82 87
defined $reason ? : 5 0 5
Condition100%2/2
$$self{"_unavailable_key"} // "error_map_geocoder_missing"
l!l
14
MC/DC100%1/1
$$self{"_unavailable_key"} // "error_map_geocoder_missing"
$$self{"_unavailable_key"}
Truth table100%2/2
A
A= $$self{"_unavailable_key"}
Aresult
11
00
1104 } unless $available;
1105  
1106 87 ▶ push @messages, $self->_message('warn_map_geocode_capped', scalar @attempt, $wanted)
Branch: 1 branch
BranchTrueFalseTotal
if $capped 7 80 87
1107 if $capped;
1108  
1109 1 push @messages, $self->_message('warn_map_geocoder_online_failed', $lookup->{'failed'}, scalar @{ $lookup->{'held'} })
1110 87 ▶ if defined($lookup->{'failed'}) && @{ $lookup->{'held'} };
Branch: 1 branch
BranchTrueFalseTotal
if defined $$lookup{"failed"} and @{$$lookup{"held"};} 1 86 87
Condition66%2/3
defined $$lookup{"failed"} and @{$$lookup{"held"};}
!ll&&!rl&&r
8601
MC/DC50%1/2
defined $$lookup{"failed"} and @{$$lookup{"held"};}
defined $$lookup{"failed"} @{$$lookup{"held"};}
Truth table66%2/3
A and B
A= defined $$lookup{"failed"}B= @{$$lookup{"held"};}
ABresult
0X0
100
111
1111  
1112 # Coarse answers are what the local data gives on its own; once Nominatim is
1113 # asked about the places it does not know, the warning is no longer true.
1114 push @messages, $self->_message('warn_map_geocoder_unavailable_data')
1115 87 ▶ if @attempt && !$self->{'openaddr'} && !$self->{'_online'};
Branch: 1 branch
BranchTrueFalseTotal
if @attempt and not $$self{"openaddr"} and not $$self{"_online"} 27 60 87
Condition100%6/6
@attempt and not $$self{"openaddr"} and not $$self{"_online"}
!ll&&!rl&&r
461427
@attempt and not $$self{"openaddr"}
!ll&&!rl&&r
271941
MC/DC100%3/3
@attempt and not $$self{"openaddr"} and not $$self{"_online"}
@attempt not $$self{"openaddr"} not $$self{"_online"}
Truth table100%4/4
A and not(B) and not(C)
A= @attemptB= not $$self{"openaddr"}C= not $$self{"_online"}
ABCresult
0XX0
10X0
1100
1111
1116  
1117 push @messages, $self->_message('warn_map_geocoder_local_only')
1118 87 ▶ if $unresolved && $self->{'_online_missing'};
Branch: 1 branch
BranchTrueFalseTotal
if $unresolved and $$self{"_online_missing"} 1 86 87
Condition100%3/3
$unresolved and $$self{"_online_missing"}
!ll&&!rl&&r
67191
MC/DC100%2/2
$unresolved and $$self{"_online_missing"}
$unresolved $$self{"_online_missing"}
Truth table100%3/3
A and B
A= $unresolvedB= $$self{"_online_missing"}
ABresult
0X0
100
111
1119  
1120 87 my @names = sort keys %backends;
1121  
1122 return {
1123 'available' => $available ? 1 : 0,
1124 'points' => \%points,
1125 'counts' => {
1126 'distinct' => $distinct,
1127 'skipped' => scalar @skipped,
1128 'cached' => $cached_resolved + $cached_unresolved,
1129 'attempted' => scalar @attempt,
1130 'lookups' => $lookups,
1131 87 ▶ 'online' => $lookup->{'online'},
Branch: 2 branches
BranchTrueFalseTotal
$available ? : 82 5 87
@names ? : 43 44 87
1132 'resolved' => $cached_resolved + $resolved,
1133 'unresolved' => $cached_unresolved + $unresolved,
1134 'not_attempted' => scalar @not_attempted,
1135 },
1136 'skipped' => \@skipped,
1137 'unresolved' => \@unresolved,
1138 'not_attempted' => \@not_attempted,
1139 'pending' => \@wait,
1140 'messages' => \@messages,
1141 'cache' => \%cache,
1142 'backends' => \%backends,
1143 'backend' => @names ? join(', ', @names) : undef,
1144 'capped' => $capped,
1145 };
1146 }
1147  
1148 # _lookup_batch($geocoder, \@values, $region) -> \%lookup
1149 #
1150 # One geocode_many() call, for a geocoder injected in place of the chain.
1151 # The count the geocoder reports for the batch is the one a caller caps its
1152 # work by; a geocoder that reports none leaves the number of values it was
1153 # handed.
1154 43 ▶sub _lookup_batch {
Subroutine _lookup_batch: called CC=5 SCAR=16.1
1155 43 my ($geocoder, $values, $region) = @_;
1156  
1157 43 ▶ my $batch = $geocoder->geocode_many(
Branch: 1 branch
BranchTrueFalseTotal
defined $region ? : 3 40 43
1158 $values,
1159 (defined($region) ? (region => $region) : ()),
1160 on_error => 'undef',
1161 );
1162  
1163 return {
1164 'answers' => (ref($batch) eq 'HASH') ? $batch : {},
1165 'lookups' => ((ref($batch) eq 'HASH') && defined($batch->{'lookups'}))
1166 ? $batch->{'lookups'}
1167 42 ▶ : scalar @{ $values },
Branch: 2 branches
BranchTrueFalseTotal
ref $batch eq 'HASH' ? : 40 2 42
ref $batch eq 'HASH' && defined $$batch{'lookups'} ? : 40 2 42
Condition66%2/3
ref $batch eq "HASH" and defined $$batch{"lookups"}
!ll&&!rl&&r
2040
MC/DC50%1/2
ref $batch eq "HASH" and defined $$batch{"lookups"}
ref $batch eq "HASH" defined $$batch{"lookups"}
Truth table66%2/3
A and B
A= ref $batch eq "HASH"B= defined $$batch{"lookups"}
ABresult
0X0
100
111
1168 'held' => [],
1169 'online' => 0,
1170 };
1171 }
1172  
1173 # _lookup_local(\@values, $region) -> \%lookup
1174 #
1175 # Each place down the Geo::Coder::List that holds the local data alone, with
1176 # the region hint. Geo::Coder::Free's own geocode_many() asks geocode() once a
1177 # place as well, so going one at a time costs nothing, and the list's cache
1178 # answers what it has found before, here or online. A lookup answered by that
1179 # cache is not counted as one. The list catches what the local geocoder throws
1180 # and logs it; a place whose lookup threw and found nothing is held, not
1181 # remembered as not found, as on the online path.
1182 15 ▶sub _lookup_local {
Subroutine _lookup_local: called CC=8 SCAR=20.8
1183 15 my ($self, $values, $region) = @_;
1184  
1185 15 my $chain = $self->{'_local_chain'};
1186 15 my (%answers, @held);
1187 15 my $lookups = 0;
1188  
1189 15 for my $value (@{ $values }) {
1190 21 $chain->flush();
1191 21 ▶ my $point = $chain->geocode(location => $value, (defined($region) ? (region => $region) : ()));
Branch: 1 branch
BranchTrueFalseTotal
defined $region ? : 5 16 21
1192 21 ▶ my @log = @{ $chain->log() // [] };
Condition50%1/2
$chain->log // []
l!l
210
MC/DC0%0/1
$chain->log // []
$chain->log
Truth table50%1/2
A
A= $chain->log
Aresult
11
00
1193  
1194 21 ▶ $lookups++ unless grep { ($_->{'geocoder'} // '') eq 'cache' } @log;
Branch: 1 branch
BranchTrueFalseTotal
unless grep {($$_{"geocoder"} // "") eq "cache";} @log 17 4 21
Condition50%1/2
$$_{"geocoder"} // ""
l!l
230
MC/DC0%0/1
$$_{"geocoder"} // ""
$$_{"geocoder"}
Truth table50%1/2
A
A= $$_{"geocoder"}
Aresult
11
00
1195  
1196 21 ▶ if (!$point && grep { defined $_->{'error'} } @log) {
Branch: 1 branch
BranchTrueFalseTotal
if (not $point and grep {defined $$_{"error"};} @log) 2 19 21
Condition100%3/3
not $point and grep {defined $$_{"error"};} @log
!ll&&!rl&&r
1182
MC/DC100%2/2
not $point and grep {defined $$_{"error"};} @log
not $point grep {defined $$_{"error"};} @log
Truth table100%3/3
not(A) and B
A= not $pointB= grep {defined $$_{"error"};} @log
ABresult
0X0
100
111
1197 2 push @held, $value;
1198 2 next;
1199 }
1200  
1201 19 $answers{$value} = $point;
1202 }
1203  
1204 return {
1205 15 'answers' => \%answers,
1206 'lookups' => $lookups,
1207 'held' => \@held,
1208 'online' => 0,
1209 };
1210 }
1211  
1212 # _lookup_online($chain, \@values, $region) -> \%lookup
1213 #
1214 # Each place down the Geo::Coder::List chain: the local data, then Nominatim.
1215 # The chain's log says which geocoders a place reached, which is how the
1216 # lookups that went online are counted, and how a Nominatim that did not answer
1217 # is told apart from a place it does not know: Geo::Coder::OSM answers nothing
1218 # in both cases, and only its last response says which.
1219 #
1220 # Once online_max places have gone online, or Nominatim has failed once, the
1221 # rest are asked of the local data alone, through _lookup_local. A place it
1222 # does not know is then held rather than remembered as not found, because Nominatim
1223 # was never asked.
1224 #
1225 # Geo::Coder::List catches whatever a geocoder in it throws, logs it and asks
1226 # the next one, so on this path a local geocoder that throws does not end the
1227 # request as it does on the batch path. A place whose lookup threw somewhere
1228 # and found nothing is held too: "not found" is remembered for good, and an
1229 # error is not an answer. Nothing here adds an eval of its own.
1230 18 ▶sub _lookup_online {
Subroutine _lookup_online: called CC=26 SCAR=32.7
1231 18 my ($self, $chain, $values, $region) = @_;
1232  
1233 18 my $online = $self->{'_online'};
1234 18 my $class = ref($online);
1235 18 my $budget = $self->{'online_max'};
1236  
1237 18 my (%answers, @held, @rest);
1238 18 my ($lookups, $asked, $failed) = (0, 0, undef);
1239  
1240 18 for my $index (0 .. $#{ $values }) {
1241 28 ▶ if (defined($failed) || (defined($budget) && ($asked >= $budget))) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $failed or defined $budget and $asked >= $budget) 6 22 28
Condition100%6/6
defined $failed or defined $budget and $asked >= $budget
l!l&&r!l&&!r
2422
defined $budget and $asked >= $budget
!ll&&!rl&&r
1574
MC/DC100%3/3
defined $failed or defined $budget and $asked >= $budget
defined $failed defined $budget $asked >= $budget
Truth table100%4/4
A or B and C
A= defined $failedB= defined $budgetC= $asked >= $budget
ABCresult
1XX1
0111
00X0
0100
1242 6 @rest = @{ $values }[$index .. $#{ $values }];
1243 6 last;
1244 }
1245  
1246 22 my $value = $values->[$index];
1247  
1248 22 $chain->flush();
1249 22 my $point = $chain->geocode(location => $value);
1250 22 ▶ my @log = @{ $chain->log() // [] };
Condition50%1/2
$chain->log // []
l!l
220
MC/DC0%0/1
$chain->log // []
$chain->log
Truth table50%1/2
A
A= $chain->log
Aresult
11
00
1251  
1252 22 ▶ $lookups++ unless grep { ($_->{'geocoder'} // '') eq 'cache' } @log;
Branch: 1 branch
BranchTrueFalseTotal
unless grep {($$_{"geocoder"} // "") eq "cache";} @log 19 3 22
Condition50%1/2
$$_{"geocoder"} // ""
l!l
380
MC/DC0%0/1
$$_{"geocoder"} // ""
$$_{"geocoder"}
Truth table50%1/2
A
A= $$_{"geocoder"}
Aresult
11
00
1253  
1254 22 ▶ my @went = grep { ($_->{'geocoder'} // '') eq $class } @log;
Condition50%1/2
$$_{"geocoder"} // ""
l!l
380
MC/DC0%0/1
$$_{"geocoder"} // ""
$$_{"geocoder"}
Truth table50%1/2
A
A= $$_{"geocoder"}
Aresult
11
00
1255 22 ▶ if (@went) {
Branch: 1 branch
BranchTrueFalseTotal
if (@went) 17 5 22
1256 17 $asked++;
1257  
1258 17 my $response = $online->response();
1259 17 my ($thrown) = grep { defined } map { $_->{'error'} } @went;
1260  
1261 17 ▶ if (!$point && (defined($thrown) || !$response || !$response->is_success())) {
Branch: 1 branch
BranchTrueFalseTotal
if (not $point and defined $thrown || !$response || !$response->is_success) 4 13 17
Condition100%9/9
not $point and defined $thrown || !$response || !$response->is_success
!ll&&!rl&&r
1034
defined $thrown or not $response or not $response->is_success
l!l&&r!l&&!r
223
defined $thrown or not $response
l!l&&r!l&&!r
115
MC/DC100%4/4
not $point and defined $thrown || !$response || !$response->is_success
not $point defined $thrown not $response not $response->is_success
Truth table100%5/5
not(A) and B or not(C) or not(D)
A= not $pointB= defined $thrownC= not $responseD= not $response->is_success
ABCDresult
0XXX0
10000
11XX1
101X1
10011
1262 # What the page is told is the status line, never the error
1263 # text: an exception may quote the request, and the request
1264 # holds the place, which is the reader's data (section 7).
1265 4 ▶ $failed = defined($thrown) ? 'an error in Geo::Coder::OSM'
Branch: 2 branches
BranchTrueFalseTotal
defined $thrown ? : 1 3 4
$response ? : 2 1 3
1266 : $response ? $response->status_line()
1267 : 'no response';
1268 4 push @held, $value;
1269 4 next;
1270 }
1271 }
1272  
1273 18 ▶ if (!$point && grep { defined $_->{'error'} && ($_->{'geocoder'} // '') ne $class } @log) {
Branch: 1 branch
BranchTrueFalseTotal
if (not $point and grep {defined $$_{"error"} and ($$_{"geocoder"} // "") ne $class;} @log) 1 17 18
Condition75%6/8
not $point and grep {defined $$_{"error"} and ($$_{"geocoder"} // "") ne $class;} @log
!ll&&!rl&&r
1521
defined $$_{'error'} && ($$_{'geocoder'} // '') ne $class
!ll&&!rl&&r
501
$$_{"geocoder"} // ""
l!l
10
MC/DC100%2/2
not $point and grep {defined $$_{"error"} and ($$_{"geocoder"} // "") ne $class;} @log
not $point grep {defined $$_{"error"} and ($$_{"geocoder"} // "") ne $class;} @log
MC/DC50%1/2
defined $$_{'error'} && ($$_{'geocoder'} // '') ne $class
defined $$_{'error'} $$_{"geocoder"}
Truth table100%3/3
not(A) and B
A= not $pointB= grep {defined $$_{"error"} and ($$_{"geocoder"} // "") ne $class;} @log
ABresult
0X0
100
111
Truth table66%2/3
A && B
A= defined $$_{'error'}B= $$_{"geocoder"}
ABresult
0X0
100
111
1274 1 push @held, $value;
1275 1 next;
1276 }
1277  
1278 17 $answers{$value} = $point;
1279 }
1280  
1281 18 ▶ if (@rest) {
Branch: 1 branch
BranchTrueFalseTotal
if (@rest) 6 12 18
1282 6 my $local = $self->_lookup_local(\@rest, $region);
1283 6 $lookups += $local->{'lookups'};
1284 6 push @held, @{ $local->{'held'} };
1285  
1286 0 my %thrown = map { $_ => 1 } @{ $local->{'held'} };
1287 6 for my $value (grep { !$thrown{$_} } @rest) {
1288 8 my ($lat, $lon) = _coordinates($local->{'answers'}{$value});
1289  
1290 8 ▶ if (defined($lat) && defined($lon)) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $lat and defined $lon) { } 4 4 8
Condition66%2/3
defined $lat and defined $lon
!ll&&!rl&&r
404
MC/DC50%1/2
defined $lat and defined $lon
defined $lat defined $lon
Truth table66%2/3
A and B
A= defined $latB= defined $lon
ABresult
0X0
100
111
1291 4 $answers{$value} = $local->{'answers'}{$value};
1292 } else {
1293 4 push @held, $value;
1294 }
1295 }
1296 }
1297  
1298 return {
1299 18 ▶ 'answers' => \%answers,
Branch: 1 branch
BranchTrueFalseTotal
!defined($failed) && @rest ? : 4 14 18
Condition100%3/3
not defined $failed and @rest
!ll&&!rl&&r
4104
MC/DC100%2/2
not defined $failed and @rest
not defined $failed @rest
Truth table100%3/3
not(A) and B
A= not defined $failedB= @rest
ABresult
0X0
100
111
1300 'lookups' => $lookups,
1301 'held' => \@held,
1302 'online' => $asked,
1303 'spent' => (!defined($failed) && @rest) ? 1 : 0,
1304 'failed' => $failed,
1305 };
1306 }
1307  
1308 # The options resolve_values() and resolve_column() share, validated once.
1309 93 ▶sub _options {
Subroutine _options: called CC=5 SCAR=16.1
1310 93 my ($self, $opts) = @_;
1311  
1312 93 my $args = validate_strict(
1313 schema => {
1314 cache => { type => 'hashref', optional => 1, default => undef },
1315 region => { type => 'string', optional => 1, default => undef },
1316 max => { type => 'scalar', optional => 1, default => undef },
1317 lookup => { type => 'scalar', optional => 1, default => 1 },
1318 },
1319 stack_trace => 1,
1320 input => $opts,
1321 );
1322  
1323 92 my $max = $DEFAULT_MAX;
1324  
1325 92 ▶ if (defined($args->{'max'})) {
Branch: 1 branch
BranchTrueFalseTotal
if (defined $$args{"max"}) 23 69 92
1326 # A cap that is not a positive integer is a caller bug, not a location
1327 # that could not be resolved: it is refused rather than defaulted,
1328 # because defaulting would silently cap a request the caller meant to
1329 # raise, or raise one they meant to cap.
1330 croak(_fmt('error_max_invalid', $args->{'max'}))
1331 23 ▶ unless ($args->{'max'} =~ /\A\d+\z/) && ($args->{'max'} > 0);
Branch: 1 branch
BranchTrueFalseTotal
unless $$args{"max"} =~ /\A\d+\z/ and $$args{"max"} > 0 7 16 23
Condition100%3/3
$$args{"max"} =~ /\A\d+\z/ and $$args{"max"} > 0
!ll&&!rl&&r
5216
MC/DC100%2/2
$$args{"max"} =~ /\A\d+\z/ and $$args{"max"} > 0
$$args{"max"} =~ /\A\d+\z/ $$args{"max"} > 0
Truth table100%3/3
A and B
A= $$args{"max"} =~ /\A\d+\z/B= $$args{"max"} > 0
ABresult
0X0
100
111
1332  
1333 16 $max = $args->{'max'};
1334 }
1335  
1336 return {
1337 'cache' => $args->{'cache'},
1338 'region' => $args->{'region'},
1339 'max' => $max,
1340 85 ▶ 'lookup' => $args->{'lookup'} ? 1 : 0,
Branch: 1 branch
BranchTrueFalseTotal
$$args{'lookup'} ? : 84 1 85
1341 };
1342 }
1343  
1344 =head2 resolve_values
1345  
1346 my $result = $geocode->resolve_values(\@values, %options);
1347 my $result = $geocode->resolve_values(\@values, \%options);
1348  
1349 Resolve a flat list of cell values, without the context of any row structure.
1350 Use this when you want to warm the cache for a column's distinct values, or
1351 when you have already extracted the values yourself and do not need per-marker
1352 row counts.
1353  
1354 The returned hashref is identical to L</resolve_column>'s, except that
1355 C<counts.rows>, C<counts.plotted>, C<counts.unusable>, and C<markers> are
1356 absent, and C<warn_map_rows_skipped> is never emitted.
1357  
1358 =head3 API SPECIFICATION
1359  
1360 =head4 INPUT
1361  
1362 \@values => arrayref, required; elements may be any scalar or undef
1363 cache => hashref, # optional; { value => [lat, lon] | 0 }
1364 region => string, # optional; two-letter country hint
1365 max => positive integer, # optional; new lookups allowed; default 500
1366 lookup => 0 | 1, # optional; 0 asks nothing (see resolve_column); default 1
1367  
1368 Duplicate values in C<\@values> cost one lookup each. C<undef>, references,
1369 empty strings, and pure-digit strings are refused before any lookup and counted
1370 as C<skipped>.
1371  
1372 =head4 OUTPUT
1373  
1374 Returns a hashref (same shape as L</resolve_column> but without C<markers>,
1375 C<counts.rows>, C<counts.plotted>, and C<counts.unusable>):
1376  
1377 {
1378 available => 1 | 0,
1379 points => { $value => { lat => $n, lon => $n, backend? => $s } },
1380 counts => {
1381 distinct => $n,
1382 skipped => $n,
1383 cached => $n,
1384 attempted => $n,
1385 lookups => $n,
1386 resolved => $n,
1387 unresolved => $n,
1388 not_attempted => $n,
1389 },
1390 skipped => [ $value, ... ],
1391 unresolved => [ $value, ... ],
1392 not_attempted => [ $value, ... ],
1393 pending => [ $value, ... ],
1394 messages => [ { key => $k, text => $t }, ... ],
1395 cache => { $value => [lat, lon] | 0 },
1396 backends => { $name => $count },
1397 backend => $name | undef,
1398 capped => 0 | 1,
1399 }
1400  
1401 Croaks with C<error_values_arrayref> if C<\@values> is not an arrayref.
1402  
1403 =head3 EXAMPLE
1404  
1405 # Build the distinct-value list yourself and warm the cache offline.
1406 my %seen;
1407 my @cities = grep { defined($_) && !$seen{$_}++ } map { $_->{'city'} } @{ $rows };
1408  
1409 my $result = $geocode->resolve_values(\@cities,
1410 cache => $cache,
1411 region => 'GB',
1412 max => 2000,
1413 );
1414  
1415 printf "New lookups: %d Resolved: %d Unresolved: %d\n",
1416 $result->{'counts'}{'attempted'},
1417 $result->{'counts'}{'resolved'},
1418 $result->{'counts'}{'unresolved'};
1419  
1420 =cut
1421  
1422 47 ▶sub resolve_values {
Subroutine resolve_values: called CC=5 SCAR=16.1
Pod documented
1423 47 my $self = shift;
1424 47 my $values = shift;
1425  
1426 47 ▶ croak(_fmt('error_values_arrayref', defined($values) ? ref($values) || $values : 'undef'))
Branch: 2 branches
BranchTrueFalseTotal
unless ref $values eq "ARRAY" 4 43 47
defined $values ? : 3 1 4
Condition66%2/3
ref $values or $values
l!l&&r!l&&!r
120
MC/DC0%0/2
ref $values or $values
ref $values $values
Truth table66%2/3
A or B
A= ref $valuesB= $values
ABresult
1X1
011
000
1427 unless ref($values) eq 'ARRAY';
1428  
1429 43 ▶ my $raw = Params::Get::get_params(undef, \@_) // {};
Condition100%2/2
Params::Get::get_params(undef, \@_) // {}
l!l
1033
MC/DC100%1/1
Params::Get::get_params(undef, \@_) // {}
Params::Get::get_params(undef, \@_)
Truth table100%2/2
A
A= Params::Get::get_params(undef, \@_)
Aresult
11
00
1430  
1431 43 return $self->_resolve($values, $self->_options($raw));
1432 }
1433  
1434 =head2 resolve_column
1435  
1436 See full documentation above, in L</SUBROUTINES/METHODS>.
1437  
1438 =cut
1439  
1440 52 ▶sub resolve_column {
Subroutine resolve_column: called CC=25 SCAR=32.2
Pod documented
1441 52 my $self = shift;
1442 52 my $rows = shift;
1443 52 my $column = shift;
1444  
1445 52 ▶ croak(_fmt('error_rows_arrayref', defined($rows) ? ref($rows) || $rows : 'undef'))
Branch: 2 branches
BranchTrueFalseTotal
unless ref $rows eq "ARRAY" 3 49 52
defined $rows ? : 2 1 3
Condition33%1/3
ref $rows or $rows
l!l&&r!l&&!r
020
MC/DC0%0/2
ref $rows or $rows
ref $rows $rows
Truth table33%1/3
A or B
A= ref $rowsB= $rows
ABresult
1X1
011
000
1446 unless ref($rows) eq 'ARRAY';
1447  
1448 49 ▶ croak(_fmt('error_column_required')) unless defined($column) && length($column);
Branch: 1 branch
BranchTrueFalseTotal
unless defined $column and length $column 3 46 49
Condition100%3/3
defined $column and length $column
!ll&&!rl&&r
2146
MC/DC100%2/2
defined $column and length $column
defined $column length $column
Truth table100%3/3
A and B
A= defined $columnB= length $column
ABresult
0X0
100
111
1449  
1450 46 ▶ my $raw = Params::Get::get_params(undef, \@_) // {};
Condition100%2/2
Params::Get::get_params(undef, \@_) // {}
l!l
1927
MC/DC100%1/1
Params::Get::get_params(undef, \@_) // {}
Params::Get::get_params(undef, \@_)
Truth table100%2/2
A
A= Params::Get::get_params(undef, \@_)
Aresult
11
00
1451  
1452 # The cells in row order, and whether the column is there at all. A row that
1453 # is not a hashref has no cell, which is one of the things that gets skipped
1454 # rather than a reason to stop.
1455 46 my @cells;
1456 46 my $found = 0;
1457  
1458 46 for my $row (@{ $rows }) {
1459 539 ▶ if (ref($row) eq 'HASH') {
Branch: 1 branch
BranchTrueFalseTotal
if (ref $row eq "HASH") 534 5 539
1460 534 ▶ $found = 1 if exists $row->{$column};
Branch: 1 branch
BranchTrueFalseTotal
if exists $row->{$column} 531 3 534
1461 534 push @cells, $row->{$column};
1462 534 next;
1463 }
1464  
1465 5 push @cells, undef;
1466 }
1467  
1468 46 ▶ croak(_fmt('error_map_loc_column', $column)) if @{ $rows } && !$found;
Branch: 1 branch
BranchTrueFalseTotal
if @{$rows;} and not $found 3 43 46
Condition100%3/3
@{$rows;} and not $found
!ll&&!rl&&r
4393
MC/DC100%2/2
@{$rows;} and not $found
@{$rows;} not $found
Truth table100%3/3
A and not(B)
A= @{$rows;}B= not $found
ABresult
0X0
100
111
1469  
1470 43 my $result = $self->_resolve(\@cells, $self->_options($raw));
1471  
1472 # How many rows hold each distinct value, and where the first of them is: what
1473 # an aggregated marker's popup and its draw order need.
1474 39 my (%row_count, %first_row);
1475  
1476 39 for my $index (0 .. $#cells) {
1477 530 ▶ my $key = (defined($cells[$index]) && !ref($cells[$index])) ? $cells[$index] : '';
Branch: 1 branch
BranchTrueFalseTotal
defined $cells[$index] && !ref($cells[$index]) ? : 516 14 530
Condition100%3/3
defined $cells[$index] and not ref $cells[$index]
!ll&&!rl&&r
104516
MC/DC100%2/2
defined $cells[$index] and not ref $cells[$index]
defined $cells[$index] not ref $cells[$index]
Truth table100%3/3
A and not(B)
A= defined $cells[$index]B= not ref $cells[$index]
ABresult
0X0
100
111
1478  
1479 530 $row_count{$key}++;
1480 530 ▶ $first_row{$key} = $index unless exists $first_row{$key};
Branch: 1 branch
BranchTrueFalseTotal
unless exists $first_row{$key} 96 434 530
1481 }
1482  
1483 39 my $plotted = 0;
1484  
1485 39 for my $key (keys %{ $result->{'points'} }) {
1486 35 $result->{'points'}{$key}{'rows'} = $row_count{$key};
1487 35 $result->{'points'}{$key}{'first_row'} = $first_row{$key};
1488  
1489 35 $plotted += $row_count{$key};
1490 }
1491  
1492 $result->{'markers'} = [
1493 35 map { { 'value' => $_, %{ $result->{'points'}{$_} } } }
1494 39 sort { $first_row{$a} <=> $first_row{$b} } keys %{ $result->{'points'} }
1495 ];
1496  
1497 39 my $rows_total = scalar @cells;
1498  
1499 39 $result->{'counts'}{'rows'} = $rows_total;
1500 39 $result->{'counts'}{'plotted'} = $plotted;
1501 39 $result->{'counts'}{'unusable'} = $rows_total - $plotted;
1502  
1503 39 ▶ if ($plotted < $rows_total) {
Branch: 1 branch
BranchTrueFalseTotal
if ($plotted < $rows_total) 28 11 39
1504 # Why the rest were not plotted, counted in rows, naming only the reasons
1505 # that apply: a list of every possible reason sent a reader looking for
1506 # empty cells that were not there.
1507 28 my %why;
1508 28 for my $value (@{ $result->{'skipped'} }) {
1509 25 ▶ my $key = (defined($value) && !ref($value)) ? $value : '';
Branch: 1 branch
BranchTrueFalseTotal
defined $value && !ref($value) ? : 22 3 25
Condition66%2/3
defined $value and not ref $value
!ll&&!rl&&r
3022
MC/DC50%1/2
defined $value and not ref $value
defined $value not ref $value
Truth table66%2/3
A and not(B)
A= defined $valueB= not ref $value
ABresult
0X0
100
111
1510 25 ▶ $why{$key} = length($key) ? 'refused' : 'empty';
Branch: 1 branch
BranchTrueFalseTotal
length $key ? : 12 13 25
1511 }
1512 28 $why{$_} = 'not_found' for grep { defined } @{ $result->{'unresolved'} };
1513 28 $why{$_} = 'later' for grep { defined } @{ $result->{'not_attempted'} };
1514  
1515 28 my %rows_by;
1516 28 $rows_by{ $why{$_} } += $row_count{$_} for grep { exists $row_count{$_} } keys %why;
1517  
1518 43 my $reasons = join(', ', map { $self->_msg("warn_map_rows_why_$_", $rows_by{$_}) }
1519 28 grep { $rows_by{$_} } qw(empty refused not_found later));
1520  
1521 28 push @{ $result->{'messages'} }, $self->_message('warn_map_rows_skipped',
1522 $plotted, $rows_total, $rows_total - $plotted, $reasons);
1523 }
1524  
1525 39 return $result;
1526 }
1527  
1528 =head2 geocoder_available
1529  
1530 my ($is_available, $reason) = $geocode->geocoder_available();
1531  
1532 Ask whether this object can make new geocoder lookups. Returns a list of two
1533 values: a boolean (C<1> or C<0>) and a human-readable reason string when the
1534 answer is no.
1535  
1536 Loads C<Geo::Coder::Free> lazily on the first call if it has not been loaded
1537 yet. Never croaks: a panel uses this to decide whether to show the location
1538 option, so the answer must always be a string, not an exception.
1539  
1540 =head3 API SPECIFICATION
1541  
1542 =head4 INPUT
1543  
1544 None. (No arguments accepted.)
1545  
1546 =head4 OUTPUT
1547  
1548 ($available, $reason)
1549  
1550 $available => 1 # geocoder is ready
1551 $available => 0 # geocoder is not available
1552 $reason => undef # when $available is 1
1553 $reason => string # error_map_geocoder_missing text; when $available is 0
1554  
1555 =head3 EXAMPLE
1556  
1557 my ($ok, $reason) = $geocode->geocoder_available();
1558  
1559 if (!$ok) {
1560 # Disable the "Map by location column" panel option.
1561 $stash->{'map_loc_disabled'} = 1;
1562 $stash->{'map_loc_disabled_reason'} = $reason;
1563 }
1564  
1565 =cut
1566  
1567 96 ▶sub geocoder_available {
Subroutine geocoder_available: called CC=5 SCAR=16.1
Pod documented
1568 96 my $self = shift;
1569  
1570 96 ▶ return (1, undef) if $self->_geocoder();
Branch: 1 branch
BranchTrueFalseTotal
if $self->_geocoder 87 9 96
1571  
1572 return (0, $self->_msg('error_map_geocoder_list_missing', $self->{'_unavailable'}))
1573 9 ▶ if ($self->{'_unavailable_key'} // '') eq 'error_map_geocoder_list_missing';
Branch: 1 branch
BranchTrueFalseTotal
if ($$self{"_unavailable_key"} // "") eq "error_map_geocoder_list_missing" 2 7 9
Condition100%2/2
$$self{"_unavailable_key"} // ""
l!l
27
MC/DC100%1/1
$$self{"_unavailable_key"} // ""
$$self{"_unavailable_key"}
Truth table100%2/2
A
A= $$self{"_unavailable_key"}
Aresult
11
00
1574  
1575 return (0, $self->_msg('error_map_geocoder_missing',
1576 $self->{'min_version'},
1577 7 ▶ $self->{'_unavailable'} // 'it is not installed'));
Condition50%1/2
$$self{"_unavailable"} // "it is not installed"
l!l
70
MC/DC0%0/1
$$self{"_unavailable"} // "it is not installed"
$$self{"_unavailable"}
Truth table50%1/2
A
A= $$self{"_unavailable"}
Aresult
11
00
1578 }
1579  
1580 =head2 file_cache
1581  
1582 my $cache = Database::BI::Model::Geocode->file_cache($model_dir);
1583 my $geocode = Database::BI::Model::Geocode->new(cache => $cache);
1584  
1585 The cache Geo::Coder::List keeps its answers in, as files under
1586 F<$model_dir/geocode-cache>: a place Nominatim found is not asked again for a
1587 month (Geo::Coder::List's C<cache_hit_duration>), across requests, restarts and
1588 every hypnotoad worker. A copy of each answer is also kept in the process's
1589 memory. The directory is made when the first answer is stored.
1590  
1591 This is a B<class method>. The map page and C<script/bi-geocode> both call it
1592 with the directory that holds F<models.json>, so they share one cache. It holds
1593 place names, as F<models.json> does, so it belongs in C<model_dir> and never in
1594 C<data_dir>.
1595  
1596 =head3 API SPECIFICATION
1597  
1598 =head4 INPUT
1599  
1600 $model_dir => string, required; an existing directory
1601  
1602 =head4 OUTPUT
1603  
1604 A L<CHI> object (the C<File> driver), for the C<cache> option of C<new()>.
1605 Croaks with C<error_cache_dir> when C<$model_dir> is not a directory.
1606  
1607 =head3 EXAMPLE
1608  
1609 my $geocode = Database::BI::Model::Geocode->new(
1610 cache => Database::BI::Model::Geocode->file_cache('.bi'),
1611 );
1612  
1613 =cut
1614  
1615 9 ▶sub file_cache {
Subroutine file_cache: called CC=5 SCAR=16.1
Pod documented
1616 9 my (undef, $dir) = @_;
1617  
1618 9 ▶ croak(_fmt('error_cache_dir', defined($dir) ? $dir : 'undef'))
Branch: 2 branches
BranchTrueFalseTotal
unless defined $dir and not ref $dir and -d $dir 5 4 9
defined $dir ? : 4 1 5
Condition100%6/6
defined $dir and not ref $dir and -d $dir
!ll&&!rl&&r
234
defined $dir and not ref $dir
!ll&&!rl&&r
117
MC/DC100%3/3
defined $dir and not ref $dir and -d $dir
defined $dir not ref $dir -d $dir
Truth table100%4/4
A and not(B) and C
A= defined $dirB= not ref $dirC= -d $dir
ABCresult
0XX0
10X0
1100
1111
1619 unless defined($dir) && !ref($dir) && -d $dir;
1620  
1621 4 return _file_cache_at(_catdir($dir, $CACHE_DIR));
1622 }
1623  
1624 # _file_cache_at($root) -> CHI cache
1625 #
1626 # Geo::Coder::List's cache as files under $root, with a copy of each answer in
1627 # this process's memory. CHI makes $root when it first stores an answer.
1628 17 ▶sub _file_cache_at {
Subroutine _file_cache_at: called CC=1 SCAR=0.0
1629 17 my $root = shift;
1630  
1631 17 require CHI;
1632  
1633 17 return CHI->new(
1634 driver => 'File',
1635 root_dir => $root,
1636 namespace => $CACHE_NAMESPACE,
1637 l1_cache => { driver => 'RawMemory', global => 1, max_size => $CACHE_MAX_SIZE },
1638 );
1639 }
1640  
1641 17 ▶sub _catdir {
Subroutine _catdir: called CC=1 SCAR=0.0
1642 17 require File::Spec;
1643 17 return File::Spec->catdir(@_);
1644 }
1645  
1646 =head2 place_rows
1647  
1648 my ($places, $name) = Database::BI::Model::Geocode->place_rows(\@rows, [ 'Address', 'Country' ]);
1649 my $result = $geocode->resolve_column($places, $name);
1650  
1651 # Places that need their state and country to be found
1652 my ($places, $name) = Database::BI::Model::Geocode->place_rows(\@rows, [ 'County' ], 'Virginia, USA');
1653  
1654 A place written across several columns, such as an address and then its
1655 country, as one column to geocode. Each row's cells are trimmed and joined in
1656 the order given, with C<", ">, leaving out the empty ones:
1657 C<"Brezje 8, Cerknica"> and C<"Slovenia"> become
1658 C<"Brezje 8, Cerknica, Slovenia">. A row with every cell empty gives C<''>,
1659 which C<resolve_column> counts as skipped.
1660  
1661 Text given as the third argument is added to the end of each place, after
1662 C<", ">, for places the cells do not name fully: C<"Accomack"> with
1663 C<"Virginia, USA"> is looked up as C<"Accomack, Virginia, USA">. A row with
1664 no place stays C<''>, so it is not found at the place the text names.
1665  
1666 The rows themselves are not changed (a SQL backend's are locked): the result is
1667 a new row for each, in the same order, holding only the joined place, so a
1668 marker's C<first_row> still indexes C<\@rows>. With a single column and no
1669 text the rows are returned as they are, under that column's name.
1670  
1671 =head3 API SPECIFICATION
1672  
1673 =head4 INPUT
1674  
1675 \@rows arrayref of hashrefs # required; a row that is not a hashref gives undef
1676 \@columns arrayref of column names # required; one or more, in order
1677 $text string # optional; added to the end of each place
1678  
1679 =head4 OUTPUT
1680  
1681 (\@places, $name)
1682 \@places one hashref { $name => $place } per row, or \@rows itself for one
1683 column and no text
1684 $name the columns joined with ", " ("Address, Country")
1685  
1686 Croaks with C<error_rows_arrayref> when C<\@rows> is not an arrayref, and with
1687 C<error_column_required> when C<\@columns> names no column.
1688  
1689 =head3 EXAMPLE
1690  
1691 # loc=Address&loc=Country on the map page
1692 my ($places, $name) = Database::BI::Model::Geocode->place_rows($records, [qw(Address Country)]);
1693 my $result = Database::BI::Model::Geocode->new->resolve_column($places, $name, cache => \%cache);
1694  
1695 =cut
1696  
1697 22 ▶sub place_rows {
Subroutine place_rows: called CC=19 SCAR=29.5
Pod documented
1698 22 my ($class, $rows, $columns, $text) = @_;
1699  
1700 22 ▶ croak(_fmt('error_rows_arrayref', defined($rows) ? ref($rows) || $rows : 'undef'))
Branch: 2 branches
BranchTrueFalseTotal
unless ref $rows eq "ARRAY" 3 19 22
defined $rows ? : 2 1 3
Condition33%1/3
ref $rows or $rows
l!l&&r!l&&!r
200
MC/DC0%0/2
ref $rows or $rows
ref $rows $rows
Truth table33%1/3
A or B
A= ref $rowsB= $rows
ABresult
1X1
011
000
1701 unless ref($rows) eq 'ARRAY';
1702  
1703 26 ▶ my @columns = grep { defined($_) && !ref($_) && length($_) }
Branch: 1 branch
BranchTrueFalseTotal
ref $columns eq 'ARRAY' ? : 18 1 19
Condition100%6/6
defined $_ && !ref($_) && length $_
!ll&&!rl&&r
3320
defined $_ and not ref $_
!ll&&!rl&&r
2123
MC/DC100%3/3
defined $_ && !ref($_) && length $_
defined $_ not ref $_ length $_
Truth table100%4/4
A and not(B) && C
A= defined $_B= not ref $_C= length $_
ABCresult
0XX0
10X0
1100
1111
1704 19 (ref($columns) eq 'ARRAY' ? @{$columns} : ());
1705 19 ▶ croak(_fmt('error_column_required')) unless @columns;
Branch: 1 branch
BranchTrueFalseTotal
unless @columns 4 15 19
1706  
1707 15 my $name = join(', ', @columns);
1708 15 ▶ $text = (defined($text) && !ref($text)) ? ($text =~ s/\A\s+|\s+\z//gr) : '';
Branch: 1 branch
BranchTrueFalseTotal
defined $text && !ref($text) ? : 8 7 15
Condition66%2/3
defined $text and not ref $text
!ll&&!rl&&r
708
MC/DC50%1/2
defined $text and not ref $text
defined $text not ref $text
Truth table66%2/3
A and not(B)
A= defined $textB= not ref $text
ABresult
0X0
100
111
1709 15 ▶ return ($rows, $name) if @columns == 1 && !length($text);
Branch: 1 branch
BranchTrueFalseTotal
if @columns == 1 and not length $text 9 6 15
Condition100%3/3
@columns == 1 and not length $text
!ll&&!rl&&r
519
MC/DC100%2/2
@columns == 1 and not length $text
@columns == 1 not length $text
Truth table100%3/3
A and not(B)
A= @columns == 1B= not length $text
ABresult
0X0
100
111
1710  
1711 my @places = map {
1712 23 my $row = $_;
1713 my $place = (ref($row) eq 'HASH')
1714 33 ? join(', ', grep { length } map {
1715 23 ▶ my $cell = $row->{$_};
Branch: 1 branch
BranchTrueFalseTotal
ref $row eq 'HASH' ? : 18 5 23
1716 33 ▶ (defined($cell) && !ref($cell)) ? ($cell =~ s/\A\s+|\s+\z//gr) : '';
Branch: 1 branch
BranchTrueFalseTotal
defined $cell && !ref($cell) ? : 29 4 33
Condition100%3/3
defined $cell and not ref $cell
!ll&&!rl&&r
3129
MC/DC100%2/2
defined $cell and not ref $cell
defined $cell not ref $cell
Truth table100%3/3
A and not(B)
A= defined $cellB= not ref $cell
ABresult
0X0
100
111
1717 } @columns)
1718 : undef;
1719 23 ▶ $place .= ", $text" if defined($place) && length($place) && length($text);
Branch: 1 branch
BranchTrueFalseTotal
if defined $place and length $place and length $text 4 19 23
Condition100%6/6
defined $place and length $place and length $text
!ll&&!rl&&r
1094
defined $place and length $place
!ll&&!rl&&r
5513
MC/DC100%3/3
defined $place and length $place and length $text
defined $place length $place length $text
Truth table100%4/4
A and B and C
A= defined $placeB= length $placeC= length $text
ABCresult
0XX0
10X0
1100
1111
1720 23 ▶ defined($place) ? +{ $name => $place } : undef;
Branch: 1 branch
BranchTrueFalseTotal
defined $place ? : 18 5 23
1721 6 } @{$rows};
1722  
1723 6 return (\@places, $name);
1724 }
1725  
1726 =head2 place_text
1727  
1728 my ($text) = Database::BI::Model::Geocode->place_text(' , Virginia, USA '); # 'Virginia, USA'
1729  
1730 Tidies text to be added to the end of each place (the third argument of
1731 C<place_rows>): spaces are trimmed and runs of them made one, and a leading
1732 comma dropped, since the place and the text are joined with C<", ">. The map
1733 page's C<loc_text=> and bi-geocode's C<--text> both go through it, so a place
1734 cached by one is found by the other.
1735  
1736 =head3 API SPECIFICATION
1737  
1738 =head4 INPUT
1739  
1740 $value string # optional; undef is no text
1741  
1742 =head4 OUTPUT
1743  
1744 ($text) the tidied text; '' for undef or blank
1745 () for a reference, a control character, or more than
1746 $PLACE_TEXT_MAX (200) characters once tidied
1747  
1748 Never croaks.
1749  
1750 =head3 EXAMPLE
1751  
1752 my ($text) = Database::BI::Model::Geocode->place_text($param)
1753 or die "Not one short line\n";
1754 my ($places, $name) = Database::BI::Model::Geocode->place_rows($rows, [ 'County' ], $text);
1755  
1756 =cut
1757  
1758 66 ▶sub place_text {
Subroutine place_text: called CC=5 SCAR=16.1
Pod documented
1759 66 my ($class, $value) = @_;
1760 66 ▶ return ('') unless defined $value;
Branch: 1 branch
BranchTrueFalseTotal
unless defined $value 8 58 66
1761 58 ▶ return if ref($value) || $value =~ /[\x00-\x1f\x7f]/;
Branch: 1 branch
BranchTrueFalseTotal
if ref $value or $value =~ /[\x00-\x1f\x7f]/ 3 55 58
Condition100%3/3
ref $value or $value =~ /[\x00-\x1f\x7f]/
l!l&&r!l&&!r
1255
MC/DC100%2/2
ref $value or $value =~ /[\x00-\x1f\x7f]/
ref $value $value =~ /[\x00-\x1f\x7f]/
Truth table100%3/3
A or B
A= ref $valueB= $value =~ /[\x00-\x1f\x7f]/
ABresult
1X1
011
000
1762 55 (my $text = $value) =~ s/\s+/ /g;
1763 55 $text =~ s/\A[\s,]+|\s+\z//g;
1764 55 ▶ return if length($text) > $PLACE_TEXT_MAX;
Branch: 1 branch
BranchTrueFalseTotal
if length $text > $Database::BI::Model::Geocode::PLACE_TEXT_MAX 1 54 55
1765 54 return ($text);
1766 }
1767  
1768 =head2 cache_from_models
1769  
1770 my $cache = Database::BI::Model::Geocode->cache_from_models(\%models, $spec_key);
1771  
1772 Extract the geocode cache for one dataset spec from a model store. Returns a
1773 B<copy> of the stored places hashref so the caller cannot accidentally modify
1774 the model store.
1775  
1776 This is a B<class method>: call it on the package name, or on an instance if
1777 you prefer; either way the invocant is ignored.
1778  
1779 =head3 API SPECIFICATION
1780  
1781 =head4 INPUT
1782  
1783 \%models => hashref, required; the structure returned by _read_models()
1784 $spec_key => string, required; the dataset spec (e.g. 'table:sales')
1785  
1786 =head4 OUTPUT
1787  
1788 Returns a plain hashref:
1789  
1790 { $place_name => [lat, lon] | 0, ... }
1791  
1792 Returns an empty hashref C<{}> for an unknown spec, a spec that has never
1793 been geocoded, or a C<\%models> argument that is not a hashref. Never
1794 croaks.
1795  
1796 =head3 EXAMPLE
1797  
1798 my $models = decode_json(Mojo::File->new($model_path)->slurp);
1799 my $cache = Database::BI::Model::Geocode->cache_from_models(
1800 $models, 'table:sales'
1801 );
1802  
1803 # $cache is now { 'Ramsgate' => [51.336, 1.416], 'Paris' => 0, ... }
1804  
1805 =cut
1806  
1807 13 ▶sub cache_from_models {
Subroutine cache_from_models: called CC=4 SCAR=13.9
Pod documented
1808 13 my ($class, $models, $spec_key) = @_;
1809  
1810 13 ▶ my $entry = (ref($models) eq 'HASH') ? $models->{'geo'}{$spec_key} : undef;
Branch: 1 branch
BranchTrueFalseTotal
ref $models eq 'HASH' ? : 12 1 13
1811  
1812 13 ▶ return {} unless ref($entry) eq 'HASH' && (ref($entry->{'places'}) eq 'HASH');
Branch: 1 branch
BranchTrueFalseTotal
unless ref $entry eq "HASH" and ref $$entry{"places"} eq "HASH" 9 4 13
Condition66%2/3
ref $entry eq "HASH" and ref $$entry{"places"} eq "HASH"
!ll&&!rl&&r
904
MC/DC50%1/2
ref $entry eq "HASH" and ref $$entry{"places"} eq "HASH"
ref $entry eq "HASH" ref $$entry{"places"} eq "HASH"
Truth table66%2/3
A and B
A= ref $entry eq "HASH"B= ref $$entry{"places"} eq "HASH"
ABresult
0X0
100
111
1813  
1814 4 return { %{ $entry->{'places'} } };
1815 }
1816  
1817 =head2 models_with_cache
1818  
1819 my $updated_models = Database::BI::Model::Geocode->models_with_cache(
1820 \%models, $spec_key, \%cache,
1821 backend => 'OpenAddresses',
1822 built => '2026-10-18T12:00:00',
1823 );
1824  
1825 Produce a new model store in which the geocode cache for C<$spec_key> is
1826 replaced with C<\%cache>. Every other key in the model store, and every
1827 other spec's geocode data, is carried over unchanged.
1828  
1829 This is a B<class method>; the invocant is ignored.
1830  
1831 This module never writes the file. The caller is responsible for passing
1832 C<$updated_models> to C<_write_models()> or its equivalent, which is where
1833 the writability and C<data_dir> guards live. The separation keeps those
1834 guards in the one place that already owns them
1835  
1836 =head3 API SPECIFICATION
1837  
1838 =head4 INPUT
1839  
1840 \%models => hashref, required; the existing model store (not modified)
1841 $spec_key => string, required; the dataset spec to update
1842 \%cache => hashref, required; { place => [lat, lon] | 0 }
1843 backend => string, optional; name of the geocoder backend that resolved
1844 built => string, optional; ISO-8601 timestamp of when the cache was built
1845  
1846 =head4 OUTPUT
1847  
1848 Returns a new hashref. The caller's C<\%models> is never modified.
1849  
1850 {
1851 ...all existing model keys...,
1852 geo => {
1853 ...all existing geo specs...,
1854 $spec_key => {
1855 places => \%cache,
1856 backend => $backend, # if supplied
1857 built => $built, # if supplied
1858 },
1859 },
1860 }
1861  
1862 =head3 EXAMPLE
1863  
1864 my $updated = Database::BI::Model::Geocode->models_with_cache(
1865 $models,
1866 'path:/data/sales.csv',
1867 $result->{'cache'},
1868 backend => $result->{'backend'},
1869 built => '2026-10-18T09:15:42',
1870 );
1871  
1872 # Now write the updated store to disk.
1873 Mojo::File->new($model_path)->spew(encode_json($updated));
1874  
1875 =cut
1876  
1877 7 ▶sub models_with_cache {
Subroutine models_with_cache: called CC=8 SCAR=20.8
Pod documented
1878 7 my ($class, $models, $spec_key, $cache, %meta) = @_;
1879  
1880 7 ▶ my $spec = { %{ (ref($models) eq 'HASH') ? ($models->{'geo'}{$spec_key} // {}) : {} } };
Branch: 1 branch
BranchTrueFalseTotal
ref $models eq 'HASH' ? : 6 1 7
Condition100%2/2
$$models{"geo"}{$spec_key} // {}
l!l
15
MC/DC100%1/1
$$models{"geo"}{$spec_key} // {}
$$models{"geo"}{$spec_key}
Truth table100%2/2
A
A= $$models{"geo"}{$spec_key}
Aresult
11
00
1881  
1882 7 ▶ $spec->{'places'} = { %{ $cache // {} } };
Condition50%1/2
$cache // {}
l!l
70
MC/DC0%0/1
$cache // {}
$cache
Truth table50%1/2
A
A= $cache
Aresult
11
00
1883  
1884 7 for my $key (qw(backend built)) {
1885 14 ▶ $spec->{$key} = $meta{$key} if defined($meta{$key});
Branch: 1 branch
BranchTrueFalseTotal
if defined $meta{$key} 6 8 14
1886 }
1887  
1888 7 ▶ my %updated = %{ (ref($models) eq 'HASH') ? $models : {} };
Branch: 1 branch
BranchTrueFalseTotal
ref $models eq 'HASH' ? : 6 1 7
1889  
1890 7 ▶ $updated{'geo'} = { %{ $updated{'geo'} // {} }, $spec_key => $spec };
Condition100%2/2
$updated{"geo"} // {}
l!l
61
MC/DC100%1/1
$updated{"geo"} // {}
$updated{"geo"}
Truth table100%2/2
A
A= $updated{"geo"}
Aresult
11
00
1891  
1892 7 return \%updated;
1893 }
1894  
1895 =head2 shared_cache_from_models
1896  
1897 my $cache = Database::BI::Model::Geocode->shared_cache_from_models(\%models);
1898  
1899 Extract the place answers every dataset shares, as a B<copy> the caller may
1900 modify. This is the cache a map request should pass to C<resolve_column>:
1901 a place geocoded for one file is then plotted for every other file that names
1902 it, without a second lookup.
1903  
1904 A model store written before the cache was shared has no shared entry; this
1905 returns C<{}> and the per-dataset caches remain readable with
1906 C<cache_from_models>. A caller that reads both gets the old answers once and
1907 writes them back to the shared entry (see C<models_with_shared_cache>).
1908  
1909 This is a B<class method>; the invocant is ignored.
1910  
1911 =head3 API SPECIFICATION
1912  
1913 =head4 INPUT
1914  
1915 \%models => hashref, required; the structure returned by _read_models()
1916  
1917 =head4 OUTPUT
1918  
1919 { $place_name => [lat, lon] | 0, ... }
1920  
1921 Returns an empty hashref for a store with no shared cache, for a malformed
1922 entry, or for a C<\%models> argument that is not a hashref. Never croaks.
1923  
1924 =head3 EXAMPLE
1925  
1926 my $models = decode_json(Mojo::File->new($model_path)->slurp);
1927 my $cache = Database::BI::Model::Geocode->shared_cache_from_models($models);
1928  
1929 # $cache is now { 'Ramsgate' => [51.336, 1.416], 'Paris' => 0, ... }
1930  
1931 =cut
1932  
1933 38 ▶sub shared_cache_from_models {
Subroutine shared_cache_from_models: called CC=5 SCAR=16.1
Pod documented
1934 38 my ($class, $models) = @_;
1935  
1936 # One level at a time: a nested read would add an empty 'geo' to the caller's
1937 # store, and the next write would save it.
1938 38 ▶ my $geo = (ref($models) eq 'HASH') ? $models->{'geo'} : undef;
Branch: 1 branch
BranchTrueFalseTotal
ref $models eq 'HASH' ? : 31 7 38
1939 38 ▶ my $entry = (ref($geo) eq 'HASH') ? $geo->{$SHARED_CACHE_KEY} : undef;
Branch: 1 branch
BranchTrueFalseTotal
ref $geo eq 'HASH' ? : 20 18 38
1940  
1941 38 ▶ return {} unless ref($entry) eq 'HASH' && (ref($entry->{'places'}) eq 'HASH');
Branch: 1 branch
BranchTrueFalseTotal
unless ref $entry eq "HASH" and ref $$entry{"places"} eq "HASH" 26 12 38
Condition100%3/3
ref $entry eq "HASH" and ref $$entry{"places"} eq "HASH"
!ll&&!rl&&r
25112
MC/DC100%2/2
ref $entry eq "HASH" and ref $$entry{"places"} eq "HASH"
ref $entry eq "HASH" ref $$entry{"places"} eq "HASH"
Truth table100%3/3
A and B
A= ref $entry eq "HASH"B= ref $$entry{"places"} eq "HASH"
ABresult
0X0
100
111
1942  
1943 12 return { %{ $entry->{'places'} } };
1944 }
1945  
1946 =head2 models_with_shared_cache
1947  
1948 my $updated_models = Database::BI::Model::Geocode->models_with_shared_cache(
1949 \%models, \%cache,
1950 backend => 'OpenAddresses',
1951 built => '2026-10-18T12:00:00',
1952 );
1953  
1954 Produce a new model store in which the shared place cache holds C<\%cache>.
1955 The cache is B<merged>, not replaced: a request writes back every place it
1956 saw, and another request may have answered a different place since this one
1957 read the store. Other model keys, and every per-dataset C<geo> entry, are
1958 carried over unchanged.
1959  
1960 This module never writes the file, and does not add a per-dataset entry. The
1961 caller passes the result to C<_write_models()> or its equivalent, where the
1962 writability and C<data_dir> guards live.
1963  
1964 =head3 API SPECIFICATION
1965  
1966 =head4 INPUT
1967  
1968 \%models => hashref, required; the existing model store (not modified)
1969 \%cache => hashref, required; { place => [lat, lon] | 0 }
1970 backend => string, optional; name of the geocoder backend that resolved
1971 built => string, optional; ISO-8601 timestamp of the cache update
1972  
1973 =head4 OUTPUT
1974  
1975 Returns a new hashref. The caller's C<\%models> is never modified.
1976  
1977 {
1978 ...all existing model keys...,
1979 geo => {
1980 ...all existing per-dataset entries...,
1981 shared => {
1982 places => \%merged_cache,
1983 backend => $backend, # if supplied
1984 built => $built, # if supplied
1985 },
1986 },
1987 }
1988  
1989 =head3 EXAMPLE
1990  
1991 my $updated = Database::BI::Model::Geocode->models_with_shared_cache(
1992 $models,
1993 $result->{'cache'},
1994 backend => $result->{'backend'},
1995 built => '2026-10-18T09:15:42',
1996 );
1997  
1998 Mojo::File->new($model_path)->spew(encode_json($updated));
1999  
2000 =cut
2001  
2002 14 ▶sub models_with_shared_cache {
Subroutine models_with_shared_cache: called CC=8 SCAR=20.8
Pod documented
2003 14 my ($class, $models, $cache, %meta) = @_;
2004  
2005 # Merge over what is already shared: the request may have read the store
2006 # before another one resolved a place, and replacing the entry would lose
2007 # that answer for every dataset rather than just this request.
2008 my %places = (
2009 14 %{ $class->shared_cache_from_models($models) },
2010 14 ▶ %{ (ref($cache) eq 'HASH') ? $cache : {} },
Branch: 1 branch
BranchTrueFalseTotal
ref $cache eq 'HASH' ? : 13 1 14
2011 );
2012  
2013 # A section of the wrong shape (a hand-edited store) is rebuilt, not dereferenced.
2014 14 ▶ my $geo = (ref($models) eq 'HASH' && ref($models->{'geo'}) eq 'HASH') ? $models->{'geo'} : {};
Branch: 1 branch
BranchTrueFalseTotal
ref $models eq 'HASH' && ref $$models{'geo'} eq 'HASH' ? : 6 8 14
Condition100%3/3
ref $models eq "HASH" and ref $$models{"geo"} eq "HASH"
!ll&&!rl&&r
356
MC/DC100%2/2
ref $models eq "HASH" and ref $$models{"geo"} eq "HASH"
ref $models eq "HASH" ref $$models{"geo"} eq "HASH"
Truth table100%3/3
A and B
A= ref $models eq "HASH"B= ref $$models{"geo"} eq "HASH"
ABresult
0X0
100
111
2015 14 my $entry = $geo->{$SHARED_CACHE_KEY};
2016 14 ▶ my $spec = { %{ (ref($entry) eq 'HASH') ? $entry : {} } };
Branch: 1 branch
BranchTrueFalseTotal
ref $entry eq 'HASH' ? : 4 10 14
2017  
2018 14 $spec->{'places'} = \%places;
2019  
2020 14 for my $key (qw(backend built)) {
2021 28 ▶ $spec->{$key} = $meta{$key} if defined($meta{$key});
Branch: 1 branch
BranchTrueFalseTotal
if defined $meta{$key} 9 19 28
2022 }
2023  
2024 14 ▶ my %updated = %{ (ref($models) eq 'HASH') ? $models : {} };
Branch: 1 branch
BranchTrueFalseTotal
ref $models eq 'HASH' ? : 11 3 14
2025  
2026 14 $updated{'geo'} = { %$geo, $SHARED_CACHE_KEY => $spec };
2027  
2028 14 return \%updated;
2029 }
2030  
2031 3 1;
2032  
2033 =head1 COMMON PITFALLS
2034  
2035 =over 4
2036  
2037 =item B<The cache sentinel 0 is not "false -- never looked up".>
2038  
2039 A cache entry of C<0> means the place was looked up and not found. An absent
2040 key means it has never been attempted. The module treats these two states
2041 differently: a C<0> entry contributes to C<counts.unresolved> without
2042 triggering a new lookup; an absent key enters the classification loop and may
2043 reach the geocoder.
2044  
2045 # WRONG -- checking definedness misses the 0 sentinel.
2046 my $resolved = defined($cache{$place});
2047  
2048 # CORRECT -- use the internal _has_position logic.
2049 my $resolved = (ref($cache{$place}) eq 'ARRAY');
2050  
2051 =item B<Pure-digit strings are skipped, not attempted.>
2052  
2053 A value like C<'42'> fails C<valid_location()> and goes into C<@skipped>. It
2054 is never stored in the cache. This is the correct behaviour for a row number
2055 that ended up in the wrong column, but it surprises callers who expect every
2056 non-empty value to be attempted.
2057  
2058 =item B<The returned cache is a copy; the caller's hashref is never modified.>
2059  
2060 Do not compare C<$cache_you_passed_in> with C<$result->{'cache'}> expecting
2061 them to be the same reference. The update is in C<$result->{'cache'}>.
2062  
2063 =item B<resolve_column croaks when the column is absent; resolve_values does not.>
2064  
2065 C<resolve_column> checks that the named column appears in at least one row.
2066 If it does not, it croaks with C<error_map_loc_column>. C<resolve_values>
2067 does not do this check, because it receives values, not rows.
2068  
2069 =item B<max counts new lookups, not rows or distinct values.>
2070  
2071 A value already in C<cache> does not count toward C<max>, even if it was
2072 looked up in a previous request. A value skipped by C<valid_location()> does
2073 not count either.
2074  
2075 =item B<new() always succeeds; geocoder load failure is not a constructor error.>
2076  
2077 You can always call C<new()> without checking whether C<Geo::Coder::Free> is
2078 installed. The load failure is deferred to the first call that needs the
2079 geocoder, and is reported through C<geocoder_available()> and the
2080 C<messages> array in the result, not through an exception.
2081  
2082 =item B<cache_from_models and models_with_cache are class methods.>
2083  
2084 You may call them on an instance for convenience, but the invocant is not used.
2085 The pattern C<< Database::BI::Model::Geocode->cache_from_models(...) >> is
2086 clearer.
2087  
2088 =item B<An injected geocoder's batch receives on_error => 'undef'.>
2089  
2090 A value an injected geocoder rejects after the C<valid_location()> pre-filter is
2091 returned as C<undef> from the batch; it does not croak. A backend that throws
2092 an exception (a real database error, a network failure) still propagates
2093 upward, because nothing here wraps the geocoder in an C<eval>.
2094  
2095 =back
2096  
2097 =head1 ENCODING
2098  
2099 =over 4
2100  
2101 =item B<Place names (values in the location column)>
2102  
2103 Passed to the geocoder as byte strings. UTF-8 place names such as
2104 "Koeln" or "Beijing" are supported if the geocoder backend handles them.
2105 The internal code does not re-encode or validate the bytes; it compares them
2106 as opaque strings for deduplication and caching.
2107  
2108 =item B<Column names>
2109  
2110 Matched against row hash keys with a string equality test. The column name
2111 must be byte-for-byte identical to the key in the row hashrefs. Column names
2112 with non-ASCII bytes are accepted.
2113  
2114 =item B<Spec keys (the $spec_key argument)>
2115  
2116 Used as hash keys in the model store. Any byte sequence is safe; UTF-8 spec
2117 keys are stored and retrieved without modification.
2118  
2119 =item B<Message strings>
2120  
2121 The built-in C<%MESSAGES> table contains ASCII-only text. If your application
2122 needs non-ASCII messages (for example, a French translation), supply an C<i18n>
2123 object to C<new()> whose C<maketext> method renders the key in the desired
2124 locale.
2125  
2126 =item B<Geocoder result strings (backend names)>
2127  
2128 Stored and reported as returned by the geocoder. The module does not impose
2129 any encoding on them.
2130  
2131 =back
2132  
2133 =head1 DIAGNOSTICS
2134  
2135 All user-visible messages are keyed in C<%Database::BI::Model::Geocode::MESSAGES>.
2136  
2137 =over 4
2138  
2139 =item C<error_map_geocoder_missing>
2140  
2141 Geocoding a location column needs Geo::Coder::Free %s or later, and %s
2142  
2143 The geocoder module is not installed (the second C<%s> is "it is not
2144 installed") or is too old (the second C<%s> is the installed version number).
2145 The first C<%s> is the floor this module checks for - C<0.48> unless the caller
2146 pinned C<min_geocoder_version>. Install or upgrade C<Geo::Coder::Free>.
2147  
2148 =item C<error_map_geocoder_list_missing>
2149  
2150 Geocoding a location column needs Geo::Coder::List 0.39 or later, and %s
2151  
2152 C<Geo::Coder::Free> loaded but C<Geo::Coder::List>, which holds the chain, did
2153 not: the C<%s> is "it is not installed" or the version that is. Install or
2154 upgrade C<Geo::Coder::List>.
2155  
2156 =item C<error_cache_dir>
2157  
2158 Geocode: the geocode cache needs a directory that exists, not "%s"
2159  
2160 C<file_cache()> was given a path that is not a directory. Create it first.
2161  
2162 =item C<error_cache_interface>
2163  
2164 Geocode: option "cache" must be a hashref or an object with get() and set(), not a %s
2165  
2166 C<new()> was given a C<cache> object that Geo::Coder::List cannot use.
2167  
2168 =item C<error_online_max_invalid>
2169  
2170 Geocode: option "online_max" must be a positive integer, not "%s"
2171  
2172 C<new()> was given an C<online_max> that is not a positive integer. Leave it
2173 out for no limit, or pass C<< online => 0 >> to ask Nominatim nothing.
2174  
2175 =item C<error_map_loc_column>
2176  
2177 Column "%s" is not in this dataset, so there is nothing to geocode
2178  
2179 The C<$column_name> argument to C<resolve_column()> did not appear as a key in
2180 any row. Check the column name, including case and any spaces.
2181  
2182 =item C<error_geocoder_interface>
2183  
2184 Geocode: the geocoder must provide %s; Geo::Coder::Free %s or later does
2185  
2186 An injected C<geocoder> object does not respond to C<valid_location()> or
2187 C<geocode_many()>. Either use the real C<Geo::Coder::Free> or add the missing
2188 method to your stub.
2189  
2190 =item C<error_max_invalid>
2191  
2192 Geocode: option "max" must be a positive integer, not "%s"
2193  
2194 The C<max> option was supplied but is not a positive integer string. Passing
2195 C<< max => 0 >> and C<< max => 'all' >> both trigger this error.
2196  
2197 =item C<error_values_arrayref>
2198  
2199 Geocode: the values must be an arrayref, not "%s"
2200  
2201 The first argument to C<resolve_values()> is not an arrayref. Pass
2202 C<\@values> (a reference), not C<@values> (a flat list).
2203  
2204 =item C<error_rows_arrayref>
2205  
2206 Geocode: the rows must be an arrayref, not "%s"
2207  
2208 The first argument to C<resolve_column()> is not an arrayref.
2209  
2210 =item C<error_column_required>
2211  
2212 Geocode: the column name is required
2213  
2214 The C<$column_name> argument to C<resolve_column()> is C<undef> or an empty
2215 string.
2216  
2217 =item C<warn_map_rows_skipped>
2218  
2219 Plotted %d of %d rows; %d could not be plotted: %s
2220  
2221 Some rows could not be plotted. The last argument says why, counted in rows,
2222 naming only the reasons that apply, joined with C<, >; each is one of
2223  
2224 warn_map_rows_why_empty %d with no place in the row
2225 warn_map_rows_why_refused %d whose place is a bare number, not a place name
2226 warn_map_rows_why_not_found %d whose place was not found
2227 warn_map_rows_why_later %d whose place has not been looked up yet
2228  
2229 in that order, rendered through the same C<i18n> object as the message. Not an
2230 error: the map renders with the rows that could be resolved.
2231  
2232 =item C<warn_map_geocode_capped>
2233  
2234 Geocoded %d of the %d places that had not been looked up before; ...
2235  
2236 The C<max> cap, or C<online_max>, prevented all pending values from being
2237 looked up in this request. The un-looked-up values are in C<not_attempted> and
2238 are not stored in the cache, so the next request picks them up.
2239  
2240 =item C<warn_map_geocoder_unavailable_data>
2241  
2242 The geocoder has no OpenAddresses data configured (OPENADDR_HOME), ...
2243  
2244 The geocoder was loaded successfully but C<OPENADDR_HOME> is not set,
2245 C<openaddr> was not passed to C<new()>, and Nominatim is not in the chain.
2246 Resolution will succeed for broad place names (cities, countries) but will fail
2247 for street-level addresses. Set the environment variable or pass
2248 C<< openaddr => $path >> to C<new()>.
2249  
2250 =item C<warn_map_geocoder_local_only>
2251  
2252 Places the local data does not know were not looked up online, which needs Geo::Coder::OSM 0.03 and LWP::UserAgent::Throttled 0.14 or later
2253  
2254 Online lookups were wanted, but one of the two modules is missing or too old,
2255 and some places were not found. Install them, or pass C<< online => 0 >> to
2256 say the local data is all that is wanted.
2257  
2258 =item C<warn_map_geocoder_online_failed>
2259  
2260 OpenStreetMap did not answer (%s), so %d places the local data does not know are left to try again later
2261  
2262 Nominatim answered with an error (the C<%s> is its status line, such as
2263 C<429 Too Many Requests>) or not at all. The places are in C<not_attempted>,
2264 uncached, and the next request asks again.
2265  
2266 =back
2267  
2268 =head1 FORMAL SPECIFICATION
2269  
2270 The following sections use Z calculus notation to describe the exact state
2271 transitions and invariants of each public operation. Notation: ∈ (member of),
2272 ∉ (not member of), ⊆ (subset), ∪ (union), ∩ (intersection), ∧ (and), ∨ (or),
2273 ¬ (not), ∀ (for all), ∃ (there exists), → (maps to), ≤ (less than or equal),
2274 ≠ (not equal).
2275  
2276 =head3 FORMAL SPECIFICATION
2277  
2278 State transitions and invariants for each operation are described below.
2279  
2280 =head2 _resolve -- value classification and batch geocoding
2281  
2282 Let V be the multiset of input values.
2283  
2284 Define the following disjoint sets that partition distinct(V):
2285  
2286 SKIPPED = { v | v = undef, ref(v), or len(v) = 0,
2287 or geocoder.valid_location(v) = false }
2288 CACHED_RESOLVED = { v | v in dom(CACHE) and _has_position(CACHE(v)) }
2289 CACHED_UNRESOLVED = { v | v in dom(CACHE) and not _has_position(CACHE(v)) }
2290 NOT_ATTEMPTED = { v | geocoder unavailable } (excludes SKIPPED and CACHED)
2291 PENDING = distinct(V) \ (SKIPPED ∪ CACHED_RESOLVED ∪ CACHED_UNRESOLVED
2292 ∪ NOT_ATTEMPTED)
2293  
2294 Partition invariant:
2295  
2296 |SKIPPED| + |CACHED_RESOLVED| + |CACHED_UNRESOLVED|
2297 + |NOT_ATTEMPTED| + |PENDING| = |distinct(V)|
2298  
2299 Cap application (MAX = the max option, default 500):
2300  
2301 ATTEMPT = PENDING[0 .. min(MAX, |PENDING|) - 1]
2302 CAPPED = |PENDING| > MAX
2303 NOT_ATTEMPTED' = NOT_ATTEMPTED ∪ PENDING[MAX .. |PENDING| - 1] (when CAPPED)
2304  
2305 Deferral (lookup => 0):
2306  
2307 R.pending = ATTEMPT
2308 NOT_ATTEMPTED' = NOT_ATTEMPTED' ∪ ATTEMPT
2309 ATTEMPT = ∅ (so CACHE' = CACHE, and nothing is asked)
2310 CAPPED = false
2311  
2312 Otherwise R.pending = ∅.
2313  
2314 Geocoding step -- for each v in ATTEMPT:
2315  
2316 BATCH = { v => chain.geocode(v) | v in ATTEMPT } (each through the list's
2317 cache; an injected geocoder: geocode_many(ATTEMPT, on_error => 'undef');
2318 a v whose lookup threw and found nothing moves to NOT_ATTEMPTED)
2319  
2320 for v in ATTEMPT:
2321 (lat, lon) = coordinates(BATCH(v))
2322 if defined(lat) and defined(lon):
2323 CACHE'(v) = [lat, lon]
2324 POINTS'(v) = { lat, lon, backend? }
2325 else:
2326 CACHE'(v) = 0
2327 UNRESOLVED_NEW += {v}
2328  
2329 Count invariants (always hold):
2330  
2331 skipped + cached + attempted + not_attempted = distinct(V)
2332 resolved + unresolved = cached + attempted
2333 resolved = |CACHED_RESOLVED| + |ATTEMPT ∩ POINTS'|
2334 unresolved = |CACHED_UNRESOLVED| + |ATTEMPT \ POINTS'|
2335 lookups <= attempted
2336  
2337 Cache monotonicity:
2338  
2339 dom(CACHE') ⊇ dom(CACHE)
2340 For all v in dom(CACHE): CACHE'(v) = CACHE(v) (existing entries not modified)
2341  
2342 =head2 resolve_column -- row augmentation
2343  
2344 Given the output R of _resolve applied to the column's cell sequence CELLS:
2345  
2346 row_count : Value -> N where row_count(v) = |{ i | CELLS[i] = v }|
2347 first_row : Value -> N where first_row(v) = min{ i | CELLS[i] = v }
2348  
2349 For each v in dom(R.points):
2350 R.points(v).rows = row_count(v)
2351 R.points(v).first_row = first_row(v)
2352  
2353 R.markers = [ { value: v, ...R.points(v) }
2354 | v <- sort_by(first_row, dom(R.points)) ]
2355  
2356 R.counts.rows = |CELLS|
2357 R.counts.plotted = sum{ row_count(v) | v in dom(R.points) }
2358 R.counts.unusable = R.counts.rows - R.counts.plotted
2359  
2360 =head2 cache_from_models -- safe extraction
2361  
2362 cache_from_models(models, spec)
2363 = if ref(models) = HASH and ref(models.geo.spec) = HASH
2364 and ref(models.geo.spec.places) = HASH
2365 then copy(models.geo.spec.places)
2366 else {}
2367  
2368 The result is a shallow copy: the caller cannot modify the model store by
2369 modifying the returned hashref, but values inside the cache (the [lat, lon]
2370 arrayrefs) are shared references.
2371  
2372 =head2 models_with_cache -- safe merge
2373  
2374 models_with_cache(models, spec, cache, backend?, built?)
2375 = { ...models,
2376 geo: { ...models.geo,
2377 spec: { ...models.geo.spec,
2378 places: copy(cache),
2379 backend: backend (if defined),
2380 built: built (if defined) } } }
2381  
2382 The caller's models hashref is never modified; a new top-level hashref is
2383 returned.
2384  
2385 =head1 STATE DIAGRAM
2386  
2387 The object is in one of three states, according to whether it has a geocoder yet.
2388 A geocoder given to C<new> is one straight away; otherwise one is looked for on
2389 the first call that needs it.
2390  
2391 new()
2392 |
2393 +--[bad arguments, or a geocoder lacking
2394 | valid_location or geocode_many]-----> croaks; no object
2395 |
2396 +--[geocoder => $g]---------------------> READY
2397 |
2398 v
2399 +-------------------+
2400 | CONSTRUCTED | Geocoder not yet lazy-loaded.
2401 | _geocoder not | resolve_*() and geocoder_available()
2402 | yet called | may trigger a load on first call.
2403 +-------------------+
2404 | |
2405 first call, first call,
2406 module OK module absent, too old or will not
2407 | load, or its new() answers nothing
2408 v v
2409 +---------+ +------------+
2410 | READY | | UNAVAILABLE|
2411 | | | |
2412 | Geocoder| | No lookups |
2413 | loaded | | possible. |
2414 | and | | Cache still|
2415 | working.| | served. |
2416 +---------+ +------------+
2417 | |
2418 resolve_*() resolve_*()
2419 geocodes + cache only;
2420 cache not_attempted
2421 | |
2422 v v
2423 +---------+ +------------+
2424 | READY | | UNAVAILABLE|
2425 | (no | | (no |
2426 | change) | | change) |
2427 +---------+ +------------+
2428  
2429 Once the object is READY or UNAVAILABLE that does not change: neither a module
2430 that appears nor one that goes away moves it, and the module is looked for once.
2431  
2432 There are two ways for a call to croak and leave the state as it was. Arguments a
2433 resolve_*() call refuses are refused before any load is attempted. A module that
2434 loads and whose new() dies sends that exception to the caller and remembers
2435 nothing, so the object stays CONSTRUCTED and the next call tries again.
2436  
2437 Resolution itself is stateless: all state is in the result hashref and the cache
2438 the caller manages.
2439  
2440 Valid state transitions summary:
2441  
2442 [START] -> CONSTRUCTED (new() with no geocoder)
2443 [START] -> READY (new() given a geocoder)
2444 [START] -> ERROR (new() refuses its arguments; croaks, no object)
2445 CONSTRUCTED -> READY (first call, module OK)
2446 CONSTRUCTED -> UNAVAILABLE (first call, no module that can be used)
2447 CONSTRUCTED -> CONSTRUCTED (arguments refused, or the module's new() dies; croaks)
2448 READY -> READY (resolve_*(), or arguments refused)
2449 UNAVAILABLE -> UNAVAILABLE (resolve_*(), or arguments refused)
2450  
2451 Value-level state transitions within a single resolve_* call. Each distinct value
2452 takes the first of these that applies, from the top. CACHE is the C<cache> in the
2453 result, a copy: the cache the caller passed in is never modified.
2454  
2455 RECEIVED
2456 |
2457 +--[undef / ref / empty]-----------> SKIPPED
2458 |
2459 +--[in cache, has position]---------> CACHED_RESOLVED --> POINTS
2460 |
2461 +--[in cache, no position (= 0)]----> CACHED_UNRESOLVED --> UNRESOLVED
2462 |
2463 +--[geocoder unavailable]-----------> NOT_ATTEMPTED
2464 |
2465 +--[valid_location() throws]--------> EXCEPTION
2466 |
2467 +--[valid_location() = false]-------> SKIPPED
2468 |
2469 +--[within cap]--------------------> ATTEMPT
2470 | |
2471 | +--[geocoder returns position]--> RESOLVED_NEW --> POINTS, CACHE==[lat,lon]
2472 | |
2473 | +--[geocoder returns no position]--> UNRESOLVED_NEW, CACHE==0
2474 | |
2475 | +--[geocoder throws]--> EXCEPTION
2476 |
2477 +--[beyond cap]-------------------> NOT_ATTEMPTED
2478  
2479 EXCEPTION is the whole request failing, on purpose: it goes to the caller, who has
2480 no result and a cache exactly as it was, so nothing can be half-written.
2481  
2482 =head1 AUTHOR
2483  
2484 Nigel Horne C<< <njh@nigelhorne.com> >>
2485  
2486 =head1 LICENCE AND COPYRIGHT
2487  
2488 Copyright 2026 Nigel Horne.
2489  
2490 Usage is subject to the GPL2 licence terms.
2491 If you use it,
2492 please let me know.
2493  
2494 =cut
« blib/lib/Database/BI/Insights.pm ↑ Summary script/database-bi »