Pipeline/Input.pm

filter: Statementstmt 100% 15 / 15 Branchbran n/a Conditioncond n/a MC/DCmcdc n/a Subroutinesub 100% 4 / 4 Podpod n/a totaltotal 100% 19 / 19 CC 1 SCAR 0.0
CC
1
Cov
100%
BEGIN
1.0
BEGIN
1.0
BEGIN
1.0
CRAP 1.0
« Pipeline/Output/UnitTest.pm ↑ Summary Pipeline/Input/File/Table.pm »
1 =pod
2
3 =head1 NAME
4
5 ETL::Pipeline::Input - Role for ETL::Pipeline input sources
6
7 =head1 SYNOPSIS
8
9 use Moose;
10 with 'ETL::Pipeline::Input';
11
12 sub run {
13 # Add code to read your data here
14 ...
15 }
16
17 =head1 DESCRIPTION
18
19 An I<input source> feeds the B<extract> part of B<ETL>. This is where data comes
20 from. These are your data sources.
21
22 A data source may be anything - a file, a database, or maybe a socket. Each
23 I<format> is an L<ETL::Pipeline> input source. For example, Excel files
24 represent one input source. Perl reads every Excel file the same way. With a few
25 judicious attributes, we can re-use the same input source for just about any
26 type of Excel file.
27
28 L<ETL::Pipeline> defines an I<input source> as a Moose object with at least one
29 method - C<run>. This role basically defines the requirement for the B<run>
30 method. It should be consumed by B<all> input source classes. L<ETL::Pipeline>
31 relies on the input source having this role.
32
33 =head2 How do I create an I<input source>?
34
35 =over
36
37 =item 1. Start a new Perl module. I recommend putting it in the C<ETL::Pipeline::Input> namespace. L<ETL::Pipeline> will pick it up automatically.
38
39 =item 2. Make your module a L<Moose> class - C<use Moose;>.
40
41 =item 3. Consume this role - C<with 'ETL::Pipeline::Input';>.
42
43 =item 4. Write the L</run> method. L</run> follows this basic algorithmn...
44
45 =over
46
47 =item a. Open the source.
48
49 =item b. Loop reading the records. Each iteration should call L<ETL::Pipeline/record> to trigger the I<transform> step.
50
51 =item c. Close the source.
52
53 =back
54
55 =item 5. Add any attributes for your class.
56
57 =back
58
59 The new source is ready to use, like this...
60
61 $etl->input( 'YourNewSource' );
62
63 You can leave off the leading B<ETL::Pipeline::Input::>.
64
65 When L<ETL::Pipeline> calls L</run>, it passes the L<ETL::Pipeline> object as
66 the only parameter.
67
68 =head2 Why this way?
69
70 Input sources mostly follow the basic algorithm of open, read, process, and
71 close. I originally had the role define methods for each of these steps. That
72 was a lot of work, and kind of confusing. This way, the input source only
73 I<needs> one code block that does all of these steps - in one place. So it's
74 easier to troubleshoot and write new sources.
75
76 In the work that I do, we have one output destination that rarely changes. It's
77 far more common to write new input sources - especially customized sources.
78 Making new sources easier saves time. Making it simpler means that more
79 developers can pick up those tasks.
80
81 =head2 Does B<ETL::Pipeline> only work with files?
82
83 No. B<ETL::Pipeline::Input> works for any source of data, such as SQL queries,
84 CSV files, or network sockets. Tailor the C<run> method for whatever suits your
85 needs.
86
87 Because files are most common, B<ETL::Pipeline> comes with a helpful role -
88 L<ETL::Pipeline::Input::File>. Consume L<ETL::Pipeline::Input::File> in your
89 inpiut source to access some standardized attributes.
90
91 =head2 Upgrading from older versions
92
93 L<ETL::Pipeline> version 3 is not compatible with input sources from older
94 versions. You will need to rewrite your custom input sources.
95
96 =over
97
98 =item Merge the C<setup>, C<finish>, and C<next_record> methods into L</run>.
99
100 =item Have L</run> call C<$etl->record> in place of C<next_record>.
101
102 =item Adjust attributes as necessary.
103
104 =back
105
106 =cut
107
108 package ETL::Pipeline::Input;
109
110 10 ▶use 5.014000;
Subroutine BEGIN: called CC=1 SCAR=0.0
111 10 ▶use warnings;
Subroutine BEGIN: called CC=1 SCAR=0.0
112
113 10 ▶use Moose::Role;
Subroutine BEGIN: called CC=1 SCAR=0.0
114
115
116 10 our $VERSION = '3.00';
117
118
119 =head1 METHODS & ATTRIBUTES
120
121 =head3 path (optional)
122
123 If you define this, the standard logging will include it. The attribute is
124 named for file inputs. But it can return any value that is meaningful to your
125 users.
126
127 =head3 position (optional)
128
129 If you define this, the standard logging includes it with error or informational
130 messages. It can be any value that helps users locate the correct place to
131 troubleshoot.
132
133 =head3 run (required)
134
135 You define this method in the consuming class. It should open the file, read
136 each record, call L<ETL::Pipeline/record> after each record, and close the file.
137 This method is the workhorse. It defines the main ETL loop.
138 L<ETL::Pipeline/record> acts as a callback.
139
140 I say I<file>. It really means I<input source> - whatever that might be.
141
142 Some important things to remember about C<run>...
143
144 =over
145
146 =item C<run> receives one parameter - the L<ETL::Pipeline> object.
147
148 =item Should include all the code to open, read, and close the input source.
149
150 =item After reading a record, call L<ETL::Pipeline/record>.
151
152 =back
153
154 If your code encounters an error, B<run> can call L<ETL::Pipeline/status> with
155 the error message. L<ETL::Pipeline/status> should automatically include the
156 record count with the error message. You should add any other troubleshooting
157 information such as file names or key fields.
158
159 $etl->status( "ERROR", "Error message here for id $id" );
160
161 For fatal errors, I recommend using the C<croak> command from L<Carp>.
162
163 =cut
164
165 10 requires 'run';
166
167
168 =head3 source
169
170 The location in the input source of the current record. For example, for files
171 this would be the file name and character position. The consuming class can set
172 this value in its L<run|ETL::Pipeline::Input/run> method.
173
174 L<Logging|ETL::Pipeline/log> uses this when displaying errors or informational
175 messages. The value should be something that helps the user troubleshoot issues.
176 It can be whatever is appropriate for the input source.
177
178 B<NOTE:> Don't capitalize the first letter, unless it's supposed to be.
179 L<Logging|ETL::Pipeline/log> will upper case the first letter if it's
180 appropriate.
181
182 =cut
183
184 10 has 'source' => (
185 default => '',
186 is => 'rw',
187 isa => 'Str',
188 );
189
190
191 =head1 SEE ALSO
192
193 L<ETL::Pipeline>, L<ETL::Pipeline::Input::File>, L<ETL::Pipeline::Output>
194
195 =head1 AUTHOR
196
197 Robert Wohlfarth <robert.j.wohlfarth@vumc.org>
198
199 =head1 LICENSE
200
201 Copyright 2021 (c) Vanderbilt University Medical Center
202
203 This program is free software; you can redistribute it and/or modify it under
204 the same terms as Perl itself.
205
206 =cut
207
208 10 ▶no Moose;
Subroutine BEGIN: called CC=1 SCAR=0.0
209
210 # Required by Perl to load the module.
211 10 1;
« Pipeline/Output/UnitTest.pm ↑ Summary Pipeline/Input/File/Table.pm »