| 1 | package MyCPAN::App::DPAN::UserManual; | ||
| 2 | use strict; | ||
| 3 | use warnings; | ||
| 4 | use utf8; | ||
| 5 | |||
| 6 | use vars qw($VERSION); | ||
| 7 | $VERSION = '1.281'; | ||
| 8 | |||
| 9 | =encoding utf8 | ||
| 10 | |||
| 11 | =head1 NAME | ||
| 12 | |||
| 13 | MyCPAN::App::DPAN::UserManual - How to manage a DPAN | ||
| 14 | |||
| 15 | =head1 DESCRIPTION | ||
| 16 | |||
| 17 | DPAN, short for D{ark|istributed|ecentralized} Perl Archive Network, | ||
| 18 | helps you create your own Perl distributions repository that you can | ||
| 19 | use with standard CPAN tools. You can put any distributions with any | ||
| 20 | versions that you like in your own repository since you completely | ||
| 21 | control it. These can be distributions from the real CPAN, private | ||
| 22 | distributions that you create yourself, or older versions of | ||
| 23 | distributions from BackPAN (the historical CPAN archive). | ||
| 24 | |||
| 25 | The DPAN stuff takes a bunch of distributions that you specify and | ||
| 26 | organizes them into a CPAN-like repository. The simplest process is to | ||
| 27 | dump all of your distributions in a single directory and run the | ||
| 28 | C<dpan> command in the same repository: | ||
| 29 | |||
| 30 | % cd my_dist_directory | ||
| 31 | % dpan | ||
| 32 | |||
| 33 | By default, C<dpan> finds and organizes all of the distributions in | ||
| 34 | the directory into the appropriate CPAN-like structure. Behind the | ||
| 35 | scenes, C<MyCPAN::Indexer> looks at each distribution and creates a | ||
| 36 | report from it. You should see a directory that contains a reports for | ||
| 37 | each distribution: | ||
| 38 | |||
| 39 | indexer_reports/ | ||
| 40 | error_reports/ | ||
| 41 | success_reports/ | ||
| 42 | |||
| 43 | From the list of reports, C<dpan> creates the right index files, | ||
| 44 | including: | ||
| 45 | |||
| 46 | authors/00whois.xml | ||
| 47 | authors/01mailrc.txt.gz | ||
| 48 | modules/02packages.details.txt.gz | ||
| 49 | modules/03modlist.data.gz | ||
| 50 | |||
| 51 | After everything is done, you can use your new repository as your CPAN | ||
| 52 | source. You can use it as a local directory, serve it through a | ||
| 53 | webserver, or put it behind an FTP server. | ||
| 54 | |||
| 55 | =head2 Deciding on which modules to add | ||
| 56 | |||
| 57 | So far, it's up to you to decide which distributions that you want in | ||
| 58 | your repository, but we'd like to create a tool that can take a single | ||
| 59 | distribution and tell you everything else it needs. | ||
| 60 | |||
| 61 | For a more general solution, you can start with a MiniCPAN which | ||
| 62 | either filters the distributions from the real CPAN or just includes | ||
| 63 | all of them. | ||
| 64 | |||
| 65 | You can keep a separate directory of your private distributions that | ||
| 66 | C<dpan> can merge for you. | ||
| 67 | |||
| 68 | =head2 Running C<dpan> for the first time | ||
| 69 | |||
| 70 | Running C<dpan> for the first time over a large repository can take | ||
| 71 | quite a bit of time. Running it over all of a MiniCPAN, about 20,000 | ||
| 72 | distributions taking up about 1 GB, can take a couple of hours. | ||
| 73 | Fortunately, on subsequent runs C<dpan> only needs to analyze the | ||
| 74 | distributions it hasn't succesfully analyzed yet and the run should be | ||
| 75 | much faster. | ||
| 76 | |||
| 77 | To play with C<dpan>, start with a directory that only has a couple of | ||
| 78 | modules in it. Once you work out how everything works and setup | ||
| 79 | everything to your satisfaction, you can run C<dpan> against a full | ||
| 80 | repository. | ||
| 81 | |||
| 82 | =head2 Running your own CPAN Search | ||
| 83 | |||
| 84 | Once you have your DPAN, you can run C<minicpan-webserver> from | ||
| 85 | C<CPAN::Mini::Webserver>. You'll have a basic website that allows you | ||
| 86 | to search for modules and read documentation just for the | ||
| 87 | distributions in your DPAN. | ||
| 88 | |||
| 89 | =head1 ADVANCED USE | ||
| 90 | |||
| 91 | =head2 Configuring C<dpan> | ||
| 92 | |||
| 93 | C<dpan> can take two different configuration files: one for its setup | ||
| 94 | and one for C<Log::Log4perl>: | ||
| 95 | |||
| 96 | % dpan -f dpan.conf -l dpan.log4perl | ||
| 97 | |||
| 98 | See the L<LOGGING> section for more details about the logging setup. | ||
| 99 | |||
| 100 | The C<dpan> configuration directives are listed in the C<dpan> | ||
| 101 | documentation. The format is a simple, line-oriented list of | ||
| 102 | key-value pairs: | ||
| 103 | |||
| 104 | organize_dists 1 | ||
| 105 | retry_errors 0 | ||
| 106 | merge_dirs my_local_modules/foo/bar | ||
| 107 | |||
| 108 | To see the configuration for any setup, you can use the C<-c> switch: | ||
| 109 | |||
| 110 | % dpan -c | ||
| 111 | alarm 15 | ||
| 112 | author_map | ||
| 113 | collator_class MyCPAN::App::DPAN::Reporter::Minimal | ||
| 114 | copy_bad_dists 0 | ||
| 115 | dispatcher_class MyCPAN::Indexer::Dispatcher::Serial | ||
| 116 | dpan_dir /Users/brian/DEv/mycpan--app--dpan | ||
| 117 | error_report_subdir /Users/brian/DEv/mycpan--app--dpan/indexer_reports/error | ||
| 118 | extra_reports_dir | ||
| 119 | fresh_start 0 | ||
| 120 | i_ignore_errors_at_my_peril 0 | ||
| 121 | ignore_missing_dists 0 | ||
| 122 | ignore_packages main MY MM DB bytes DynaLoader | ||
| 123 | indexer_class MyCPAN::App::DPAN::Indexer | ||
| 124 | indexer_id Joe Example <joe@example.com> | ||
| 125 | interface_class MyCPAN::Indexer::Interface::Text | ||
| 126 | log_file_watch_time 30 | ||
| 127 | organize_dists 1 | ||
| 128 | parallel_jobs 1 | ||
| 129 | pause_full_name DPAN user <CENSORED> | ||
| 130 | pause_id DPAN | ||
| 131 | prefer_bin 0 | ||
| 132 | queue_class MyCPAN::App::DPAN::SkipQueue | ||
| 133 | relative_paths_in_report 1 | ||
| 134 | report_dir /Users/brian/DEv/mycpan--app--dpan/indexer_reports | ||
| 135 | reporter_class MyCPAN::App::DPAN::Reporter::Minimal | ||
| 136 | retry_errors 1 | ||
| 137 | skip_perl 0 | ||
| 138 | success_report_subdir /Users/brian/DEv/mycpan--app--dpan/indexer_reports/success | ||
| 139 | system_id an unnamed system | ||
| 140 | use_real_whois 0 | ||
| 141 | worker_class MyCPAN::Indexer::Worker | ||
| 142 | |||
| 143 | There are some directives that you'll probably want to set right away because | ||
| 144 | they are specific to your setup: | ||
| 145 | |||
| 146 | dpan_dir /path/to/my/dpan/repository | ||
| 147 | indexer_id Joe Example <joe@example.com> | ||
| 148 | pause_full_name DPAN user <CENSORED> | ||
| 149 | pause_id DPAN | ||
| 150 | system_id an unnamed system | ||
| 151 | |||
| 152 | You are probably safe with the remaining defaults which configure | ||
| 153 | C<dpan> for the most common situation. | ||
| 154 | |||
| 155 | =head2 Dealing with indexing failures | ||
| 156 | |||
| 157 | If there's an error, you'll see some error output and C<dpan> will | ||
| 158 | dump the error into a file for that distribution under | ||
| 159 | F<indexer_reports/error_reports/>. | ||
| 160 | |||
| 161 | There are two common reasons for an index failure: either the analysis | ||
| 162 | could not complete in the alloted time (by default 15 seconds) or | ||
| 163 | C<dpan> could not unpack the distribution. Although rare, the next | ||
| 164 | most frequent problem comes from an unexpected distribution structure. | ||
| 165 | |||
| 166 | We're developing a bunch of reports that we can distribute separately | ||
| 167 | so you don't have to do this by hand. If your run into problem | ||
| 168 | distributions, let us know. | ||
| 169 | |||
| 170 | =head3 Time-outs | ||
| 171 | |||
| 172 | If you see the error "Alarm rang", it means the analysis timed-out. | ||
| 173 | You can set a longer time by configuring the alarm time in your | ||
| 174 | configuration file: | ||
| 175 | |||
| 176 | alarm: 120 | ||
| 177 | |||
| 178 | Some distributions can take an extremely long time (more than a couple | ||
| 179 | minutes) to unpack. This time might include the transfer speed over your | ||
| 180 | network if you have to get the file over NFS, etc), the size of the distribution, | ||
| 181 | the speed at which you can write files. | ||
| 182 | |||
| 183 | =head3 Distribution unpacking | ||
| 184 | |||
| 185 | C<MyCPAN::Indexer> relies on C<Archive::Extract> to unpack distributions. | ||
| 186 | C<Archive::Extract> can try a pure Perl solution through C<Archive::Tar> | ||
| 187 | or use an external binary. | ||
| 188 | |||
| 189 | =head3 Can't find modules | ||
| 190 | |||
| 191 | CPAN authors can do almost anything they like with their modules, so | ||
| 192 | C<MyCPAN::Indexer> might have some trouble indexing some modules. The | ||
| 193 | most frequent problems comes from errors unpacking archives. | ||
| 194 | |||
| 195 | Although C<MyCPAN::Indexer> is constantly trying to improve its | ||
| 196 | ability to analyze distributions, DPAN specifically disables | ||
| 197 | C<MyCPAN::Indexer>'s preferred method of running the build file and | ||
| 198 | inspecting F<blib/> to see what showed up. C<dpan> tries not to run | ||
| 199 | any code, so it sometimes can't guess what would have shown up in | ||
| 200 | F<blib/>. It does its best though. | ||
| 201 | |||
| 202 | Aside from improvements in C<MyCPAN::Indexer>'s ability to deal with | ||
| 203 | odd situations, C<dpan> has another way to handle these problematic | ||
| 204 | distributions. You can configure the C<extra_reports> directive so the | ||
| 205 | indexer can use pre-prepared reports in addition to the reports that | ||
| 206 | it generates. These extra reports can be ones that you create by hand | ||
| 207 | with information that you know about the module or reports that you | ||
| 208 | get from a more in-depth index. | ||
| 209 | |||
| 210 | =head2 Adding your local distributions | ||
| 211 | |||
| 212 | You could just dump the private distributions you want to add into | ||
| 213 | the DPAN directory, but you can also copy them in from other | ||
| 214 | directories: | ||
| 215 | |||
| 216 | merge_dirs /repo/foo/bar /repo/baz/quux | ||
| 217 | |||
| 218 | This is quite handy when you are using C<CPAN::Mini>, which tries to | ||
| 219 | remove files it doesn't think belong in the repository. After you | ||
| 220 | update your MiniCPAN, C<dpan> can copy these additional modules into | ||
| 221 | your DPAN repository. | ||
| 222 | |||
| 223 | =head1 USE CASES | ||
| 224 | |||
| 225 | C<dpan> is usually only part of the process to manage your DPAN. The | ||
| 226 | particular process depends on your needs, and there are several ways | ||
| 227 | that you could manage it. | ||
| 228 | |||
| 229 | =head2 A small private repository | ||
| 230 | |||
| 231 | If you prefer a very small repository that contains only the | ||
| 232 | distributions that your application uses, you have a bit of work to | ||
| 233 | do. It's on our to-do list to automatically list and download all of | ||
| 234 | the distributions that a particular application uses, but we're not | ||
| 235 | that far yet. | ||
| 236 | |||
| 237 | ... Magic happens ... | ||
| 238 | |||
| 239 | Assuming the magic that gets you all of the distributions that you | ||
| 240 | need, put all of those distributions in the a single directory and run | ||
| 241 | C<dpan>. | ||
| 242 | |||
| 243 | =head2 Tracking a MiniCPAN | ||
| 244 | |||
| 245 | You can base your DPAN on a MiniCPAN. There are a few steps to go | ||
| 246 | through, so you might create a shell script to handle this for you. | ||
| 247 | |||
| 248 | Update your MiniCPAN: | ||
| 249 | |||
| 250 | % minicpan | ||
| 251 | |||
| 252 | The F<~/.minicpanrc> configuration file should use your DPAN | ||
| 253 | directory as the value for C<local>: | ||
| 254 | |||
| 255 | local: /path/to/your/dpan | ||
| 256 | remote: some CPAN mirror | ||
| 257 | |||
| 258 | The C<minicpan> is going to try to clean up your directory, so your | ||
| 259 | local modules might disappear. That's not a problem. There is | ||
| 260 | currently a bug in C<CPAN::Mini> that will also clean up source | ||
| 261 | control files and other files that might disappear. We're working on | ||
| 262 | that too. Here's a patch to C<CPAN::Mini>: | ||
| 263 | |||
| 264 | http://github.com/briandfoy/cpan-mini/commit/6cc882cc09b2987ce0f3a4f8087ea751feaa88f1 | ||
| 265 | |||
| 266 | You can filter distributions from your MiniCPAN with the hooks that | ||
| 267 | C<CPAN::Mini> provides. See it's documentation for the details. | ||
| 268 | |||
| 269 | Once you update with C<minicpan>, your repository is stale. That's | ||
| 270 | okay. It's time to run C<dpan>: | ||
| 271 | |||
| 272 | % dpan -f dpan.conf | ||
| 273 | |||
| 274 | If you have C<merge_dirs> configured, C<dpan> will pull those | ||
| 275 | distributions and put them into DPAN with the fake CPAN author that | ||
| 276 | you specified in C<pause_id> ("DPAN" by default). If you don't want to | ||
| 277 | merge in this way. you can copy the distributions into your MiniCPAN | ||
| 278 | with rsync or something else. Ensure you keep the originals so | ||
| 279 | C<minicpan> doesn't delete them. | ||
| 280 | |||
| 281 | Once complete, start up C<minicpan_webserver> from | ||
| 282 | C<CPAN::Mini::Webserver>: | ||
| 283 | |||
| 284 | % minicpan_webserver | ||
| 285 | |||
| 286 | C<minicpan_webserver> uses the C<local> value from F<~/.minicpanrc>. | ||
| 287 | |||
| 288 | =head2 Keeping DPAN in source control | ||
| 289 | |||
| 290 | DPAN might be most useful when you keep it in source control. At the | ||
| 291 | end of an indexing run, C<dpan> can commit the changes to source | ||
| 292 | control. There are some adjustments that you need to make, however. | ||
| 293 | |||
| 294 | First, you have to ensure that C<minicpan> won't remove your source | ||
| 295 | control directories. There's a patch for that: | ||
| 296 | |||
| 297 | http://github.com/briandfoy/cpan-mini/commit/6cc882cc09b2987ce0f3a4f8087ea751feaa88f1 | ||
| 298 | |||
| 299 | Next, configure a C<postflight_class> for C<dpan>. Start with the | ||
| 300 | C<MyCPAN::App::DPAN::SVNPostFlight> for an example. At the end of | ||
| 301 | processing, C<dpan> calls the C<run> method in C<postflight_class>. | ||
| 302 | In the example, C<MyCPAN::App::DPAN::SVNPostFlight> figures out what | ||
| 303 | to remove or add to your subversion repository and commits the | ||
| 304 | result. It's more fully explained in the example, which is intended as | ||
| 305 | a starting point for your own process. | ||
| 306 | |||
| 307 | =head1 LOGGING | ||
| 308 | |||
| 309 | C<dpan> uses C<Log::Log4perl>, which you can configure any way that you | ||
| 310 | like. Each component has its own logging category: | ||
| 311 | |||
| 312 | Coordinator | ||
| 313 | Queue | ||
| 314 | Dispatcher | ||
| 315 | Worker | ||
| 316 | Reporter | ||
| 317 | Collator | ||
| 318 | PostFlight | ||
| 319 | |||
| 320 | For more details on the components, see C<MyCPAN::Indexer::Tutorial>. There | ||
| 321 | are some example Log4perl configurations in the C<MyCPAN::Indexer> and | ||
| 322 | C<MyCPAN::App::DPAN> distributions. | ||
| 323 | |||
| 324 | =head1 GETTING MORE HELP | ||
| 325 | |||
| 326 | If you have any other questions, don't hesitate to ask. If you need help | ||
| 327 | setting up a DPAN, we can also arrange for private help. | ||
| 328 | |||
| 329 | =head1 SEE ALSO | ||
| 330 | |||
| 331 | MyCPAN::Indexer::Tutorial, dpan | ||
| 332 | |||
| 333 | =head1 SOURCE AVAILABILITY | ||
| 334 | |||
| 335 | This code is in Github: | ||
| 336 | |||
| 337 | git://github.com/briandfoy/mycpan-app-dpan.git | ||
| 338 | |||
| 339 | =head1 AUTHOR | ||
| 340 | |||
| 341 | brian d foy, C<< <bdfoy@cpan.org> >> | ||
| 342 | |||
| 343 | =head1 COPYRIGHT AND LICENSE | ||
| 344 | |||
| 345 | Copyright © 2010-2018, brian d foy <bdfoy@cpan.org>. All rights reserved. | ||
| 346 | |||
| 347 | You may redistribute this under the terms of the Artistic License 2.0. | ||
| 348 | |||
| 349 | =cut | ||
| 350 | |||
| 351 | 1; |