File Coverage

blib/lib/AnyEvent/Fork.pm
Criterion Covered Total %
statement 86 110 78.1
branch 18 30 60.0
condition 6 18 33.3
subroutine 15 20 75.0
pod 11 11 100.0
total 136 189 71.9


line stmt bran cond sub pod time code
1             =head1 NAME
2              
3             AnyEvent::Fork - everything you wanted to use fork() for, but couldn't
4              
5             =head1 SYNOPSIS
6              
7             use AnyEvent::Fork;
8              
9             AnyEvent::Fork
10             ->new
11             ->require ("MyModule")
12             ->run ("MyModule::server", my $cv = AE::cv);
13              
14             my $fh = $cv->recv;
15              
16             =head1 DESCRIPTION
17              
18             This module allows you to create new processes, without actually forking
19             them from your current process (avoiding the problems of forking), but
20             preserving most of the advantages of fork.
21              
22             It can be used to create new worker processes or new independent
23             subprocesses for short- and long-running jobs, process pools (e.g. for use
24             in pre-forked servers) but also to spawn new external processes (such as
25             CGI scripts from a web server), which can be faster (and more well behaved)
26             than using fork+exec in big processes.
27              
28             Special care has been taken to make this module useful from other modules,
29             while still supporting specialised environments such as L
30             or L.
31              
32             =head2 WHAT THIS MODULE IS NOT
33              
34             This module only creates processes and lets you pass file handles and
35             strings to it, and run perl code. It does not implement any kind of RPC -
36             there is no back channel from the process back to you, and there is no RPC
37             or message passing going on.
38              
39             If you need some form of RPC, you could use the L
40             companion module, which adds simple RPC/job queueing to a process created
41             by this module.
42              
43             And if you need some automatic process pool management on top of
44             L, you can look at the L
45             companion module.
46              
47             Or you can implement it yourself in whatever way you like: use some
48             message-passing module such as L, some pipe such as
49             L, use L on both sides to send
50             e.g. JSON or Storable messages, and so on.
51              
52             =head2 COMPARISON TO OTHER MODULES
53              
54             There is an abundance of modules on CPAN that do "something fork", such as
55             L, L, L
56             or L. There are modules that implement their own
57             process management, such as L.
58              
59             The problems that all these modules try to solve are real, however, none
60             of them (from what I have seen) tackle the very real problems of unwanted
61             memory sharing, efficiency or not being able to use event processing, GUI
62             toolkits or similar modules in the processes they create.
63              
64             This module doesn't try to replace any of them - instead it tries to solve
65             the problem of creating processes with a minimum of fuss and overhead (and
66             also luxury). Ideally, most of these would use AnyEvent::Fork internally,
67             except they were written before AnyEvent:Fork was available, so obviously
68             had to roll their own.
69              
70             =head2 PROBLEM STATEMENT
71              
72             There are two traditional ways to implement parallel processing on UNIX
73             like operating systems - fork and process, and fork+exec and process. They
74             have different advantages and disadvantages that I describe below,
75             together with how this module tries to mitigate the disadvantages.
76              
77             =over 4
78              
79             =item Forking from a big process can be very slow.
80              
81             A 5GB process needs 0.05s to fork on my 3.6GHz amd64 GNU/Linux box. This
82             overhead is often shared with exec (because you have to fork first), but
83             in some circumstances (e.g. when vfork is used), fork+exec can be much
84             faster.
85              
86             This module can help here by telling a small(er) helper process to fork,
87             which is faster then forking the main process, and also uses vfork where
88             possible. This gives the speed of vfork, with the flexibility of fork.
89              
90             =item Forking usually creates a copy-on-write copy of the parent
91             process.
92              
93             For example, modules or data files that are loaded will not use additional
94             memory after a fork. Exec'ing a new process, in contrast, means modules
95             and data files might need to be loaded again, at extra CPU and memory
96             cost.
97              
98             But when forking, you still create a copy of your data structures - if
99             the program frees them and replaces them by new data, the child processes
100             will retain the old version even if it isn't used, which can suddenly and
101             unexpectedly increase memory usage when freeing memory.
102              
103             For example, L is an image viewer optimised for large
104             directories (millions of pictures). It also forks subprocesses for
105             thumbnail generation, which inherit the data structure that stores all
106             file information. If the user changes the directory, it gets freed in
107             the main process, leaving a copy in the thumbnailer processes. This can
108             lead to many times the memory usage that would actually be required. The
109             solution is to fork early (and being unable to dynamically generate more
110             subprocesses or do this from a module)... or to use L.
111              
112             There is a trade-off between more sharing with fork (which can be good or
113             bad), and no sharing with exec.
114              
115             This module allows the main program to do a controlled fork, and allows
116             modules to exec processes safely at any time. When creating a custom
117             process pool you can take advantage of data sharing via fork without
118             risking to share large dynamic data structures that will blow up child
119             memory usage.
120              
121             In other words, this module puts you into control over what is being
122             shared and what isn't, at all times.
123              
124             =item Exec'ing a new perl process might be difficult.
125              
126             For example, it is not easy to find the correct path to the perl
127             interpreter - C<$^X> might not be a perl interpreter at all. Worse, there
128             might not even be a perl binary installed on the system.
129              
130             This module tries hard to identify the correct path to the perl
131             interpreter. With a cooperative main program, exec'ing the interpreter
132             might not even be necessary, but even without help from the main program,
133             it will still work when used from a module.
134              
135             =item Exec'ing a new perl process might be slow, as all necessary modules
136             have to be loaded from disk again, with no guarantees of success.
137              
138             Long running processes might run into problems when perl is upgraded
139             and modules are no longer loadable because they refer to a different
140             perl version, or parts of a distribution are newer than the ones already
141             loaded.
142              
143             This module supports creating pre-initialised perl processes to be used as
144             a template for new processes at a later time, e.g. for use in a process
145             pool.
146              
147             =item Forking might be impossible when a program is running.
148              
149             For example, POSIX makes it almost impossible to fork from a
150             multi-threaded program while doing anything useful in the child - in
151             fact, if your perl program uses POSIX threads (even indirectly via
152             e.g. L or L), you cannot call fork on the perl level
153             anymore without risking memory corruption or worse on a number of
154             operating systems.
155              
156             This module can safely fork helper processes at any time, by calling
157             fork+exec in C, in a POSIX-compatible way (via L).
158              
159             =item Parallel processing with fork might be inconvenient or difficult
160             to implement. Modules might not work in both parent and child.
161              
162             For example, when a program uses an event loop and creates watchers it
163             becomes very hard to use the event loop from a child program, as the
164             watchers already exist but are only meaningful in the parent. Worse, a
165             module might want to use such a module, not knowing whether another module
166             or the main program also does, leading to problems.
167              
168             Apart from event loops, graphical toolkits also commonly fall into the
169             "unsafe module" category, or just about anything that communicates with
170             the external world, such as network libraries and file I/O modules, which
171             usually don't like being copied and then allowed to continue in two
172             processes.
173              
174             With this module only the main program is allowed to create new processes
175             by forking (because only the main program can know when it is still safe
176             to do so) - all other processes are created via fork+exec, which makes it
177             possible to use modules such as event loops or window interfaces safely.
178              
179             =back
180              
181             =head1 EXAMPLES
182              
183             This is where the wall of text ends and code speaks.
184              
185             =head2 Create a single new process, tell it to run your worker function.
186              
187             AnyEvent::Fork
188             ->new
189             ->require ("MyModule")
190             ->run ("MyModule::worker, sub {
191             my ($master_filehandle) = @_;
192              
193             # now $master_filehandle is connected to the
194             # $slave_filehandle in the new process.
195             });
196              
197             C might look like this:
198              
199             package MyModule;
200              
201             sub worker {
202             my ($slave_filehandle) = @_;
203              
204             # now $slave_filehandle is connected to the $master_filehandle
205             # in the original process. have fun!
206             }
207              
208             =head2 Create a pool of server processes all accepting on the same socket.
209              
210             # create listener socket
211             my $listener = ...;
212              
213             # create a pool template, initialise it and give it the socket
214             my $pool = AnyEvent::Fork
215             ->new
216             ->require ("Some::Stuff", "My::Server")
217             ->send_fh ($listener);
218              
219             # now create 10 identical workers
220             for my $id (1..10) {
221             $pool
222             ->fork
223             ->send_arg ($id)
224             ->run ("My::Server::run");
225             }
226              
227             # now do other things - maybe use the filehandle provided by run
228             # to wait for the processes to die. or whatever.
229              
230             C might look like this:
231              
232             package My::Server;
233              
234             sub run {
235             my ($slave, $listener, $id) = @_;
236              
237             close $slave; # we do not use the socket, so close it to save resources
238              
239             # we could go ballistic and use e.g. AnyEvent here, or IO::AIO,
240             # or anything we usually couldn't do in a process forked normally.
241             while (my $socket = $listener->accept) {
242             # do sth. with new socket
243             }
244             }
245              
246             =head2 use AnyEvent::Fork as a faster fork+exec
247              
248             This runs C, with standard output redirected to F
249             and standard error redirected to the communications socket. It is usually
250             faster than fork+exec, but still lets you prepare the environment.
251              
252             open my $output, ">/tmp/log" or die "$!";
253              
254             AnyEvent::Fork
255             ->new
256             ->eval ('
257             # compile a helper function for later use
258             sub run {
259             my ($fh, $output, @cmd) = @_;
260              
261             # perl will clear close-on-exec on STDOUT/STDERR
262             open STDOUT, ">&", $output or die;
263             open STDERR, ">&", $fh or die;
264              
265             exec @cmd;
266             }
267             ')
268             ->send_fh ($output)
269             ->send_arg ("/bin/echo", "hi")
270             ->run ("run", my $cv = AE::cv);
271              
272             my $stderr = $cv->recv;
273              
274             =head2 For stingy users: put the worker code into a C section.
275              
276             When you want to be stingy with files, you can put your code into the
277             C section of your module (or program):
278              
279             use AnyEvent::Fork;
280              
281             AnyEvent::Fork
282             ->new
283             ->eval (do { local $/; })
284             ->run ("doit", sub { ... });
285              
286             __DATA__
287              
288             sub doit {
289             ... do something!
290             }
291              
292             =head2 For stingy standalone programs: do not rely on external files at
293             all.
294              
295             For single-file scripts it can be inconvenient to rely on external
296             files - even when using a C section, you still need to C an
297             external perl interpreter, which might not be available when using
298             L, L or L for example.
299              
300             Two modules help here - L forks a template process
301             for all further calls to C, and L
302             forks the main program as a template process.
303              
304             Here is how your main program should look like:
305              
306             #! perl
307              
308             # optional, as the very first thing.
309             # in case modules want to create their own processes.
310             use AnyEvent::Fork::Early;
311              
312             # next, load all modules you need in your template process
313             use Example::My::Module
314             use Example::Whatever;
315              
316             # next, put your run function definition and anything else you
317             # need, but do not use code outside of BEGIN blocks.
318             sub worker_run {
319             my ($fh, @args) = @_;
320             ...
321             }
322              
323             # now preserve everything so far as AnyEvent::Fork object
324             # in $TEMPLATE.
325             use AnyEvent::Fork::Template;
326              
327             # do not put code outside of BEGIN blocks until here
328              
329             # now use the $TEMPLATE process in any way you like
330              
331             # for example: create 10 worker processes
332             my @worker;
333             my $cv = AE::cv;
334             for (1..10) {
335             $cv->begin;
336             $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
337             push @worker, shift;
338             $cv->end;
339             });
340             }
341             $cv->recv;
342              
343             =head1 CONCEPTS
344              
345             This module can create new processes either by executing a new perl
346             process, or by forking from an existing "template" process.
347              
348             All these processes are called "child processes" (whether they are direct
349             children or not), while the process that manages them is called the
350             "parent process".
351              
352             Each such process comes with its own file handle that can be used to
353             communicate with it (it's actually a socket - one end in the new process,
354             one end in the main process), and among the things you can do in it are
355             load modules, fork new processes, send file handles to it, and execute
356             functions.
357              
358             There are multiple ways to create additional processes to execute some
359             jobs:
360              
361             =over 4
362              
363             =item fork a new process from the "default" template process, load code,
364             run it
365              
366             This module has a "default" template process which it executes when it is
367             needed the first time. Forking from this process shares the memory used
368             for the perl interpreter with the new process, but loading modules takes
369             time, and the memory is not shared with anything else.
370              
371             This is ideal for when you only need one extra process of a kind, with the
372             option of starting and stopping it on demand.
373              
374             Example:
375              
376             AnyEvent::Fork
377             ->new
378             ->require ("Some::Module")
379             ->run ("Some::Module::run", sub {
380             my ($fork_fh) = @_;
381             });
382              
383             =item fork a new template process, load code, then fork processes off of
384             it and run the code
385              
386             When you need to have a bunch of processes that all execute the same (or
387             very similar) tasks, then a good way is to create a new template process
388             for them, loading all the modules you need, and then create your worker
389             processes from this new template process.
390              
391             This way, all code (and data structures) that can be shared (e.g. the
392             modules you loaded) is shared between the processes, and each new process
393             consumes relatively little memory of its own.
394              
395             The disadvantage of this approach is that you need to create a template
396             process for the sole purpose of forking new processes from it, but if you
397             only need a fixed number of processes you can create them, and then destroy
398             the template process.
399              
400             Example:
401              
402             my $template = AnyEvent::Fork->new->require ("Some::Module");
403            
404             for (1..10) {
405             $template->fork->run ("Some::Module::run", sub {
406             my ($fork_fh) = @_;
407             });
408             }
409              
410             # at this point, you can keep $template around to fork new processes
411             # later, or you can destroy it, which causes it to vanish.
412              
413             =item execute a new perl interpreter, load some code, run it
414              
415             This is relatively slow, and doesn't allow you to share memory between
416             multiple processes.
417              
418             The only advantage is that you don't have to have a template process
419             hanging around all the time to fork off some new processes, which might be
420             an advantage when there are long time spans where no extra processes are
421             needed.
422              
423             Example:
424              
425             AnyEvent::Fork
426             ->new_exec
427             ->require ("Some::Module")
428             ->run ("Some::Module::run", sub {
429             my ($fork_fh) = @_;
430             });
431              
432             =back
433              
434             =head1 THE C CLASS
435              
436             This module exports nothing, and only implements a single class -
437             C.
438              
439             There are two class constructors that both create new processes - C
440             and C. The C method creates a new process by forking an
441             existing one and could be considered a third constructor.
442              
443             Most of the remaining methods deal with preparing the new process, by
444             loading code, evaluating code and sending data to the new process. They
445             usually return the process object, so you can chain method calls.
446              
447             If a process object is destroyed before calling its C method, then
448             the process simply exits. After C is called, all responsibility is
449             passed to the specified function.
450              
451             As long as there is any outstanding work to be done, process objects
452             resist being destroyed, so there is no reason to store them unless you
453             need them later - configure and forget works just fine.
454              
455             =over 4
456              
457             =cut
458              
459             package AnyEvent::Fork;
460              
461 8     8   12215 use common::sense;
  8         88  
  8         52  
462              
463 8     8   6794 use Errno ();
  8         13276  
  8         302  
464              
465 8     8   12668 use AnyEvent;
  8         64343  
  8         411  
466 8     8   8698 use AnyEvent::Util ();
  8         120146  
  8         377  
467              
468 8     8   7011 use IO::FDPass;
  8         3641  
  8         21636  
469              
470             our $VERSION = 1.31;
471              
472             # the early fork template process
473             our $EARLY;
474              
475             # the empty template process
476             our $TEMPLATE;
477              
478             sub QUEUE() { 0 }
479             sub FH() { 1 }
480             sub WW() { 2 }
481             sub PID() { 3 }
482             sub CB() { 4 }
483              
484             sub _new {
485 9     9   70 my ($self, $fh, $pid) = @_;
486              
487 9         125 AnyEvent::Util::fh_nonblocking $fh, 1;
488              
489 9         149 $self = bless [
490             [], # write queue - strings or fd's
491             $fh,
492             undef, # AE watcher
493             $pid,
494             ], $self;
495              
496 9         187 $self
497             }
498              
499             sub _cmd {
500 20     20   43 my $self = shift;
501              
502             # ideally, we would want to use "a (w/a)*" as format string, but perl
503             # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
504             # it.
505 20         29 push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
  20         183  
506              
507             $self->[WW] ||= AE::io $self->[FH], 1, sub {
508             do {
509             # send the next "thing" in the queue - either a reference to an fh,
510             # or a plain string.
511              
512 30 100       97 if (ref $self->[QUEUE][0]) {
513             # send fh
514 10 50       72 unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
  10         894  
515 0 0 0     0 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
516 0         0 undef $self->[WW];
517 0         0 die "AnyEvent::Fork: file descriptor send failure: $!";
518             }
519              
520 10         23 shift @{ $self->[QUEUE] };
  10         90  
521              
522             } else {
523             # send string
524 20         7454 my $len = syswrite $self->[FH], $self->[QUEUE][0];
525              
526 20 50       73 unless ($len) {
527 0 0 0     0 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
528 0         0 undef $self->[WW];
529 0         0 die "AnyEvent::Fork: command write failure: $!";
530             }
531              
532 20         63 substr $self->[QUEUE][0], 0, $len, "";
533 20 50       69 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
  20         80  
534             }
535 8     8   1465 } while @{ $self->[QUEUE] };
  30         99  
536              
537             # everything written
538 8         62 undef $self->[WW];
539              
540             # invoke run callback, if any
541 8 100       247 if ($self->[CB]) {
542 2         17 $self->[CB]->($self->[FH]);
543 2         106 @$self = ();
544             }
545 20   66     342 };
546              
547             () # make sure we don't leak the watcher
548 20         40948 }
549              
550             # fork template from current process, used by AnyEvent::Fork::Early/Template
551             sub _new_fork {
552 5     5   30 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
553 5         337 my $parent = $$;
554              
555 5         12497 my $pid = fork;
556              
557 5 100       503 if ($pid eq 0) {
    50          
558 3         8852 require AnyEvent::Fork::Serve;
559 3         14 $AnyEvent::Fork::Serve::OWNER = $parent;
560 3         50 close $fh;
561 3         231 $0 = "$parent AnyEvent::Fork/exec";
562 3         19 AnyEvent::Fork::Serve::serve ($slave);
563 0         0 exit 0;
564             } elsif (!$pid) {
565 0         0 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
566             }
567              
568 2         145 AnyEvent::Fork->_new ($fh, $pid)
569             }
570              
571             =item my $proc = new AnyEvent::Fork
572              
573             Create a new "empty" perl interpreter process and returns its process
574             object for further manipulation.
575              
576             The new process is forked from a template process that is kept around
577             for this purpose. When it doesn't exist yet, it is created by a call to
578             C first and then stays around for future calls.
579              
580             =cut
581              
582             sub new {
583 4     4 1 1507 my $class = shift;
584              
585 4   66     31 $TEMPLATE ||= $class->new_exec;
586 4         19 $TEMPLATE->fork
587             }
588              
589             =item $new_proc = $proc->fork
590              
591             Forks C<$proc>, creating a new process, and returns the process object
592             of the new process.
593              
594             If any of the C functions have been called before fork, then they
595             will be cloned in the child. For example, in a pre-forked server, you
596             might C the listening socket into the template process, and then
597             keep calling C and C.
598              
599             =cut
600              
601             sub fork {
602 5     5 1 12 my ($self) = @_;
603              
604 5         28 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
605              
606 5         293 $self->send_fh ($slave);
607 5         22 $self->_cmd ("f");
608              
609 5         36 AnyEvent::Fork->_new ($fh)
610             }
611              
612             =item my $proc = new_exec AnyEvent::Fork
613              
614             Create a new "empty" perl interpreter process and returns its process
615             object for further manipulation.
616              
617             Unlike the C method, this method I spawns a new perl process
618             (except in some cases, see L for details). This
619             reduces the amount of memory sharing that is possible, and is also slower.
620              
621             You should use C whenever possible, except when having a template
622             process around is unacceptable.
623              
624             The path to the perl interpreter is divined using various methods - first
625             C<$^X> is investigated to see if the path ends with something that looks
626             as if it were the perl interpreter. Failing this, the module falls back to
627             using C<$Config::Config{perlpath}>.
628              
629             The path to perl can also be overriden by setting the global variable
630             C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
631             invocations.
632              
633             =cut
634              
635             our $PERL;
636              
637             sub new_exec {
638 3     3 1 73 my ($self) = @_;
639              
640 3 100       28 return $EARLY->fork
641             if $EARLY;
642              
643 2 50       14 unless (defined $PERL) {
644             # first find path of perl
645 2         6 my $perl = $^X;
646              
647             # first we try $^X, but the path must be absolute (always on win32), and end in sth.
648             # that looks like perl. this obviously only works for posix and win32
649 2 50 33     58 unless (
      33        
650             ($^O eq "MSWin32" || $perl =~ m%^/%)
651             && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
652             ) {
653             # if it doesn't look perlish enough, try Config
654 0         0 require Config;
655 0         0 $perl = $Config::Config{perlpath};
656 0         0 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
657             }
658              
659 2         5 $PERL = $perl;
660             }
661              
662 2         2310 require Proc::FastSpawn;
663              
664 2         1121 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
665 2         154 Proc::FastSpawn::fd_inherit (fileno $slave);
666              
667             # new fh's should always be set cloexec (due to $^F),
668             # but hey, not on win32, so we always clear the inherit flag.
669 2         9 Proc::FastSpawn::fd_inherit (fileno $fh, 0);
670              
671             # quick. also doesn't work in win32. of course. what did you expect
672             #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
673 2         69 my %env = %ENV;
674 2 50       36 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
675              
676 2 50       955 my $pid = Proc::FastSpawn::spawn (
677             $PERL,
678             [$PERL, "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
679             [map "$_=$env{$_}", keys %env],
680             ) or die "unable to spawn AnyEvent::Fork server: $!";
681              
682 2         40 $self->_new ($fh, $pid)
683             }
684              
685             =item $pid = $proc->pid
686              
687             Returns the process id of the process I
688             process running AnyEvent::Fork>, and C otherwise. As a general
689             rule (that you cannot rely upon), processes created via C,
690             L or L are direct
691             children, while all other processes are not.
692              
693             Or in other words, you do not normally have to take care of zombies for
694             processes created via C, but when in doubt, or zombies are a problem,
695             you need to check whether a process is a diretc child by calling this
696             method, and possibly creating a child watcher or reap it manually.
697              
698             =cut
699              
700             sub pid {
701 0     0 1 0 $_[0][PID]
702             }
703              
704             =item $proc = $proc->eval ($perlcode, @args)
705              
706             Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
707             the strings specified by C<@args>, in the "main" package.
708              
709             This call is meant to do any custom initialisation that might be required
710             (for example, the C method uses it). It's not supposed to be used
711             to completely take over the process, use C for that.
712              
713             The code will usually be executed after this call returns, and there is no
714             way to pass anything back to the calling process. Any evaluation errors
715             will be reported to stderr and cause the process to exit.
716              
717             If you want to execute some code (that isn't in a module) to take over the
718             process, you should compile a function via C first, and then call
719             it via C. This also gives you access to any arguments passed via the
720             C methods, such as file handles. See the L
721             a faster fork+exec> example to see it in action.
722              
723             Returns the process object for easy chaining of method calls.
724              
725             It's common to want to call an iniitalisation function with some
726             arguments. Make sure you actually pass C<@_> to that function (for example
727             by using C<&name> syntax), and do not just specify a function name:
728              
729             $proc->eval ('&MyModule::init', $string1, $string2);
730              
731             =cut
732              
733             sub eval {
734 5     5 1 908 my ($self, $code, @args) = @_;
735              
736 5         72 $self->_cmd (e => pack "(w/a*)*", $code, @args);
737              
738 5         79 $self
739             }
740              
741             =item $proc = $proc->require ($module, ...)
742              
743             Tries to load the given module(s) into the process
744              
745             Returns the process object for easy chaining of method calls.
746              
747             =cut
748              
749             sub require {
750 0     0 1 0 my ($self, @modules) = @_;
751              
752 0         0 s%::%/%g for @modules;
753 0         0 $self->eval ('require "$_.pm" for @_', @modules);
754              
755 0         0 $self
756             }
757              
758             =item $proc = $proc->send_fh ($handle, ...)
759              
760             Send one or more file handles (I file descriptors) to the process,
761             to prepare a call to C.
762              
763             The process object keeps a reference to the handles until they have
764             been passed over to the process, so you must not explicitly close the
765             handles. This is most easily accomplished by simply not storing the file
766             handles anywhere after passing them to this method - when AnyEvent::Fork
767             is finished using them, perl will automatically close them.
768              
769             Returns the process object for easy chaining of method calls.
770              
771             Example: pass a file handle to a process, and release it without
772             closing. It will be closed automatically when it is no longer used.
773              
774             $proc->send_fh ($my_fh);
775             undef $my_fh; # free the reference if you want, but DO NOT CLOSE IT
776              
777             =cut
778              
779             sub send_fh {
780 10     10 1 2163 my ($self, @fh) = @_;
781              
782 10         31 for my $fh (@fh) {
783 10         35 $self->_cmd ("h");
784 10         42 push @{ $self->[QUEUE] }, \$fh;
  10         45  
785             }
786              
787             $self
788 10         20 }
789              
790             =item $proc = $proc->send_arg ($string, ...)
791              
792             Send one or more argument strings to the process, to prepare a call to
793             C. The strings can be any octet strings.
794              
795             The protocol is optimised to pass a moderate number of relatively short
796             strings - while you can pass up to 4GB of data in one go, this is more
797             meant to pass some ID information or other startup info, not big chunks of
798             data.
799              
800             Returns the process object for easy chaining of method calls.
801              
802             =cut
803              
804             sub send_arg {
805 0     0 1 0 my ($self, @arg) = @_;
806              
807 0         0 $self->_cmd (a => pack "(w/a*)*", @arg);
808              
809 0         0 $self
810             }
811              
812             =item $proc->run ($func, $cb->($fh))
813              
814             Enter the function specified by the function name in C<$func> in the
815             process. The function is called with the communication socket as first
816             argument, followed by all file handles and string arguments sent earlier
817             via C and C methods, in the order they were called.
818              
819             The process object becomes unusable on return from this function - any
820             further method calls result in undefined behaviour.
821              
822             The function name should be fully qualified, but if it isn't, it will be
823             looked up in the C
package.
824              
825             If the called function returns, doesn't exist, or any error occurs, the
826             process exits.
827              
828             Preparing the process is done in the background - when all commands have
829             been sent, the callback is invoked with the local communications socket
830             as argument. At this point you can start using the socket in any way you
831             like.
832              
833             If the communication socket isn't used, it should be closed on both sides,
834             to save on kernel memory.
835              
836             The socket is non-blocking in the parent, and blocking in the newly
837             created process. The close-on-exec flag is set in both.
838              
839             Even if not used otherwise, the socket can be a good indicator for the
840             existence of the process - if the other process exits, you get a readable
841             event on it, because exiting the process closes the socket (if it didn't
842             create any children using fork).
843              
844             =over 4
845              
846             =item Compatibility to L
847              
848             If you want to write code that works with both this module and
849             L, you need to write your code so that it assumes
850             there are two file handles for communications, which might not be unix
851             domain sockets. The C function should start like this:
852              
853             sub run {
854             my ($rfh, @args) = @_; # @args is your normal arguments
855             my $wfh = fileno $rfh ? $rfh : *STDOUT;
856              
857             # now use $rfh for reading and $wfh for writing
858             }
859              
860             This checks whether the passed file handle is, in fact, the process
861             C handle. If it is, then the function was invoked visa
862             L, so STDIN should be used for reading and
863             C should be used for writing.
864              
865             In all other cases, the function was called via this module, and there is
866             only one file handle that should be sued for reading and writing.
867              
868             =back
869              
870             Example: create a template for a process pool, pass a few strings, some
871             file handles, then fork, pass one more string, and run some code.
872              
873             my $pool = AnyEvent::Fork
874             ->new
875             ->send_arg ("str1", "str2")
876             ->send_fh ($fh1, $fh2);
877              
878             for (1..2) {
879             $pool
880             ->fork
881             ->send_arg ("str3")
882             ->run ("Some::function", sub {
883             my ($fh) = @_;
884              
885             # fh is nonblocking, but we trust that the OS can accept these
886             # few octets anyway.
887             syswrite $fh, "hi #$_\n";
888              
889             # $fh is being closed here, as we don't store it anywhere
890             });
891             }
892              
893             # Some::function might look like this - all parameters passed before fork
894             # and after will be passed, in order, after the communications socket.
895             sub Some::function {
896             my ($fh, $str1, $str2, $fh1, $fh2, $str3) = @_;
897              
898             print scalar <$fh>; # prints "hi #1\n" and "hi #2\n" in any order
899             }
900              
901             =cut
902              
903             sub run {
904 0     0 1 0 my ($self, $func, $cb) = @_;
905              
906 0         0 $self->[CB] = $cb;
907 0         0 $self->_cmd (r => $func);
908             }
909              
910             =back
911              
912              
913             =head2 CHILD PROCESS INTERFACE
914              
915             This module has a limited API for use in child processes.
916              
917             =over 4
918              
919             =item @args = AnyEvent::Fork::Serve::run_args
920              
921             This function, which only exists before the C method is called,
922             returns the arguments that would be passed to the run function, and clears
923             them.
924              
925             This is mainly useful to get any file handles passed via C, but
926             works for any arguments passed via C<< send_I >> methods.
927              
928             =back
929              
930              
931             =head2 EXPERIMENTAL METHODS
932              
933             These methods might go away completely or change behaviour, at any time.
934              
935             =over 4
936              
937             =item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
938              
939             Flushes all commands out to the process and then calls the callback with
940             the communications socket.
941              
942             The process object becomes unusable on return from this function - any
943             further method calls result in undefined behaviour.
944              
945             The point of this method is to give you a file handle that you can pass
946             to another process. In that other process, you can call C
947             AnyEvent::Fork $fh> to create a new C object from it,
948             thereby effectively passing a fork object to another process.
949              
950             =cut
951              
952             sub to_fh {
953 3     3 1 1202 my ($self, $cb) = @_;
954              
955 3         10 $self->[CB] = $cb;
956              
957 3 100       14 unless ($self->[WW]) {
958 1         8 $self->[CB]->($self->[FH]);
959 1         20 @$self = ();
960             }
961             }
962              
963             =item new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
964              
965             Takes a file handle originally rceeived by the C method and creates
966             a new C object. The child process itself will not change in
967             any way, i.e. it will keep all the modifications done to it before calling
968             C.
969              
970             The new object is very much like the original object, except that the
971             C method will return C even if the process is a direct child.
972              
973             =cut
974              
975             sub new_from_fh {
976 0     0 1   my ($class, $fh) = @_;
977              
978 0           $class->_new ($fh)
979             }
980              
981             =back
982              
983             =head1 PERFORMANCE
984              
985             Now for some unscientific benchmark numbers (all done on an amd64
986             GNU/Linux box). These are intended to give you an idea of the relative
987             performance you can expect, they are not meant to be absolute performance
988             numbers.
989              
990             OK, so, I ran a simple benchmark that creates a socket pair, forks, calls
991             exit in the child and waits for the socket to close in the parent. I did
992             load AnyEvent, EV and AnyEvent::Fork, for a total process size of 5100kB.
993              
994             2079 new processes per second, using manual socketpair + fork
995              
996             Then I did the same thing, but instead of calling fork, I called
997             AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
998             socket from the child to close on exit. This does the same thing as manual
999             socket pair + fork, except that what is forked is the template process
1000             (2440kB), and the socket needs to be passed to the server at the other end
1001             of the socket first.
1002              
1003             2307 new processes per second, using AnyEvent::Fork->new
1004              
1005             And finally, using C instead C, using vforks+execs to exec
1006             a new perl interpreter and compile the small server each time, I get:
1007              
1008             479 vfork+execs per second, using AnyEvent::Fork->new_exec
1009              
1010             So how can C<< AnyEvent->new >> be faster than a standard fork, even
1011             though it uses the same operations, but adds a lot of overhead?
1012              
1013             The difference is simply the process size: forking the 5MB process takes
1014             so much longer than forking the 2.5MB template process that the extra
1015             overhead is canceled out.
1016              
1017             If the benchmark process grows, the normal fork becomes even slower:
1018              
1019             1340 new processes, manual fork of a 20MB process
1020             731 new processes, manual fork of a 200MB process
1021             235 new processes, manual fork of a 2000MB process
1022              
1023             What that means (to me) is that I can use this module without having a bad
1024             conscience because of the extra overhead required to start new processes.
1025              
1026             =head1 TYPICAL PROBLEMS
1027              
1028             This section lists typical problems that remain. I hope by recognising
1029             them, most can be avoided.
1030              
1031             =over 4
1032              
1033             =item leaked file descriptors for exec'ed processes
1034              
1035             POSIX systems inherit file descriptors by default when exec'ing a new
1036             process. While perl itself laudably sets the close-on-exec flags on new
1037             file handles, most C libraries don't care, and even if all cared, it's
1038             often not possible to set the flag in a race-free manner.
1039              
1040             That means some file descriptors can leak through. And since it isn't
1041             possible to know which file descriptors are "good" and "necessary" (or
1042             even to know which file descriptors are open), there is no good way to
1043             close the ones that might harm.
1044              
1045             As an example of what "harm" can be done consider a web server that
1046             accepts connections and afterwards some module uses AnyEvent::Fork for the
1047             first time, causing it to fork and exec a new process, which might inherit
1048             the network socket. When the server closes the socket, it is still open
1049             in the child (which doesn't even know that) and the client might conclude
1050             that the connection is still fine.
1051              
1052             For the main program, there are multiple remedies available -
1053             L is one, creating a process early and not using
1054             C is another, as in both cases, the first process can be exec'ed
1055             well before many random file descriptors are open.
1056              
1057             In general, the solution for these kind of problems is to fix the
1058             libraries or the code that leaks those file descriptors.
1059              
1060             Fortunately, most of these leaked descriptors do no harm, other than
1061             sitting on some resources.
1062              
1063             =item leaked file descriptors for fork'ed processes
1064              
1065             Normally, L does start new processes by exec'ing them,
1066             which closes file descriptors not marked for being inherited.
1067              
1068             However, L and L offer
1069             a way to create these processes by forking, and this leaks more file
1070             descriptors than exec'ing them, as there is no way to mark descriptors as
1071             "close on fork".
1072              
1073             An example would be modules like L, L or L. Both create
1074             pipes for internal uses, and L might open a connection to the X
1075             server. L and L can deal with fork, but Gtk2 might have
1076             trouble with a fork.
1077              
1078             The solution is to either not load these modules before use'ing
1079             L or L, or to delay
1080             initialising them, for example, by calling C manually.
1081              
1082             =item exiting calls object destructors
1083              
1084             This only applies to users of L and
1085             L, or when initialising code creates objects
1086             that reference external resources.
1087              
1088             When a process created by AnyEvent::Fork exits, it might do so by calling
1089             exit, or simply letting perl reach the end of the program. At which point
1090             Perl runs all destructors.
1091              
1092             Not all destructors are fork-safe - for example, an object that represents
1093             the connection to an X display might tell the X server to free resources,
1094             which is inconvenient when the "real" object in the parent still needs to
1095             use them.
1096              
1097             This is obviously not a problem for L, as you used
1098             it as the very first thing, right?
1099              
1100             It is a problem for L though - and the solution
1101             is to not create objects with nontrivial destructors that might have an
1102             effect outside of Perl.
1103              
1104             =back
1105              
1106             =head1 PORTABILITY NOTES
1107              
1108             Native win32 perls are somewhat supported (AnyEvent::Fork::Early is a nop,
1109             and ::Template is not going to work), and it cost a lot of blood and sweat
1110             to make it so, mostly due to the bloody broken perl that nobody seems to
1111             care about. The fork emulation is a bad joke - I have yet to see something
1112             useful that you can do with it without running into memory corruption
1113             issues or other braindamage. Hrrrr.
1114              
1115             Since fork is endlessly broken on win32 perls (it doesn't even remotely
1116             work within it's documented limits) and quite obviously it's not getting
1117             improved any time soon, the best way to proceed on windows would be to
1118             always use C and thus never rely on perl's fork "emulation".
1119              
1120             Cygwin perl is not supported at the moment due to some hilarious
1121             shortcomings of its API - see L for more details. If you never
1122             use C and always use C to create processes, it should
1123             work though.
1124              
1125             =head1 USING AnyEvent::Fork IN SUBPROCESSES
1126              
1127             AnyEvent::Fork itself cannot generally be used in subprocesses. As long as
1128             only one process ever forks new processes, sharing the template processes
1129             is possible (you could use a pipe as a lock by writing a byte into it to
1130             unlock, and reading the byte to lock for example)
1131              
1132             To make concurrent calls possible after fork, you should get rid of the
1133             template and early fork processes. AnyEvent::Fork will create a new
1134             template process as needed.
1135              
1136             undef $AnyEvent::Fork::EARLY;
1137             undef $AnyEvent::Fork::TEMPLATE;
1138              
1139             It doesn't matter whether you get rid of them in the parent or child after
1140             a fork.
1141              
1142             =head1 SEE ALSO
1143              
1144             L, to avoid executing a perl interpreter at all
1145             (part of this distribution).
1146              
1147             L, to create a process by forking the main
1148             program at a convenient time (part of this distribution).
1149              
1150             L, for another way to create processes that is
1151             mostly compatible to this module and modules building on top of it, but
1152             works better with remote processes.
1153              
1154             L, for simple RPC to child processes (on CPAN).
1155              
1156             L, for simple worker process pool (on CPAN).
1157              
1158             =head1 AUTHOR AND CONTACT INFORMATION
1159              
1160             Marc Lehmann
1161             http://software.schmorp.de/pkg/AnyEvent-Fork
1162              
1163             =cut
1164              
1165             1
1166