| 1 | package XS::Manifesto; | ||
| 2 | |||
| 3 | our $VERSION = '1.0.3'; | ||
| 4 | |||
| 5 | =head1 NAME | ||
| 6 | |||
| 7 | XS::Manifesto - Shared XS modules manifesto | ||
| 8 | |||
| 9 | =cut | ||
| 10 | |||
| 11 | =head1 THE CURRENT STATE | ||
| 12 | |||
| 13 | Out of the box Perl offers tools to create XS-modules, i.e. to write code in | ||
| 14 | C/C++ language for it to be both fast and be available from Perl via tools | ||
| 15 | like L<ExtUtils::MakeMaker>. | ||
| 16 | |||
| 17 | This approach works well, however it has B<limited extensibility>. First, it's | ||
| 18 | hard to call C/XS-code from other C/XS-code; while it I<can> be done, it's | ||
| 19 | possible only via a I<Perl layer> which has a huge performance penalty compared | ||
| 20 | to a direct C/C++ function call. Type safety is also lost, which is quite | ||
| 21 | important for early compile-time error detection with C/C++. Second, there is | ||
| 22 | no way to share I<source code>, e.g. header files and | ||
| 23 | native-type-to-Perl-type mappings (aka I<typemaps>). Without that information, | ||
| 24 | C/C++ types and templates from one module cannot be reused in another C/C++ | ||
| 25 | module. | ||
| 26 | |||
| 27 | There is an alternative approach on CPAN - L<Alien>, which tries to make | ||
| 28 | non-Perl libraries available for Perl. The approach I<reuses> system libraries | ||
| 29 | or downloads and builds them. It successfully solves I<sharing native code> | ||
| 30 | issue, i.e. C/C++ headers, but not sharing XS-code. One could argue that it's | ||
| 31 | Perl own fault that it ships no mechanism to share typemaps. Maybe it's even | ||
| 32 | not possible to implement this at all with pure C-layer. | ||
| 33 | |||
| 34 | The L<Alien>-specific issue is that it doesn't aim for | ||
| 35 | B<sharing binary (executable) code> (see | ||
| 36 | L<Alien CAVEATS | https://metacpan.org/pod/Alien#CAVEATS>). | ||
| 37 | While this approach has benefits, like the requirement of an Alien-library to | ||
| 38 | be a build-only (compile-only) dependency, the ease of module upgrades (without | ||
| 39 | rebuilding dependencies), etc., it has it's own limitations: | ||
| 40 | |||
| 41 | First, the non-sharing of library code means duplication of it in a processes' | ||
| 42 | memory. Let's say, there is an C<Alien::libX> and XS-libraries C<My::libA> | ||
| 43 | and C<My::libB>, which both use C-interface of C<libX>. Statically compiled | ||
| 44 | C<Alien::libX> will be duplicated in the both of XS-libraries. While memory is | ||
| 45 | considered to be cheap nowadays, it can still be an issue. | ||
| 46 | |||
| 47 | Second, as both XS-libraries C<My::libA> and C<My::libB> use C<Alien::libX> | ||
| 48 | independently, they can upgrade C<Alien::libX> independently. While it's a | ||
| 49 | benefit in some circumstances, in can lead to them losing binary compatibility | ||
| 50 | between themselves, i.e. data structures created via C<My::libA::libX> might | ||
| 51 | be not allowed to be transferred to C<My::libB::libX>. In other words, it's | ||
| 52 | only possible to have final XS-modules without binary inter-dependencies. | ||
| 53 | |||
| 54 | |||
| 55 | Third, as there is no support for XS-modules from L<Alien>, it makes it | ||
| 56 | impossible to have cross-dependent binary B<hierarchy> of XS-modules, like: | ||
| 57 | |||
| 58 | |||
| 59 | +---------------------------------+ | ||
| 60 | | | | ||
| 61 | v | | ||
| 62 | alien::libX <-- xs::libA <-- xs::libB <-- xs::libC <-- xs::libD | ||
| 63 | ^ | | ||
| 64 | | | | ||
| 65 | +--------------------------------------+ | ||
| 66 | |||
| 67 | as the modules are statically compiled, there is no runtime dependency between | ||
| 68 | C<xs::libA> (C<libA.a>) and C<xs::libB> (C<libB.a>); the C<libB.a> just embeds | ||
| 69 | (copies) C<libA.a> directly into own code. The object code copying propagates | ||
| 70 | through all further dependecies, upto C<xs::libD>. The opposite approach is | ||
| 71 | to have shared libraries, without any duplication. | ||
| 72 | |||
| 73 | |||
| 74 | =head1 THE INTUITION | ||
| 75 | |||
| 76 | It should be possible to have B<fast> applications in Perl. It should be | ||
| 77 | possible to have low-level components (like parsers, event loops, protocol | ||
| 78 | handlers, etc.) written in C/C++, while being able to access as much as | ||
| 79 | possible of their functions from Perl. The middleware components (like | ||
| 80 | application servers, session managers, etc.) can be written in Perl or in | ||
| 81 | C/C++ for performance-critical parts. The higher level application logic | ||
| 82 | is suitable mostly for Perl, with the exception of very limited | ||
| 83 | performance-critical parts. | ||
| 84 | |||
| 85 | There is an exception from the last rule: if the application models/code | ||
| 86 | B<have to be shared> with non-Perl applications (e.g. in game-application | ||
| 87 | with L<Unity|https://unity.com/> or L<Unreal|https://www.unrealengine.com> | ||
| 88 | engines), it obviously should be written in C/C++. But they still should | ||
| 89 | be accessible from Perl servers. | ||
| 90 | |||
| 91 | I<Summa summarum> any part of an application can be written in Perl and/or | ||
| 92 | in C/C++, the transition should be transparent. And as it's much faster to | ||
| 93 | create code in Perl, it's most likely that during the early stages of the | ||
| 94 | developement Perl code dominates, while later, as an application grows and | ||
| 95 | performance becomes an issue, parts of it are replaced with XS modules written | ||
| 96 | in C/C++. This make it possible to have a gradual evolution instead of radical | ||
| 97 | solutions like "let's rewrite everything in Go!". | ||
| 98 | |||
| 99 | |||
| 100 | =head1 THE PRINCIPLES | ||
| 101 | |||
| 102 | =over 2 | ||
| 103 | |||
| 104 | =item 1. B<Share binary code> | ||
| 105 | |||
| 106 | It's like L<Alien> modules build with system modules or with shared libraries | ||
| 107 | support. | ||
| 108 | |||
| 109 | Provided by L<XS::Install>. | ||
| 110 | |||
| 111 | =item 2. B<Application Binary Interface (ABI) runtime version check> | ||
| 112 | |||
| 113 | If C<XS::libraryX> had been compiled with the C<XS::libraryY v1.0> dependency, | ||
| 114 | but when loaded it finds out that the actual version of C<XS::libraryY> is | ||
| 115 | different (e.g. C<v1.1>), it by default refuses to load and asks for | ||
| 116 | recompilation with the actual dependency verison C<XS::libraryY v1.1>. | ||
| 117 | |||
| 118 | Provided by L<XS::Install>. | ||
| 119 | |||
| 120 | =item 3. B<No ABI backward compatibility> | ||
| 121 | |||
| 122 | Perl itself does not guarantees ABI-compatibility between major releases. | ||
| 123 | Maintaining ABI-compatibility has it's own costs, as well as possible | ||
| 124 | performance penalty, e.g. instead of a direct access of a public property | ||
| 125 | of a C-structure it now must be accessed via a function, preventing inlines | ||
| 126 | from a C/C++ compiler. | ||
| 127 | |||
| 128 | Usually C++ libraries do not maintain ABI-compatibility, so let it be. As the | ||
| 129 | drawback/consequence, if a base C<libraryX> is upgraded, all other modules | ||
| 130 | which depend on it should be recompiled. We exchange here build time for the | ||
| 131 | runtime performance. | ||
| 132 | |||
| 133 | =item 4. B<Sharing C++ XS-code> | ||
| 134 | |||
| 135 | There has been several attempts to share typemaps, i.e. make it possible to | ||
| 136 | reuse in C<xs::libraryX> conversion rules between C and Perl layers already | ||
| 137 | defined in C<xs::libraryY>. Unfortunately all attempts are unsatisfying, mostly | ||
| 138 | because there are no tools nor language support in C<C> for that. | ||
| 139 | |||
| 140 | However with modern C<C++> the situation is a bit different, as the powerful | ||
| 141 | template mechanism is provided by C<C++>. It can be used to share | ||
| 142 | I<C++ typemaps>. It is even possible to share C<C>-typemaps this way as long | ||
| 143 | as they are compile-time compatible with C<C++>. | ||
| 144 | |||
| 145 | Provided by L<XS::Framework>. | ||
| 146 | |||
| 147 | =item 5. B<Dual interface: C/C++ and Perl> | ||
| 148 | |||
| 149 | Every XS-module should have Perl interface to make it possible to use it as | ||
| 150 | self-contained module from Perl. It should also have a C/C++ interface to make | ||
| 151 | it possible to use it from other XS-module from theirs C/C++ code, i.e. embed | ||
| 152 | as a type or use in inheritance. | ||
| 153 | |||
| 154 | C<C++ typemap> should also be provided for easy using of the module. It is | ||
| 155 | considered as a part of C/C++ interface of a XS-module. | ||
| 156 | |||
| 157 | =back | ||
| 158 | |||
| 159 | |||
| 160 | =head1 EXAMPLE | ||
| 161 | |||
| 162 | There is a module L<Date>, which provides methods for serialization and parsing | ||
| 163 | dates in various formats. It is very fast, and when C<Date.pm> is loaded it | ||
| 164 | loads it's C++ XS backend as C<Date.so> (or C<Date.dll> on Windows). | ||
| 165 | It depends on L<XS::Framework>, so C<Framework.so> is also loaded and it's C++ | ||
| 166 | functions are used directly from C<Date.so>. | ||
| 167 | |||
| 168 | There is a module L<URI::XS> (say C<uri.so>), which parses/serializes URIs. | ||
| 169 | It also depends on L<XS::Framework>. | ||
| 170 | |||
| 171 | Now there is a module L<Protocol::HTTP>. When its C<http.so> is loaded, it | ||
| 172 | also loads C<Framework.so>, C<uri.so> and C<Date.so>. So when date or URIs are | ||
| 173 | parsed/serialized from Perl code using L<Protocol::HTTP>, it directly invokes | ||
| 174 | code from C<uri.so> or C<Date.so>, without routing via Perl (which isn't fast). | ||
| 175 | |||
| 176 | When URI/Date objects are returned to Perl layer from the L<Protocol::HTTP> XS | ||
| 177 | API, they're exactly the same C++ objects (i.e. no any additional memory | ||
| 178 | allocations for C++ objects) just wrapped into Perl SV (scalars) and it's | ||
| 179 | possible to simply use them in Perl APIs of the corresponding modules | ||
| 180 | (L<URI::XS> or L<Date>), which exists completely outside of L<Protocol::HTTP> | ||
| 181 | space. | ||
| 182 | |||
| 183 | L<XS::Install> (tooling support) and L<XS::Framework> (XS/C++ support) makes | ||
| 184 | it possible to do that sharing in easy way. | ||
| 185 | |||
| 186 | =head1 SEE ALSO | ||
| 187 | |||
| 188 | L<Alien> | ||
| 189 | |||
| 190 | L<ExtUtils::MakeMaker> | ||
| 191 | |||
| 192 | L<XS::Install> | ||
| 193 | |||
| 194 | L<XS::Framework> | ||
| 195 | |||
| 196 | =head1 AUTHORS | ||
| 197 | |||
| 198 | Pronin Oleg (SYBER) <syber@crazypanda.ru>, Crazy Panda LTD | ||
| 199 | |||
| 200 | Sergey Aleynikov (RANDIR) <sergey.aleynikov@gmail.com>, Crazy Panda LTD | ||
| 201 | |||
| 202 | Ivan Baidakou (DMOL) <i.baydakov@crazypanda.ru>, Crazy Panda LTD | ||
| 203 | |||
| 204 | =head1 LICENSE | ||
| 205 | |||
| 206 | You may distribute this code under the same terms as Perl itself. | ||
| 207 | |||
| 208 | =cut | ||
| 209 | |||
| 210 | |||
| 211 | 1; |