blib/lib/Sys/Binmode.pm

filter: Statementstmt 100% 13 / 13 Branchbran n/a Conditioncond n/a MC/DCmcdc n/a Subroutinesub 100% 4 / 4 Podpod n/a totaltotal 100% 17 / 17 CC 1 SCAR 0.0
CC
1
Cov
100%
BEGIN
1.0
BEGIN
1.0
import
1.0
CRAP 1.0
« Binmode.xs ↑ Summary
1 package Sys::Binmode;
2  
3 14 ▶use strict;
Subroutine BEGIN: called CC=1 SCAR=0.0
4 14 ▶use warnings;
Subroutine BEGIN: called CC=1 SCAR=0.0
5  
6 13 our $VERSION = '0.05';
7  
8 =encoding utf-8
9  
10 =head1 NAME
11  
12 Sys::Binmode - A fix for Perl’s system call character encoding
13  
14 =begin html
15  
16 <a href='https://coveralls.io/github/FGasper/p5-Sys-Binmode?branch=master'><img src='https://coveralls.io/repos/github/FGasper/p5-Sys-Binmode/badge.svg?branch=master' alt='Coverage Status' /></a>
17  
18 =end html
19  
20 =head1 SYNOPSIS
21  
22 use Sys::Binmode;
23  
24 my $foo = "\xff";
25 $foo .= "\x{100}";
26 chop $foo;
27  
28 # Prints a single octet (0xFF) and a newline:
29 print $foo, $/;
30  
31 # In Perl 5.32 this may print the same single octet, or it may
32 # print UTF-8-encoded U+00FF. With Sys::Binmode, though, it always
33 # gives the single octet, just like print:
34 exec 'echo', $foo;
35  
36 =head1 DESCRIPTION
37  
38 tl;dr: Use this module in B<all> new code.
39  
40 =head1 BACKGROUND
41  
42 Ideally, a Perl application doesn’t need to know how the interpreter stores
43 a given string internally. Perl can thus store any Unicode code point while
44 still optimizing for size and speed when storing “bytes-compatible”
45 strings—i.e., strings whose code points all lie below 256. Perl’s
46 “optimized” string storage format is faster and less memory-hungry, but it
47 can only store code points 0-255. The “unoptimized” format, on the other
48 hand, can store any Unicode code point.
49  
50 Of course, Perl doesn’t I<always> optimize “bytes-compatible” strings;
51 Perl can also, if
52 it wants, store such strings “unoptimized” (i.e., in Perl’s internal
53 “loose UTF-8” format), too. For code points 0-127 (ASCII printables,
54 controls, and DEL) there’s actually no
55 difference between the two forms, but for 128-255 the formats differ. (cf.
56 L<perlunicode/The "Unicode Bug">) This means that anything that reads
57 Perl’s internals B<MUST> differentiate between the two forms in order to
58 use the string correctly.
59  
60 Alas, that differentiation doesn’t always happen. When it doesn’t, Perl
61 outputs code points 128-255 differently depending on whether the
62 containing string is “optimized” or not.
63  
64 Remember, though: Perl applications I<should> I<not> I<care> about
65 Perl’s string storage internals like optimized/unoptimized. (This is why,
66 for example, the L<bytes>
67 pragma is discouraged.) The catch, though, is that without that knowledge,
68 B<the> B<application> B<can’t> B<know> B<what> B<it> B<actually> B<says>
69 B<to> B<the> B<outside> B<world!>
70  
71 Thus, applications must either monitor Perl’s string-storage internals
72 or accept unpredictable behavior, both of which are categorically bad.
73  
74 (Perl’s documentation calls the “unoptimized” format “upgraded”, while
75 it calls the “optimized” format “downgraded”. The rest of this document
76 will favor Perl’s terms.)
77  
78 =head1 HOW THIS MODULE (PARTLY) FIXES THE PROBLEM
79  
80 This module provides predictable behavior for Perl’s built-in functions by
81 downgrading all strings before giving them to the operating system. It’s
82 equivalent to—but faster than!—prefixing your system calls with
83 C<utf8::downgrade()> (cf. L<utf8>) on all arguments.
84  
85 Predictable behavior is B<always> a good thing; ergo, you should
86 use this module in B<all> new code.
87  
88 =head1 CAVEAT: CHARACTER ENCODING
89  
90 If you apply this module injudiciously to existing code you may see
91 exceptions or character corruption where previously things worked fine.
92  
93 This can
94 happen if you’ve neglected to encode one or more strings before
95 sending them to the OS. Without Sys::Binmode, Perl sends upgraded
96 strings to the OS in UTF-8 encoding. In essence, it’s an implicit
97 UTF-8 auto-encode, which is kind of nice, except that it depends on
98 Perl’s internals, which are unpredictable. Sys::Binmode removes
99 that implicit UTF-8 auto-encode, which of course will break things
100 that need it.
101  
102 The fix is to apply an explicit UTF-8 encode prior to the system call
103 that throws the error. This is what we should do I<anyway>;
104 Sys::Binmode just enforces that better.
105  
106 =head2 Example: The L<utf8> Pragma
107  
108 The widely-used L<utf8> pragma particularly exemplifies this problem.
109  
110 If you have code like this:
111  
112 use utf8;
113  
114 mkdir "épée";
115  
116 … then adding this module will change your program’s behavior in ways you’ll
117 probably dislike.
118  
119 Consider the string C<épée>. Without the C<utf8> pragma (but assuming that
120 the code I<is> actually written in UTF-8) this is 6
121 characters because the two C<é>s are 2 bytes each (so 2 + 1 + 2 + 1),
122 and without the C<utf8> pragma each byte in a string constant becomes its own
123 character, even if multiple bytes make up a single UTF-8 character. Since
124 nothing I<probably> upgrades that string on its way to
125 C<mkdir()>, the OS will receive the intended 6 bytes and create a directory
126 with a UTF-8-encoded name.
127  
128 I<With> C<utf8>, though, C<épée> is B<4> characters, not 6, because
129 this string is now UTF-8-decoded. Those 4 characters all lie beneath 256,
130 so the string is still bytes-compatible. Thus, if you C<print()> that string
131 you’ll get 4 bytes of Latin-1, which probably B<isn’t> what you want.
132  
133 C<mkdir()>, though, I<probably> still creates a directory with a 6-byte (UTF-8)
134 name. This happens when Perl itself stores C<épée> in upgraded (i.e.,
135 “unoptimized”) form. If that’s the case, that means Perl’s I<internal> buffer
136 of C<épée> is still the 6 bytes of UTF-8, even though to the Perl
137 I<application> it’s a 4-character string. Perl’s C<mkdir()> doesn’t care
138 about characters, though; it just gives Perl’s internal buffer to the
139 OS’s create-directory function. So by violating its own abstraction, Perl
140 happens to achieve something that is I<sometimes> useful.
141  
142 There are still two problems, though:
143  
144 =over
145  
146 =item * 1. Inconsistency: C<print()> sends 4 bytes to the OS while
147 C<mkdir()> (again, I<probably>) outputs 6.
148  
149 =item * 2. Uncertainty: C<épée> I<could> be stored downgraded rather than
150 upgraded, which would cause C<mkdir()> to send 4 bytes instead.
151  
152 =back
153  
154 C<print()>’s outputting of 4 bytes here is actually the B<correct> behavior
155 because it doesn’t depend on whether Perl stores the string upgraded or
156 downgraded. Sys::Binmode extends that correct behavior to C<mkdir()> and
157 other such Perl commands.
158  
159 Of course, in the end, we want C<mkdir()> to receive 6 bytes of UTF-8, not
160 4 bytes of Latin-1. To achieve that, just do as you normally do with
161 C<print()>: encode your string before you give it to the OS.
162  
163 use utf8;
164 use Encode;
165  
166 mkdir encode("UTF-8", "épée");
167  
168 This is what your code should look like, regardless of Sys::Binmode;
169 the omitted encoding step was a bug that Perl’s own abstraction-violation
170 bug I<might> have obscured for you. Sys::Binmode fixes Perl’s bug,
171 which makes you fix your own bug, too.
172  
173 =head2 Non-POSIX Operating Systems (e.g., Windows)
174  
175 In a POSIX operating system, an application’s communication with the
176 OS happens entirely through byte strings. Thus, treating all
177 OS-destined strings as byte strings is good and natural.
178  
179 In Windows, though, things are weirder. For example, Windows
180 exposes multiple APIs for creating a directory, and the one Perl uses (as of
181 5.32, anyway) only accepts code points 0-255. In this context Sys::Binmode
182 doesn’t I<break> anything, but it does reinforce one of Perl’s unfortunate
183 limitations on Windows.
184  
185 Sys::Binmode is a good idea anywhere that Perl sends byte strings to the OS.
186 For now, as far as I know, that’s everywhere that Perl runs. If that’s not
187 true, please file a bug.
188  
189 =head1 WHERE ELSE THIS PROBLEM CAN APPEAR
190  
191 The unpredictable-behavior problem that this module fixes in core Perl is
192 also common in L<CPAN|http://cpan.org>’s XS modules due to rampant
193 use of L<the SvPV macro|https://perldoc.perl.org/perlapi#SvPV> and
194 variants. SvPV is basically Perl’s L<bytes> pragma in C: it gives
195 you the string’s
196 internal bytes with no regard for what those bytes represent. This, of course,
197 is problematic for the same reason why the L<bytes> pragma is. XS authors
198 I<generally> should prefer
199 L<SvPVbyte|https://perldoc.perl.org/perlapi#SvPVbyte>
200 or L<SvPVutf8|https://perldoc.perl.org/perlapi#SvPVutf8> in lieu of
201 SvPV unless the C code in question handles Perl’s encoding abstraction.
202  
203 Note in particular that, as of Perl 5.32, the default XS typemap converts
204 scalars to C C<char *> and C<const char *> via an SvPV variant. This means
205 that any module that uses that conversion logic also has this problem.
206 So XS authors should also avoid the default typemap for such conversions.
207 (Again, though, use of the default typemap in this context is regrettably
208 commonplace.)
209  
210 Before Perl 5.18 this problem also affected %ENV. 5.18 introduced
211 an auto-downgrade when setting %ENV similar to what this module does.
212  
213 =head1 LEXICAL SCOPING
214  
215 If, for some reason, you I<want> Perl’s unpredictable default behavior,
216 you can disable this module for a given block via
217 C<no Sys::Binmode>, thus:
218  
219 use Sys::Binmode;
220  
221 system 'echo', $foo; # predictable/sane/happy
222  
223 {
224  
225 # You should probably explain here why you’re doing this.
226 no Sys::Binmode;
227  
228 system 'echo', $foo; # nasal demons
229 }
230  
231 =head1 AFFECTED BUILT-INS
232  
233 =over
234  
235 =item * C<exec>, C<system>, and C<readpipe>
236  
237 =item * C<do> and C<require>
238  
239 =item * File tests (e.g., C<-e>) and the following:
240 C<chdir>, C<chmod>, C<chown>, C<chroot>, C<ioctl>,
241 C<link>, C<lstat>, C<mkdir>, C<open>, C<opendir>, C<readlink>, C<rename>,
242 C<rmdir>, C<stat>, C<symlink>, C<sysopen>, C<truncate>,
243 C<unlink>, C<utime>
244  
245 =item * C<bind>, C<connect>, C<setsockopt>, and C<send> (last argument)
246  
247 =item * C<syscall>
248  
249 =back
250  
251 =head2 Omissions
252  
253 =over
254  
255 =item * C<crypt> already does as Sys::Binmode would make it do.
256  
257 =item * C<select> (the 4-argument one) has the bug that Sys::Binmode fixes,
258 but since it’s a performance-sensitive call where upgraded strings are
259 unlikely, this library doesn’t wrap it.
260  
261 =back
262  
263 =head1 KNOWN ISSUES
264  
265 L<autodie> creates functions named, e.g., C<chmod> in the
266 namespace of the module that C<import()>s it. Those functions lack
267 the compiler “hint” that tells Sys::Binmode to do its work; thus,
268 L<autodie “clobbers” Sys::Binmode|https://github.com/pjf/autodie/issues/113>.
269 C<CORE::*> functions will still have Sys::Binmode, but of course they won’t
270 throw exceptions.
271  
272 =head1 TODO
273  
274 =over
275  
276 =item * C<dbmopen> and the System V IPC functions aren’t covered here.
277 If you’d like them, ask.
278  
279 =item * There’s room for optimization, if that’s gainful.
280  
281 =item * Ideally this behavior should be in Perl’s core distribution.
282  
283 =item * Even more ideally, Perl should adopt this behavior as I<default>.
284 Maybe someday!
285  
286 =back
287  
288 =cut
289  
290 #----------------------------------------------------------------------
291  
292 13 require XSLoader;
293 13 XSLoader::load(__PACKAGE__, $VERSION);
294  
295 21 ▶sub import {
Subroutine import: called CC=1 SCAR=0.0
296 21 $^H{ _HINT_KEY() } = 1;
297  
298 21 return;
299 }
300  
301 1 ▶sub unimport {
Subroutine unimport: called CC=1 SCAR=0.0
302 1 delete $^H{ _HINT_KEY() };
303 }
304  
305 #----------------------------------------------------------------------
306  
307 =head1 ACKNOWLEDGEMENTS
308  
309 Thanks to Leon Timmermans (LEONT) and Paul Evans (PEVANS) for some
310 debugging and design help.
311  
312 =head1 LICENSE & COPYRIGHT
313  
314 Copyright 2021 Gasper Software Consulting. All rights reserved.
315  
316 This library is licensed under the same license as Perl.
317  
318 =cut
319  
320 13 1;
« Binmode.xs ↑ Summary