blib/lib/Syntax/Highlight/Perl.pm untested

filter: Statementstmt 0.0% 0 / 513 Branchbran 0.0% 0 / 284 Conditioncond 0.0% 0 / 392 MC/DCmcdc 0.0% 0 / 213 Subroutinesub 0.0% 0 / 36 Podpod 0.0% 0 / 24 totaltotal 0.0% 0 / 1462 CC 143 SCAR 79.6
CC
143
Cov
0%
_process_token
6320.0
_format_line
380.0
format_token
90.0
CRAP 2857.0
1 package Syntax::Highlight::Perl;
2 #
3 # Syntax::Highlight::Perl
4 #
5 # Created 01-23-2001 by Cory Johns
6 # Last Modified 04-04-2001 by Cory Johns
7 #
8 # (See end of file (or POD) for revision history.)
9 #
10 # Freely distributable under the same conditions as Perl itself.
11 #
12  
13 ($VERSION) = '1.00' =~ /([.,\d]+)/;
14  
15  
16 =head1 NAME
17  
18 Syntax::Highlight::Perl - Highlighting of Perl Syntactical Structures
19  
20 =head1 VERSION
21  
22 This file documents Syntax::Highlight::Perl version B<1.0>.
23  
24 =head1 SYNOPSIS
25  
26 # simple procedural
27 use Syntax::Highlight::Perl ':BASIC'; # or ':FULL'
28  
29 print format_string($my_string);
30  
31  
32 # OO
33 use Syntax::Highlight::Perl;
34  
35 my $formatter = new Syntax::Highlight::Perl;
36 print $formatter->format_string($my_string);
37  
38 =head1 DESCRIPTION
39  
40 This module provides syntax highlighting for Perl code. The design bias is roughly line-oriented
41 and streamed (ie, processing a file line-by-line in a single pass). Provisions I<may> be made
42 in the future for tasks related to "back-tracking" (ie, re-doing a single line in the middle of
43 a stream) such as speeding up state copying.
44  
45 =over 4
46  
47 =head2 Constructors
48  
49 The only constructor provided is C<new()>. When called on an existing object, C<new()> will
50 create a new I<B<copy>> of that object. Otherwise, C<new()> creates a new copy of the (internal)
51 I<Default Object>. Note that the use of the procedural syntax modifies the I<Default Object>
52 and that those changes I<will> be reflected in any subsequent C<new()> calls.
53  
54 =head2 Formatting
55  
56 Formatting is done using the C<format_string()> method. Call C<format_string()> with one or more
57 strings to format, or it will default to using C<$_>.
58  
59 =head2 Setting and Getting Formats
60  
61 You can set the text used for formatting a syntax element using C<set_format()> (or set
62 the start and end format individually using C<set_start_format()> and C<set_end_format()>,
63 respectively).
64  
65 You can also retrieve the text used for formatting for an element via C<get_start_format()>
66 or C<get_end_format>. Bulk retrieval of the names or values of defined formats is possible
67 via C<get_format_names_list()> (names), C<get_start_format_values_list()> and C<get_end_format_values_list()>.
68  
69 See L<"FORMAT TYPES"> later in this document for information on what format elements can be used.
70  
71 =head2 Checking and Setting the State
72  
73 You can check certain aspects of the state of the formatter via the methods: C<in_heredoc()>,
74 C<in_string()>, C<in_pod()>, C<was_pod()>, C<in_data()>, and C<line_count()>.
75  
76 You can reset all of the above states (and a few other internal ones) using C<reset()>.
77  
78 =head2 Stable and Unstable Formatting Modes
79  
80 You can set or check the stability of formatting via C<unstable()>.
81  
82 In unstable (TRUE) mode, formatting is not considered to be persistent with nested formats.
83 Or, put another way, when unstable, the formatter can only "remember" one format at a time
84 and must reinstate formatting for each token. An example of unstable formatting is using
85 ANSI color escape sequences in a terminal.
86  
87 In stable (FALSE) mode (the default), formatting is considered persistent within arbitrarily
88 nested formats. Even in stable mode, however, formatting is never allowed to span multiple lines;
89 it is always fully closed at the end of the line and reinstated at the beginning of a new line,
90 if necessary. This is to ensure properly balanced tags when only formatting a partial code snippet.
91 An example of stable formatting is HTML.
92  
93 =head2 Substitutions
94  
95 Using C<define_substitution()>, you can have the formatter substitute certain strings with others,
96 after the original string has been parsed (but before formatting is applied). This is useful
97 for escaping characters special to the output mode (eg, E<gt> and E<lt> in HTML) without them
98 affecting the way the code is parsed.
99  
100 You can retrieve the current substitutions (as a hash-ref) via C<substitutions()>.
101  
102 =back
103  
104 =head1 FORMAT TYPES
105  
106 The Syntax::Highlight::Perl formatter recognizes and differentiates between many Perl syntactical
107 elements. Each type of syntactical element has a Format Type associated with it. There is also a
108 'DEFAULT' type that is applied to any element who's Format Type does not have a value.
109  
110 Several of the Format Types have underscores in their name. This underscore is special, and
111 indicates that the Format Type can be "generalized." This means that you can assign a value to
112 just the first part of the Format Type name (the part before the underscore) and that value will
113 be applied to all Format Types with the same first part. For example, the Format Types for all
114 types of variables begin with "Variable_". Thus, if you assign a value to the Format Type "Variable",
115 it will be applied to any type of variable. Generalized Format Types take precedence over
116 non-generalized Format Types. So the value assigned to "Variable" would be applied to
117 "Variable_Scalar", even if "Variable_Scalar" had a value explicitly assigned to it.
118  
119 You can also define a "short-cut" name for each Format Type that can be generalized. The short-cut
120 name would be the part of the Format Type name after the underscore. For example, the short-cut
121 for "Variable_Scalar" would be "Scalar". Short-cut names have the least precedence and are only
122 assigned if neither the generalized Type name, nor the full Type name have values.
123  
124 Following is a list of all the syntactical elements that Syntax::Highlight::Perl currently
125 recognizes, along with a short description of what each would be applied to.
126  
127 =over 4
128  
129 =item Comment_Normal
130  
131 A normal Perl comment. Starts with '#' and goes until the end of the line.
132  
133 =item Comment_POD
134  
135 Inline documentation. Starts with a line beginning with an equal sign ('=') followed by
136 a word (eg: '=pod') and continuing until a line beginning with '=cut'.
137  
138 =item Directive
139  
140 Either the "she-bang" line at the beginning of the file, or a line directive altering what
141 the compiler thinks the current line and file is.
142  
143 =item Label
144  
145 A loop or statement label (to be the target of a goto, next, last or redo).
146  
147 =item Quote
148  
149 Any string or character that begins or ends a String. Including, but not necessarily limited to:
150 quote-like regular expression operators (C<m//>, C<s///>, C<tr///>, etc), a Here-Document terminating
151 line, the lone period terminating a format, and, of course, normal quotes (C<'>, C<">, C<`>, C<q{}>,
152 C<qq{}>, C<qr{}>, C<qx{}>).
153  
154 =item String
155  
156 Any text within quotes, C<format>s, Here-Documents, Regular Expressions, and the like.
157  
158 =item Subroutine
159  
160 The identifier used to define, identify, or call a subroutine (or method). Note that
161 Syntax::Highlight::Perl cannot recognize a subroutine if it is called without using parentheses
162 or an ampersand, or methods called using the indirect object syntax. It formats those as barewords.
163  
164 =item Variable_Scalar
165  
166 A scalar variable.
167  
168 Note that (theoretically) this format is not applied to non-scalar variables that are
169 being used as scalars (ie: array or hash lookups, nor references to anything other than scalars).
170 Syntax::Highlight::Perl figures out (or at least tries to) the actual I<type> of the variable
171 being used (by looking at how you're subscripting it) and formats it accordingly. The first
172 character of the variable (ie, the C<$>, C<@>, C<%>, or C<*>) tells you the type of value being
173 used, and the color (hopefully) tells you the type of variable being used to get that value.
174  
175 (See L<"KNOWN ISSUES"> for information about when this doesn't work quite right.)
176  
177 =item Variable_Array
178  
179 An array variable (but not usually a slice; see above).
180  
181 =item Variable_Hash
182  
183 A hash variable.
184  
185 =item Variable_Typeglob
186  
187 A typeglob. Note that typeglobs not beginning with an asterisk (*) (eg: filehandles) are formatted
188 as barewords. This is because, well, they are.
189  
190 =item Whitespace
191  
192 Whitespace. Not usually formatted but it can be.
193  
194 =item Character
195  
196 A special, or backslash-escaped, character. For example: C<\n> (newline), or C<\d> (digits).
197  
198 Only occurs within strings or regular expressions.
199  
200 =item Keyword
201  
202 A Perl keyword. Some examples include: my, local, sub, next.
203  
204 Note that Perl does not make any distinction between keywords and built-in functions (at least
205 not in the documentation). Thus I had to make a subjective call as to what would be considered
206 keywords and what would be built-in functions.
207  
208 The list of keywords can be found (and overloaded) in the variable
209 C<$Syntax::Highlight::Perl::keyword_list_re> as a pre-compiled regular expression.
210  
211 =item Builtin_Function
212  
213 A Perl built-in function, called as a function (ie, using parentheses).
214  
215 The list of built-in functions can be found (and overloaded) in the variable
216 C<$Syntax::Highlight::Perl::builtin_list_re> as a pre-compiled regular expression.
217  
218 =item Builtin_Operator
219  
220 A Perl built-in function, called as a list or unary operator (ie, without using parentheses).
221  
222 The list of built-in functions can be found (and overloaded) in the variable
223 C<$Syntax::Highlight::Perl::builtin_list_re> as a pre-compiled regular expression.
224  
225 =item Operator
226  
227 A Perl operator.
228  
229 The list of operators can be found (and overloaded) in the variable
230 C<$Syntax::Highlight::Perl::operator_list_re> as a pre-compiled regular expression.
231  
232 =item Bareword
233  
234 A bareword. This can be user-defined subroutine called without parentheses, a typeglob used
235 without an asterisk (*), or just a plain old bareword.
236  
237 =item Package
238  
239 The name of a package or pragmatic module.
240  
241 Note that this does not apply to the package portion of a fully qualified variable name.
242  
243 =item Number
244  
245 A numeric literal.
246  
247 =item Symbol
248  
249 A symbol (ie, non-operator punctuation).
250  
251 =item CodeTerm
252  
253 The special tokens that signal the end of executable code and the begining of the
254 DATA section. Specifically, 'C<__END__>' and 'C<__DATA__>'.
255  
256 =item DATA
257  
258 Anything in the DATA section (see C<CodeTerm>).
259  
260 =back
261  
262 =head1 PROCEDURAL vs. OBJECT ORIENTED
263  
264 Syntax::Highlight::Perl uses OO method-calls internally (and actually defines a Default Object
265 that is used when the functions are invoked procedurally) so you will not gain anything
266 (efficiency-wise) by using the procedural interface. It is just a matter of style.
267  
268 It is actually recommended that you use the OO interface, as this allows you to instantiate
269 multiple, concurrent-yet-separate formatters. Though I cannot think of I<why> you would I<need>
270 multiple formatters instantiated. :-)
271  
272 One point to note: the C<new()> method uses the Default Object to initialize new objects. This
273 means that any changes to the state of the Default Object (including Format definitions) made by
274 using the procedural interface will be reflected in any subsequently created objects. This can
275 be useful in some cases (eg, call C<set_format()> procedurally just before creating a batch of new
276 objects to define default Formats for them all) but will most likely lead to trouble.
277  
278 =cut
279  
280 use strict;
281  
282 #===================================================================================================
283 # Yay constants!
284 #===================================================================================================
285  
286 use constant FALSE => 0;
287 use constant TRUE => not FALSE;
288  
289  
290  
291 #===================================================================================================
292 # Setup package exportation.
293 #===================================================================================================
294  
295 use base 'Exporter';
296  
297 use vars qw(@EXPORT_OK %EXPORT_TAGS);
298  
299 @EXPORT_OK = qw(
300 reset
301 unstable
302 in_heredoc
303 in_string
304 in_pod
305 was_pod
306 in_data
307 line_count
308 substitutions
309 define_substitution
310 set_start_format
311 set_end_format
312 set_format
313 get_start_format
314 get_end_format
315 get_format_names_list
316 get_start_format_values_list
317 get_end_format_values_list
318 format_string
319 format_token
320 );
321  
322 %EXPORT_TAGS = (
323 BASIC => [ qw(
324 set_format
325 format_string
326 ) ],
327  
328 FULL => [ @EXPORT_OK ],
329 );
330  
331  
332  
333 #===================================================================================================
334 # Define package globals.
335 #===================================================================================================
336  
337 #
338 # This is the (overloadable) name of the current object.
339 # Used to bless new objects.
340 #
341 use vars qw($THIS_CLASS);
342 $THIS_CLASS = __PACKAGE__;
343  
344  
345 #
346 # These regular expressions do much of the work for this package. They recognize and separate
347 # the lines into small, meaningful (and usefull) tokens.
348 #
349 use vars qw($quotes_re $heredoc_re $identifier_re $builtin_vars_re $number_re);
350 use vars qw($keyword_list_re $builtin_list_re $operator_list_re);
351  
352 #
353 # Quotes and quote-like constructs.
354 #
355 # Note: This regex primarily matches quote _instigators_ (as opposed to terminators)
356 # (when it can tell the difference). For example, given:
357 #
358 # qw (blah blah blah)
359 #
360 # it will match 'qw ('. It matches up to the open parenthesis so that we will
361 # be able to tell later on what we should look for to close the quote (ie ')').
362 #
363 $quotes_re = qr/(?: q[qxwr]? | m | s | tr | y ) (?: \s+ [^\s\#] | [^\w\s] ) | [\'\"\`] /x;
364  
365 #
366 # Here-document instigators.
367 #
368 $heredoc_re = qr/<<(?:\s*['"`]\w+['"`]|\w+)?/;
369  
370 #
371 # Identifiers (variables, subroutines, and packages).
372 #
373 $identifier_re = qr/(?:(?:[A-Za-z_]|::)(?:\w|::)*)/;
374  
375 #
376 # Variable class specifiers.
377 #
378 my $varchars = qr/(?:(?:[\@\%\*]|\$\#?)\$*)/;
379  
380 #
381 # Perl builtin variables.
382 #
383 $builtin_vars_re =
384 qr/ \$\#?_
385 | \$(?:\^[LAECDFHIMOPRSTWX]|[0-9&`'+*.\/|,\\";#%=\-~^:?!@\$<>()\[\]])
386 | \$\#?ARGV(?:\s*\[)?
387 | \$\#?INC\s*\[
388 | \$(?:ENV|SIG|INC)\s*\{
389 | \@(?:_|ARGV|INC)
390 | \%(?:INC|ENV|SIG)
391 /x;
392  
393 #
394 # Numbers.
395 #
396 $number_re = qr/0x[\da-fA-F]+|[_.\d]+([eE][-+]?\d+)?/;
397  
398 #
399 # The list of keywords. Okay, so it's ugly.
400 #
401 $keyword_list_re =
402 qr/ continue
403 | foreach
404 | require
405 | package
406 | scalar
407 | format
408 | unless
409 | local
410 | until
411 | while
412 | elsif
413 | next
414 | last
415 | goto
416 | else
417 | redo
418 | sub
419 | for
420 | use
421 | no
422 | if
423 | my
424 /x;
425  
426 #
427 # The list of builtin functions. Damn, that's ugly.
428 #
429 $builtin_list_re =
430 qr/ getprotobynumber
431 | getprotobyname
432 | getservbyname
433 | gethostbyaddr
434 | gethostbyname
435 | getservbyport
436 | getnetbyaddr
437 | getnetbyname
438 | getsockname
439 | getpeername
440 | setpriority
441 | getprotoent
442 | setprotoent
443 | getpriority
444 | endprotoent
445 | getservent
446 | setservent
447 | endservent
448 | sethostent
449 | socketpair
450 | getsockopt
451 | gethostent
452 | endhostent
453 | setsockopt
454 | setnetent
455 | quotemeta
456 | localtime
457 | prototype
458 | getnetent
459 | endnetent
460 | rewinddir
461 | wantarray
462 | getpwuid
463 | closedir
464 | getlogin
465 | readlink
466 | endgrent
467 | getgrgid
468 | getgrnam
469 | shmwrite
470 | shutdown
471 | readline
472 | endpwent
473 | setgrent
474 | readpipe
475 | formline
476 | truncate
477 | dbmclose
478 | syswrite
479 | setpwent
480 | getpwnam
481 | getgrent
482 | getpwent
483 | ucfirst
484 | sysread
485 | setpgrp
486 | shmread
487 | sysseek
488 | sysopen
489 | telldir
490 | defined
491 | opendir
492 | connect
493 | lcfirst
494 | getppid
495 | binmode
496 | syscall
497 | sprintf
498 | getpgrp
499 | readdir
500 | seekdir
501 | waitpid
502 | reverse
503 | unshift
504 | symlink
505 | dbmopen
506 | semget
507 | msgrcv
508 | rename
509 | listen
510 | chroot
511 | msgsnd
512 | shmctl
513 | accept
514 | unpack
515 | exists
516 | fileno
517 | shmget
518 | system
519 | unlink
520 | printf
521 | gmtime
522 | msgctl
523 | semctl
524 | values
525 | rindex
526 | substr
527 | splice
528 | length
529 | msgget
530 | select
531 | socket
532 | return
533 | caller
534 | delete
535 | alarm
536 | ioctl
537 | index
538 | undef
539 | lstat
540 | times
541 | srand
542 | chown
543 | fcntl
544 | close
545 | write
546 | umask
547 | rmdir
548 | study
549 | sleep
550 | chomp
551 | untie
552 | print
553 | utime
554 | mkdir
555 | atan2
556 | split
557 | crypt
558 | flock
559 | chmod
560 | BEGIN
561 | bless
562 | chdir
563 | semop
564 | shift
565 | reset
566 | link
567 | stat
568 | chop
569 | grep
570 | fork
571 | dump
572 | join
573 | open
574 | tell
575 | pipe
576 | exit
577 | glob
578 | warn
579 | each
580 | bind
581 | sort
582 | pack
583 | eval
584 | push
585 | keys
586 | getc
587 | kill
588 | seek
589 | sqrt
590 | send
591 | wait
592 | rand
593 | tied
594 | read
595 | time
596 | exec
597 | recv
598 | eof
599 | chr
600 | int
601 | ord
602 | exp
603 | pos
604 | pop
605 | sin
606 | log
607 | abs
608 | oct
609 | hex
610 | tie
611 | cos
612 | vec
613 | END
614 | ref
615 | map
616 | die
617 | \-C
618 | \-b
619 | \-S
620 | \-u
621 | \-t
622 | \-p
623 | \-l
624 | \-d
625 | \-f
626 | \-g
627 | \-s
628 | \-z
629 | uc
630 | \-k
631 | \-e
632 | \-O
633 | \-T
634 | \-B
635 | \-M
636 | do
637 | \-A
638 | \-X
639 | \-W
640 | \-c
641 | \-R
642 | \-o
643 | \-x
644 | lc
645 | \-w
646 | \-r
647 /x;
648  
649 #
650 # The list of operators. Also pretty ugly.
651 #
652 $operator_list_re =
653 qr/ xor
654 | \.\.\.
655 | and
656 | not
657 | \|\|\=
658 | cmp
659 | \>\>\=
660 | \<\<\=
661 | \<\=\>
662 | \&\&\=
663 | or
664 | \=\>
665 | \!\~
666 | \^\=
667 | \&\=
668 | \|\=
669 | \.\=
670 | x\=
671 | \%\=
672 | \/\=
673 | \*\=
674 | \-\=
675 | \+\=
676 | \=\~
677 | \*\*
678 | \-\-
679 | \.\.
680 | \|\|
681 | \&\&
682 | \+\+
683 | \-\>
684 | ne
685 | eq
686 | \!\=
687 | \=\=
688 | ge
689 | le
690 | gt
691 | lt
692 | \>\=
693 | \<\=
694 | \>\>
695 | \<\<
696 | \,
697 | \=
698 | \:
699 | \?
700 | \^
701 | \|
702 | x
703 | \%
704 | \/
705 | \*
706 | \<
707 | \&
708 | \\
709 | \~
710 | \!
711 | \>
712 | \.
713 | \-
714 | \+
715 /x;
716  
717  
718  
719  
720 #===================================================================================================
721 # Package-private variables.
722 #
723 # Basically, just defaults.
724 #===================================================================================================
725  
726  
727  
728 #
729 # The default object variable.
730 #
731 # Note that all of these values (including Formats) will be *COPIED* (recursively)
732 # into any newly created object.
733 #
734 # Also note that any changes made to this object (via procedural-style calls to almost
735 # any of the exported functions) _will be reflected in any objects created after those
736 # changes are made_. In other words, you probably don't want to mix OO-style and
737 # Procedural-style unless you really know what you're doing.
738 #
739 # Further note that this object is blessed when this package is 'use'd (or 'require'd)
740 # (ie, before you have a chance to overload the $THIS_CLASS package-global variable).
741 # If you're inheriting from this package, you'll probably want to re-bless this object.
742 # Although it is perhaps not totally necessary as long as you overload the $THIS_CLASS
743 # package-global since new() re-blesses any copies of this object using that class.
744 #
745 use vars qw($Default_Object);
746 $Default_Object = bless {
747  
748 #
749 # State variables.
750 #
751  
752 'in_string' => FALSE, # Boolean.
753 'awaiting_multi' => FALSE, # Boolean. Waiting for second part of Multipart construct.
754 'found_multi' => FALSE, # Boolean.
755 'awaiting_options' => FALSE, # Boolean. Waiting for options for a Optioned construct.
756 'quote_instigator' => '',
757 'quote_terminator' => '',
758 'quote_type' => '', # Any in (Interpolated, Multipart, Optioned)
759 'nested_quote' => 0, # Anything >= 0.
760  
761 'awaiting_variable' => FALSE, # Boolean. Indicates we're awaiting a variable identifier.
762 'awaiting_class' => '', # Any in (Scalar, Array, Hash). Class of the variable we're waiting on.
763  
764 'in_heredoc' => 0, # Anything >= 0 (boolean count).
765 'here_terminator' => [], # List of Here-Doc terminators that we're waiting for.
766  
767 'in_pod' => FALSE, # Boolean. Indicates that we're inside a POD.
768 'was_pod' => FALSE, # Boolean. Indicates that we _were_ inside a POD. (For in_pod() method.)
769  
770 'in_data' => FALSE, # Boolean. We don't format __DATA__.
771  
772 'in_comment' => FALSE, # Boolean. Indicates we're inside a normal comment.
773  
774 'last_token' => '', # The last (non-whitespace, non-comment) token processed.
775 'last_token_type' => '', # Type of the last token.
776  
777 'reentrant' => FALSE, # Boolean. Indicates we're formatting a sub-token.
778  
779 'line_count' => 0, # The number of lines that have been processed.
780  
781 'unstable' => FALSE, # Boolean. Indicates formats are not persistent (eg. terminal colors).
782  
783  
784 #
785 # Option tables.
786 #
787  
788 'Substitution Table'=> {}, # Table of characters to replace before formatting.
789  
790 'Formats' => {
791 'DEFAULT' => ['', ''],
792 },
793  
794 }, $THIS_CLASS;
795  
796  
797  
798  
799 #===================================================================================================
800 # Public Methods
801 #
802 # The stuff.
803 #===================================================================================================
804  
805  
806  
807  
808 =head1 METHODS
809  
810 =over 4
811  
812 =item new PACKAGE
813  
814 =item new OBJECT
815  
816 Creates a new object. If called on an existing object, creates a new copy of that
817 object (which is thenceforth totally separate from the original).
818  
819 =cut
820  
821 sub new {
822  
823 my $class = shift;
824 my $true_class = ref($class) || $class; # We don't want our class being the string version of a reference.
825 my $source = ref($class) ? $class : $Default_Object; # If called as method of exisiting object, copy.
826  
827 #
828 # Hrm. Maybe I'll eventually take a look inside Data::Dumper and see how
829 # it goes about descending the complex structures and write a non-eval
830 # deep-copy routine. :-)
831 #
832 # Seems kinda silly to put that much work onto the perl parser when we could
833 # just copy it in-memory.
834 #
835 use Data::Dumper; # For deep-copy.
836 use Carp; # For croak().
837  
838 local $SIG{__WARN__} = sub { chomp($@ = $_[0]) }; # eval() sometimes doesn't put all it's errors in $@
839  
840 local $Data::Dumper::Deepcopy = 1; # Descend into nested (complex) structures.
841 local $Data::Dumper::Terse = 1; # Don't print variable names and assignments.
842 local $Data::Dumper::Purity = 1; # Ensure nested references are correctly recreated.
843  
844 my $code = Dumper($source); # Save in case of errors.
845  
846 my $self = eval($code) or
847 croak "Deepcopy failed in $true_class\::new() with the error:\n $@\non the code:\n$code";
848  
849 return bless($self, $true_class);
850  
851 }
852  
853  
854  
855  
856 =item reset
857  
858 Resets the object's internal state. This breaks out of strings and here-docs, ends PODs,
859 resets the line-count, and otherwise gets the object back into a "normal" state to begin
860 processing a new stream.
861  
862 Note that this does B<I<not>> reset any user options (including formats and format stability).
863  
864 =cut
865  
866 sub reset {
867  
868 my $self = ref($_[0]) ? shift : $Default_Object;
869  
870 $self->{'in_string'} = FALSE;
871 $self->{'awaiting_multi'} = FALSE;
872 $self->{'found_multi'} = FALSE;
873 $self->{'awaiting_options'} = FALSE;
874 $self->{'quote_instigator'} = '';
875 $self->{'quote_terminator'} = '';
876 $self->{'quote_type'} = '';
877  
878 $self->{'awaiting_variable'} = FALSE;
879 $self->{'awaiting_class'} = '';
880  
881 $self->{'in_heredoc'} = 0;
882 @{$self->{'here_terminator'}} = (); # No need for a whole new anon. array. Just empty this one.
883  
884 $self->{'in_pod'} = FALSE;
885 $self->{'was_pod'} = FALSE;
886  
887 $self->{'in_data'} = FALSE;
888  
889 $self->{'in_comment'} = FALSE;
890  
891 $self->{'last_token'} = '';
892 $self->{'last_token_type'} = '';
893  
894 $self->{'reentrant'} = FALSE;
895  
896 $self->{'line_count'} = 0;
897  
898 }
899  
900  
901  
902  
903 =item unstable EXPR
904  
905 =item unstable
906  
907 Returns true if the formatter is in unstable mode.
908  
909 If called with a non-zero number, puts the formatter into unstable formatting mode.
910  
911 In unstable mode, it is assumed that formatting is not persistent one token to the
912 next and that each token must be explicitly formatted.
913  
914 =cut
915  
916 sub unstable {
917 my $self = ref($_[0]) ? shift : $Default_Object;
918 return @_ ? $self->{'unstable'} = shift : $self->{'unstable'};
919 }
920  
921  
922  
923  
924 =item in_heredoc
925  
926 Returns true if the next string to be formatted will be inside a Here-Document.
927  
928 =cut
929  
930 sub in_heredoc {
931 my $self = ref($_[0]) ? shift : $Default_Object;
932 return $self->{'in_heredoc'};
933 }
934  
935  
936  
937  
938 =item in_string
939  
940 Returns true if the next string to be formatted will be inside a multi-line string.
941  
942 =cut
943  
944 sub in_string {
945 my $self = ref($_[0]) ? shift : $Default_Object;
946 return $self->{'in_string'};
947 }
948  
949  
950  
951  
952 =item in_pod
953  
954 Returns true if the formatter would consider the next string passed to it as begin within
955 a POD structure. This is false immediately before any POD instigators
956 (C<=pod>, C<=head1>, C<=item>, etc), true immediately after an instigator, throughout the POD and
957 immediately before the POD terminator (C<=cut>), and false immediately after the POD terminator.
958  
959 =cut
960  
961 sub in_pod {
962 my $self = ref($_[0]) ? shift : $Default_Object;
963 return $self->{'in_pod'};
964 }
965  
966  
967  
968  
969 =item was_pod
970  
971 Returns true if the last line of the string just formatted was part of a POD structure.
972 This includes the C</^=\w+/> POD instigators and terminators.
973  
974 =cut
975  
976 sub was_pod {
977 my $self = ref($_[0]) ? shift : $Default_Object;
978 return $self->{'was_pod'};
979 }
980  
981  
982  
983  
984 =item in_data
985  
986 Returns true if the next string to be formatted will be inside
987 the DATA section (ie, follows a C<__DATA__> or C<__END__> tag).
988  
989 =cut
990  
991 sub in_data {
992 my $self = ref($_[0]) ? shift : $Default_Object;
993 return $self->{'in_data'};
994 }
995  
996  
997  
998  
999 =item line_count
1000  
1001 Returns the number of lines processed by the formatter.
1002  
1003 =cut
1004  
1005 sub line_count {
1006 my $self = ref($_[0]) ? shift : $Default_Object;
1007 return $self->{'line_count'};
1008 }
1009  
1010  
1011  
1012  
1013 =item substitutions
1014  
1015 Returns a reference to the substitution table used. The substitution
1016 table is a hash whose keys are the strings to be replaced, and whose values
1017 are what to replace them with.
1018  
1019 =cut
1020  
1021 sub substitutions {
1022  
1023 my $self = ref($_[0]) ? shift : $Default_Object;
1024  
1025 return $self->{'Substitution Table'};
1026  
1027 }
1028  
1029  
1030  
1031  
1032 =item define_substitution HASH_REF
1033  
1034 =item define_substitution LIST
1035  
1036 Allows user to define certain characters that will be substituted
1037 before formatting is done (but after they have been processed for
1038 meaning).
1039  
1040 If the first parameter is a reference to a hash, the formatter
1041 will replace it's own hash with the given one, and subsequent
1042 changes to the hash outside the formatter will be reflected.
1043  
1044 Otherwise, it will copy the arguments passed into it's own
1045 hash, and any substitutions already defined (but not in the
1046 parameter list) will be preserved. (ie, the new substitutions
1047 will be added, without destroying what was there already.)
1048  
1049 =cut
1050  
1051 sub define_substitution {
1052  
1053 my $self = ref($_[0]) ? shift : $Default_Object;
1054  
1055 if(ref($_[0]) eq 'HASH') {
1056 $self->{'Substitution Table'} = $_[0];
1057 } else {
1058 my %tmphash = @_;
1059 @{$self->{'Substitution Table'}}{keys %tmphash} = values %tmphash;
1060 }
1061  
1062 }
1063  
1064  
1065  
1066  
1067 =item set_start_format HASH_REF
1068  
1069 =item set_start_format LIST
1070  
1071 Given either a list of keys/values, or a reference to a hash of keys/values,
1072 copy them into the object's Formats list.
1073  
1074 =cut
1075  
1076 sub set_start_format {
1077  
1078 my $self = ref($_[0]) ? shift : $Default_Object;
1079 my %tmphash = ref($_[0]) ? %{$_[0]} : @_;
1080  
1081 $self->{'Formats'}{$_}[0] = $tmphash{$_} foreach(keys %tmphash);
1082  
1083 }
1084  
1085  
1086  
1087  
1088 =item set_end_format HASH_REF
1089  
1090 =item set_end_format LIST
1091  
1092 Given either a list of keys/values, or a reference to a hash of keys/values,
1093 copy them into the object's Formats list.
1094  
1095 =cut
1096  
1097 sub set_end_format {
1098  
1099 my $self = ref($_[0]) ? shift : $Default_Object;
1100 my %tmphash = ref($_[0]) ? %{$_[0]} : @_;
1101  
1102 $self->{'Formats'}{$_}[1] = $tmphash{$_} foreach(keys %tmphash);
1103  
1104 }
1105  
1106  
1107  
1108  
1109 =item set_format LIST
1110  
1111 Sets the formatting string for one or more formats.
1112  
1113 You should pass a list of keys/values where the keys are the format names and the values
1114 are references to arrays containing the starting and ending formatting strings (in that order)
1115 for that format.
1116  
1117 =cut
1118  
1119 sub set_format {
1120  
1121 my $self = ref($_[0]) ? shift : $Default_Object;
1122 my %tmphash = ref($_[0]) ? %{$_[0]} : @_;
1123  
1124 foreach(keys %tmphash) {
1125 @{$self->{'Formats'}{$_}}[0 .. $#{$tmphash{$_}}] = @{$tmphash{$_}}[0 .. $#{$tmphash{$_}}];
1126 }
1127  
1128 }
1129  
1130  
1131  
1132  
1133 =item get_start_format LIST
1134  
1135 Retrieve the string that is inserted to begin a given format type (starting format string).
1136  
1137 The names are looked for in the following order:
1138  
1139 B<First:> Prefer the names joined by underscore, from most general to least. For example, given
1140 ("Variable", "Scalar"): "Variable" then "Variable_Scalar".
1141  
1142 B<Second:> Then try each name singly, in reverse order. For example,
1143 "Scalar" then "Variable".
1144  
1145 See L<"FORMAT TYPES"> for more information.
1146  
1147 =cut
1148  
1149 sub get_start_format {
1150  
1151 my $self = (@_ && ref($_[0]) ? shift : $Default_Object);
1152  
1153 #
1154 # Prefer the names joined by an underscore from most general to least.
1155 # For example, the parameters:
1156 # 'Identifier', 'Variable', 'Scalar'
1157 # tries 'Identifier' first, then 'Identifier_Variable', and finally 'Identifier_Variable_Scalar'.
1158 #
1159 my $format_id = '';
1160 foreach my $format (@_) {
1161 $format_id .= '_' if($format_id ne '');
1162 $format_id .= $format;
1163 return $self->{'Formats'}{$format_id}[0] if(exists $self->{'Formats'}{$format_id});
1164 }
1165  
1166 #
1167 # Otherwise, look for each name singly, in reverse order.
1168 # In other words, if they say:
1169 # 'Variable', 'Scalar'
1170 # look for 'Scalar' first, then 'Variable'.
1171 #
1172 foreach my $i (-$#_ .. 0) {
1173 return $self->{'Formats'}{$_[$i]}[0] if(exists $self->{'Formats'}{$_[$i]});
1174 }
1175  
1176 #
1177 # Otherwise, return the DEFAULT.
1178 #
1179 return $self->{'Formats'}{'DEFAULT'}[0];
1180  
1181 }
1182  
1183  
1184  
1185  
1186 =item get_end_format LIST
1187  
1188 Retrieve the string that is inserted to end a given format type (ending format string).
1189  
1190 =cut
1191  
1192 sub get_end_format {
1193  
1194 my $self = (@_ && ref($_[0]) ? shift : $Default_Object);
1195  
1196 #
1197 # Prefer the names joined by an underscore from most general to least.
1198 # For example, the parameters:
1199 # 'Identifier', 'Variable', 'Scalar'
1200 # tries 'Identifier' first, then 'Identifier_Variable', and finally 'Identifier_Variable_Scalar'.
1201 #
1202 my $format_id = '';
1203 foreach my $format (@_) {
1204 $format_id .= '_' if($format_id ne '');
1205 $format_id .= $format;
1206 return $self->{'Formats'}{$format_id}[1] if(exists $self->{'Formats'}{$format_id});
1207 }
1208
1209 #
1210 # Otherwise, look for each name in reverse order.
1211 # In other words, if they say:
1212 # 'Variable', 'Scalar'
1213 # look for 'Scalar' first, then 'Variable'.
1214 #
1215 for my $i (-$#_ .. 0) {
1216 return $self->{'Formats'}{$_[$i]}[1] if(exists $self->{'Formats'}{$_[$i]});
1217 }
1218  
1219 #
1220 # Otherwise, return the DEFAULT.
1221 #
1222 return $self->{'Formats'}{'DEFAULT'}[1];
1223  
1224 }
1225  
1226  
1227  
1228  
1229 =item get_format_names_list
1230  
1231 Returns a list of the I<names> of all the Formats defined.
1232  
1233 =cut
1234  
1235 sub get_format_names_list {
1236  
1237 my $self = (@_ && ref $_[0] ? shift : $Default_Object);
1238  
1239 return keys %{$self->{'Formats'}};
1240  
1241 }
1242  
1243  
1244  
1245  
1246 =item get_start_format_values_list
1247  
1248 Returns a list of the I<values> of all the start Formats defined (in the same order
1249 as the names returned by C<get_format_names_list()>).
1250  
1251 =cut
1252  
1253 sub get_start_format_values_list {
1254  
1255 my $self = (@_ && ref $_[0] ? shift : $Default_Object);
1256  
1257 return map $$_[0], values %{$self->{'Formats'}};
1258  
1259 }
1260  
1261  
1262  
1263  
1264 =item get_end_format_values_list
1265  
1266 Returns a list of the I<values> of all the end Formats defined (in the same order
1267 as the names returned by C<get_format_names_list()>).
1268  
1269 =cut
1270  
1271 sub get_end_format_values_list {
1272  
1273 my $self = (@_ && ref $_[0] ? shift : $Default_Object);
1274  
1275 return map $$_[1], values %{$self->{'Formats'}};
1276  
1277 }
1278  
1279  
1280  
1281  
1282 =item format_string LIST
1283  
1284 Formats one or more strings of Perl code. If no strings are specified, defaults to C<$_>.
1285 Returns the list of formatted strings (or the first string formatted if called in scalar context).
1286  
1287 B<Note:> The end of the string is considered to be the end of a line, regardless of whether
1288 or not there is a trailing line-break (but trailing line-breaks will I<not> cause an extra, empty
1289 line).
1290  
1291 B<Another Note:> The function actually uses C<$/> to determine line-breaks, unless C<$/> is
1292 set to C<\n> (newline). If C<$/> I<is> C<\n>, then it looks for the first match
1293 of C<m/\r?\n|\n?\r/> in the string and uses that to determine line-breaks. This is to make
1294 it easy to handle non-unix text. Whatever characters it ends up using as line-breaks are
1295 preserved.
1296  
1297 =cut
1298  
1299 sub format_string {
1300  
1301 my $self = (@_ && ref($_[0]) ? shift : $Default_Object);
1302  
1303 @_ or push @_, $_; # Default to $_ if no strings passed.
1304 splice(@_, 1) unless(wantarray); # Ignore all but the first string passed if called in scalar context.
1305  
1306 my @results = ();
1307  
1308 foreach (@_) {
1309  
1310 my $string = $_; # Have to do this to avoid "Modification of a read-only value..." errors.
1311  
1312 #
1313 # We support any funky kind of line terminator but it must be
1314 # consistant throughout the string (though not between strings).
1315 #
1316 # Unless $/ is custom. Then we use it.
1317 #
1318 my $line_sep = ($/ eq "\n" and $string =~ /(\r\n?|\n\r?)/) ? $1 : $/;
1319  
1320 #
1321 # Don't treat trailing $line_sep's as an extra, blank line.
1322 # But do save it, because we'll re-add it later.
1323 #
1324 my $chomped = ($string =~ s/\Q$line_sep\E$//s) ? $line_sep : '';
1325  
1326 my @lines = split quotemeta($line_sep), $string, -1; # Split on $line_sep...
1327  
1328 #
1329 # Compensate for a "problem" w/ split. Specifically:
1330 #
1331 # $str = <any non-empty string>;
1332 # split('-', $str, -1); # returns ($str)
1333 #
1334 # $str = '';
1335 # split('-', $str, -1) # does _not_ return ($str); returns () instead!
1336 #
1337 @lines = ('') unless(@lines); # Replace empty list with list of 1 empty string.
1338  
1339 $_ = $self->_format_line($_) foreach(@lines); # Format. Note that this modifies @lines!
1340  
1341 push @results, join($line_sep, @lines) . $chomped; # And recombine.
1342  
1343 }
1344  
1345 #
1346 # If we're called in scalar context and not given anything to work with,
1347 # avoid "uninitialized value" warnings by returning an empty string.
1348 #
1349 return (wantarray ? @results : (@results ? $results[0] : ''));
1350  
1351 }
1352  
1353  
1354  
1355  
1356 =item format_token TOKEN, LIST
1357  
1358 Returns TOKEN wrapped in the start and end Formats corresponding to LIST
1359 (as would be returned by C<get_start_format( LIST )> and C<get_end_format( LIST )>,
1360 respectively).
1361  
1362 No syntax checking is done on TOKEN but substitutions defined with C<define_substitution()>
1363 are performed.
1364  
1365 =cut
1366  
1367 sub format_token {
1368  
1369 my $self = (@_ && ref($_[0]) ? shift : $Default_Object);
1370 my $token = shift;
1371  
1372 return '' if(not(defined $token) or $token eq '');
1373  
1374 #
1375 # Do any substitutions that are defined.
1376 #
1377 my %tmp_table = %{$self->{'Substitution Table'}};
1378 my $keys = join('|', map qr/\Q$_\E/, keys %tmp_table);
1379 $token =~ s/($keys)/$tmp_table{$1}/eg if(defined $keys and $keys ne '');
1380  
1381 #
1382 # Pointer to the array containing the formats, so we don't have to look them up twice.
1383 #
1384 my $ra_formats = undef;
1385  
1386 #
1387 # Prefer the names joined by an underscore from most general to least.
1388 # For example, the parameters:
1389 # 'Identifier', 'Variable', 'Scalar'
1390 # tries 'Identifier' first, then 'Identifier_Variable', and finally 'Identifier_Variable_Scalar'.
1391 #
1392 my $format_id = '';
1393 foreach my $format (@_) {
1394 $format_id .= '_' if($format_id ne '');
1395 $format_id .= $format;
1396 if(exists $self->{'Formats'}{$format_id}) {
1397 $ra_formats = $self->{'Formats'}{$format_id};
1398 last;
1399 }
1400 }
1401  
1402 #
1403 # Otherwise, look for each name singly, in reverse order.
1404 # In other words, if they say:
1405 # 'Variable', 'Scalar'
1406 # look for 'Scalar' first, then 'Variable'.
1407 #
1408 unless(defined $ra_formats) {
1409 foreach my $i (-$#_ .. 0) {
1410 if(exists $self->{'Formats'}{$_[$i]}) {
1411 $ra_formats = $self->{'Formats'}{$_[$i]};
1412 last;
1413 }
1414 }
1415 }
1416  
1417 #
1418 # Otherwise, if nothing found, use the DEFAULT.
1419 #
1420 unless(defined $ra_formats) {
1421 $ra_formats = $self->{'Formats'}{'DEFAULT'};
1422 }
1423  
1424 return $ra_formats->[0] . $token . $ra_formats->[1];
1425  
1426 }
1427  
1428  
1429  
1430  
1431  
1432 #===================================================================================================
1433 # Private Methods
1434 #
1435 # Don't call these, please (unless you really want to).
1436 #===================================================================================================
1437  
1438  
1439 #
1440 # Define typeglob aliases for internal use (to save typing).
1441 #
1442 {
1443 no strict 'refs';
1444 my $pkg = __PACKAGE__;
1445  
1446 *{ "$pkg\::_start" } = \&{ "$pkg\::get_start_format" };
1447 *{ "$pkg\::_end" } = \&{ "$pkg\::get_end_format" };
1448 *{ "$pkg\::_wrap" } = \&{ "$pkg\::format_token" };
1449 }
1450  
1451  
1452  
1453 #
1454 # *The* method. :-)
1455 #
1456 # Formats a line of Perl code.
1457 #
1458 # Note:
1459 #
1460 # When I say "line", I mean that the whole string will be treated
1461 # as a *single line*. Regardless of embedded newlines. In fact,
1462 # embedded newlines will be treated as whitespace and will not affect
1463 # the parsing of anything, *EVEN COMMENTS*! Yes, I do realize that
1464 # this is wrong from Perl's point of view. Deal. I might fix that
1465 # later. Who knows. You should be using format_string() anyway.
1466 #
1467 # Another Note:
1468 #
1469 # This method only processes a *single* line (scalar variable) at a time.
1470 # Any aditional parameters will be ignored/lost. Defaults to $_ if no
1471 # parameters are given.
1472 #
1473  
1474 sub _format_line {
1475  
1476 my $self = (@_ && ref($_[0]) ? shift : $Default_Object);
1477 my $line = (@_ ? shift : $_); # Default to $_.
1478  
1479 my $new_line = ''; # Buffer variable for holding the processed part of the line.
1480
1481  
1482  
1483 #
1484 # Check for special case of recursive call indicating an incorrectly tokenized line-piece.
1485 # (ie, don't do start-of-line stuff)
1486 #
1487 unless($self->{'reentrant'}) {
1488  
1489 #
1490 # Increase the line-count.
1491 #
1492 ++$self->{'line_count'};
1493  
1494 #
1495 # After parsing __END__ or __DATA__, we do no more formatting.
1496 #
1497 # Note: I just figured out that Here-Documents actually "override"
1498 # __END__ and __DATA__. By "override", I mean that, though you
1499 # can specify __END__ or __DATA__ on the same line as another
1500 # statement (and it affects anything on the following lines),
1501 # if the statements before the __(END|DATA)__ cause perl to expect
1502 # a Here-Document, it will process the Here-Document lines before
1503 # instating the DATA section. Consider the following code as an
1504 # example (it will print the string "This will be printed\n"):
1505 #
1506 # print << "HEREDOC"; __END__ This is not processed as code but cannot be read from DATA.
1507 # This will be printed
1508 # HEREDOC
1509 # This is data and can be read from DATA.
1510 #
1511 # The formatter will handle such insanity correctly.
1512 #
1513 return $self->_wrap($line, 'DATA') if($self->{'in_data'} and not $self->{'in_heredoc'});
1514  
1515  
1516 #
1517 # Do POD stuff.
1518 #
1519 if(not $self->{'in_string'} and not $self->{'in_heredoc'} and
1520 ($line =~ /^=\w+/ and $self->{'last_token'} =~ /^[;\}]?$/)) {
1521  
1522 $self->{'was_pod'} = $self->{'in_pod'} = TRUE;
1523  
1524 }
1525  
1526 if($self->{'in_pod'}) {
1527  
1528 #
1529 # Check for end of POD (if we're in one).
1530 #
1531 $self->{'in_pod'} = FALSE if($line =~ /^=cut/);
1532  
1533 #
1534 # Format as a comment.
1535 # Note that even the instigator and terminator get formatted as comment.
1536 #
1537 return $self->_wrap($line, 'Comment', 'POD');
1538  
1539 } else {
1540  
1541 $self->{'was_pod'} = FALSE;
1542  
1543 }
1544  
1545  
1546 #
1547 # Check for "here-doc" terminator. It must be the ONLY thing on the line.
1548 #
1549 if($self->{'in_heredoc'} and $line =~ /^$self->{'here_terminator'}[0]$/) {
1550  
1551 #
1552 # We found one. Decrement the here-doc level and remove the terminator from the list.
1553 #
1554 --$self->{'in_heredoc'};
1555 shift @{$self->{'here_terminator'}};
1556  
1557 $self->{'last_token'} = $line;
1558 $self->{'last_token_type'} = 'Quote';
1559  
1560 return $self->_wrap($line, 'Quote'); # Mark here-doc terminators as quotes.
1561  
1562 }
1563  
1564  
1565 #
1566 # Check for "she-bang" line (#!/usr/bin/perl). Note that it *must* be the first line.
1567 #
1568 if($self->{'line_count'} == 1 and $line =~ /^#!/) {
1569 return $self->_wrap($line, 'Directive');
1570 }
1571  
1572 #
1573 # Check for line directives.
1574 #
1575 if($line =~ /^(#\s*line\s+\d+(?:\s+(?:"[^"]*"|\S+))?)(.*)$/) {
1576 return $self->_wrap($1, 'Directive') . $self->_wrap($2, 'Comment', 'Normal');
1577 }
1578  
1579  
1580 #
1581 # String formatting can continue for multiple lines.
1582 #
1583 if(not $self->{'unstable'} and ($self->{'in_string'} or $self->{'in_heredoc'})) {
1584 $new_line = $self->_start('String');
1585 }
1586  
1587 } # End of unless($reentrant) (line-based stuff) block.
1588 else {
1589 if($self->{'reentrant'} > 20) {
1590 my $sr = (caller(1))[3]; # Get the subroutine that called us (caller(0) returns us).
1591 my $ln = (caller(0))[2]; # Get the line that _we_ were called from.
1592  
1593 die "$0: deep recursion ( > 20 levels) caught in "
1594 . __PACKAGE__ . "::_format_line() called from $sr at line $ln\n";
1595 }
1596 }
1597  
1598  
1599 #
1600 # "Optimization" to prevent doing more work on empty strings than we have to.
1601 #
1602 # Note: this _must_ be done _after_ the line-based construct
1603 # checks (the unless($reentrant) stuff).
1604 #
1605 return '' if(not(defined $line) or $line eq '');
1606  
1607 #
1608 # Tokenize the line.
1609 #
1610 my @tokens = _tokenize($line);
1611  
1612  
1613 #
1614 # Main loop for processing the tokenized line.
1615 #
1616 foreach my $token (@tokens) {
1617  
1618 #
1619 # Skip empty tokens as a special case.
1620 #
1621 next if($token eq '');
1622  
1623 $new_line .= $self->_process_token($token);
1624  
1625 }
1626  
1627  
1628  
1629 #
1630 # End of line processing.
1631 #
1632 # EOL processing doesn't occur if we're reentrant because that means that
1633 # we're not actually processing a line (rather a piece of a line).
1634 #
1635 if(not $self->{'reentrant'}) {
1636  
1637 #
1638 # Comments always end at the end of the line.
1639 #
1640 if($self->{'in_comment'}) {
1641  
1642 $self->{'in_comment'} = FALSE;
1643 $new_line .= $self->_end('Comment', 'Normal') unless($self->{'unstable'});
1644  
1645 }
1646  
1647 #
1648 # Strings run to the end of line but the formats should never run *off* the end of line.
1649 # Instablility of formatting makes it pointless to start and end a line-wide format.
1650 #
1651 if(not $self->{'unstable'} and ($self->{'in_string'} or $self->{'in_heredoc'})) {
1652 $new_line .= $self->_end('String');
1653 }
1654  
1655 #
1656 # Setting $self->{'in_heredoc'} is postponed until end of line because the here-doc doesn't
1657 # actually start until the next line. Due to the possibility of "nested" here-doc's,
1658 # we set $self->{'in_heredoc'} to the number of terminators we've gotten.
1659 #
1660 # For example, the code:
1661 #
1662 # print << "END1", "****\n", << "END2";
1663 #
1664 # Creates two terminators ('END1' and 'END2').
1665 #
1666 $self->{'in_heredoc'} = @{$self->{'here_terminator'}};
1667  
1668 }
1669  
1670 #
1671 # Return the formatted line.
1672 #
1673 return $new_line;
1674  
1675 } # End of _format_line()
1676  
1677  
1678  
1679 #
1680 # This _private_ function takes a line passed to it and breaks it into usefull pieces.
1681 # I was using a single regex to do this but it didn't handle precedence properly.
1682 #
1683 sub _tokenize {
1684  
1685 my @tokens = ();
1686  
1687 local $_ = shift; # Line to tokenize.
1688  
1689 #
1690 # Loop through the line, breaking off tokens as we go. The tokens we look for
1691 # (in order of precedence) are:
1692 #
1693 # Comments
1694 # Quotes and quote-like constructs
1695 # Here-Doc initiators
1696 # Subroutine calls
1697 # Variable identifiers
1698 # Whitespace
1699 # Special (backslashed, escaped) characters
1700 # Keywords and builtin functions
1701 # Operators
1702 # Barewords
1703 # Everything else (symbols)
1704 #
1705 LOOP: {
1706 if(m/\G($quotes_re)/gc or # Quote Instigator
1707 m/\G($heredoc_re)/gc or # Here-Doc Instigator
1708 m/\G(__END__|__DATA__)/gc or # Code Terminators
1709 m/\G($keyword_list_re)(?!\w|::)/gc or # Keywords
1710 m/\G($builtin_list_re(?!\w|::)(\s*\()?)/gc or # Built-in Functions
1711 m/\G((?:\\\&|\&)$varchars?\s*(?:$identifier_re|\{))/gc or # Subroutine: &ident
1712 m/\G((?!$operator_list_re\s*\()$identifier_re\s*\()/gc or # Subroutine: ident(
1713 m/\G(\\?$varchars(?:$identifier_re(?:\s*[\[\{])?|\s*\{))/gc or # Variable
1714 m/\G(\\?(?:$varchars(?![@%]))?$builtin_vars_re(?:\s*[\[\{])?)/gc or # Built-in Variables
1715 m/\G(\s+)/gc or # Whitespace
1716 m/\G(\\.)/gc or # Character
1717 m/\G($identifier_re(\s*(=>|:(?!:)))?)/gc or # Bareword / Label
1718 m/\G($number_re)/gc or # Numbers
1719 m/\G($operator_list_re)/gc or # Operator
1720 m/\G(.)/gc # Symbol
1721 ) {
1722 push @tokens, $1;
1723 redo LOOP;
1724 };
1725 };
1726  
1727 return @tokens;
1728  
1729 }
1730  
1731  
1732  
1733 #
1734 # This _private_ function tests a token against any or all of the available Formats,
1735 # and formats it accordingly. It also handles initiating and terminating quoted constructs.
1736 #
1737 sub _process_token {
1738 my $self = (ref($_[0]) ? shift : $Default_Object);
1739 my $token = shift;
1740  
1741  
1742 #**********************************************************
1743 # Preemptive tests.
1744 #**********************************************************
1745  
1746  
1747 #
1748 # After parsing __END__ or __DATA__, we no longer format tokens
1749 # (er, rather, we format them as DATA). Even ones on the same
1750 # line as the __END__ or __DATA__.
1751 #
1752 # This block only handles tokens on the same line as the __END/DATA__.
1753 # See _format_line() for the code that handles the lines following the __END/DATA__ line.
1754 #
1755 if($self->{'in_data'} and not $self->{'in_heredoc'}) {
1756 return $self->_wrap($token, 'DATA');
1757 }
1758  
1759  
1760 #
1761 # The only things we will allow between a class specifier and it's identifier are:
1762 # A comment
1763 # Whitespace
1764 #
1765 if($self->{'awaiting_variable'}) {
1766 unless($token =~ /^$identifier_re/ or $self->{'in_comment'} or $token eq '#' or $token =~ /^\s+$/) {
1767 $self->{'awaiting_variable'} = FALSE;
1768 }
1769 }
1770  
1771  
1772  
1773 #**********************************************************
1774 # Normal token tests.
1775 #**********************************************************
1776  
1777  
1778 #
1779 # Test for special character.
1780 #
1781 if($token =~ /^(\\.)(.+)?$/) {
1782  
1783 my $preamble = $1;
1784 my $postamble = defined($2) ? $2 : '';
1785  
1786 #
1787 # We only do special characters when we're inside an
1788 # interpolated string or a here-doc, or the character
1789 # is super-special (meaning either a double back-slash,
1790 # or a back-slash escaped quote-terminator).
1791 #
1792 unless(
1793 (
1794 $self->{'in_string'} and (
1795 $self->{'quote_type'} =~ /Interpolated/
1796 or $token eq "\\\\"
1797 or $token =~ /\\\Q$self->{'quote_terminator'}\E/
1798 )
1799 )
1800 or $self->{'in_heredoc'}
1801 ) {
1802 $postamble = substr($preamble, 1, 1) . $postamble; # The character (plus other stuff)
1803 $preamble = substr($preamble, 0, 1); # The backslash
1804  
1805 ++$self->{'reentrant'};
1806 $preamble = $self->_format_line($preamble);
1807 $postamble = $self->_format_line($postamble);
1808 --$self->{'reentrant'};
1809  
1810 return $preamble . $postamble;
1811 }
1812  
1813 $self->{'last_token'} = $preamble;
1814 $self->{'last_token_type'} = 'Character';
1815  
1816 if(defined $postamble) {
1817 ++$self->{'reentrant'};
1818 $postamble = $self->_format_line($postamble);
1819 --$self->{'reentrant'};
1820 }
1821  
1822 return $self->_wrap($preamble, 'Character') . $postamble;
1823  
1824 }
1825  
1826  
1827  
1828 #
1829 # Test for comment.
1830 #
1831 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and ($token eq '#' or $self->{'in_comment'})) {
1832  
1833 #
1834 # Start a new comment.
1835 #
1836 if($token eq '#' and not $self->{'in_comment'}) {
1837 $self->{'in_comment'} = TRUE;
1838 if($self->{'unstable'}) {
1839 return $self->_wrap($token, 'Comment', 'Normal')
1840 } else {
1841 return $self->_start('Comment', 'Normal') . $self->_wrap($token, '')
1842 }
1843 return (
1844 $self->{'unstable'}
1845 ?
1846 $self->_wrap($token, 'Comment', 'Normal')
1847 :
1848 $self->_start('Comment', 'Normal') . $self->_wrap($token, '')
1849 );
1850 }
1851  
1852 return (
1853 $self->{'unstable'}
1854 ?
1855 $self->_wrap($token, 'Comment', 'Normal')
1856 :
1857 $self->_wrap($token, '')
1858 );
1859 }
1860  
1861  
1862 #
1863 # Test for (DATA Instigators) special tokens that signal the end of executable code.
1864 # We will format nothing after we encounter one of these.
1865 #
1866 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and ($token eq '__END__' or $token eq '__DATA__')) {
1867  
1868 $self->{'in_data'} = TRUE;
1869  
1870 return $self->_wrap($token, 'CodeTerm');
1871  
1872 }
1873  
1874  
1875  
1876 #
1877 # Quote-related tests.
1878 #
1879  
1880 #
1881 # Check for nested quotes.
1882 #
1883 # If our quotes are bracketing and we encounter another Instigator inside
1884 # the string, we require a matching terminator before we end the string.
1885 #
1886 # For example:
1887 #
1888 # qq{ Because of this -> { the string does not end here -> }, it ends here -> }
1889 #
1890 if($self->{'in_string'} and
1891 $token !~ /\Q$self->{'quote_terminator'}\E/ and
1892 $token =~ /\Q$self->{'quote_instigator'}\E/
1893 ) {
1894  
1895 #
1896 # Token matches the quote instigator but not the terminator so it must be
1897 # a bracketing type of quote. We need to increase the count of nested
1898 # quotes. We do not do anything else w/ the token, just let it be processed
1899 # by subsequent code.
1900 #
1901 ++$self->{'nested_quote'};
1902  
1903 }
1904
1905 #
1906 # Check for second part of Multipart quoted construct (if we're waiting for one).
1907 #
1908 # Comments (handled above) and whitespace are ignored between parts of a Multipart construct.
1909 #
1910 if($self->{'awaiting_multi'} and $token =~ /\S/) {
1911
1912 #
1913 # Anything other than a comment or whitespace causes us to stop waiting
1914 # for the second part. (It's technically an error.)
1915 #
1916 $self->{'awaiting_multi'} = FALSE;
1917  
1918 if($token =~ /\Q$self->{'quote_instigator'}\E/) {
1919  
1920 #
1921 # We have found the second part!
1922 #
1923  
1924 $self->{'last_token'} = $token;
1925 $self->{'last_token_type'} = 'Quote';
1926  
1927 $self->{'found_multi'} = TRUE;
1928 $self->{'in_string'} = TRUE;
1929  
1930 return
1931 $self->_wrap($token, 'Quote')
1932 . ($self->{'unstable'} ? '' : $self->_start('String'));
1933  
1934 }
1935  
1936 }
1937  
1938 #
1939 # Check for options to an optioned quoted construct.
1940 #
1941 if($self->{'awaiting_options'}) {
1942  
1943 $self->{'awaiting_options'} = FALSE;
1944  
1945 if($token =~ /^([cgimosexd]+|s.*)$/) {
1946  
1947 my $postamble = '';
1948  
1949 $self->{'last_token'} = $token;
1950 $self->{'last_token_type'} = 'Quote';
1951  
1952 #
1953 # Check for lone `s' option incorrectly tokenized as a s/// instigator
1954 # and recursively re-process it if necessary.
1955 #
1956 if($token =~ /^(s)([^cgimoxed]*)$/) {
1957 $token = $1;
1958 $self->{'last_token'} = $token;
1959  
1960 ++$self->{'reentrant'};
1961 $postamble = $self->_format_line($2);
1962 --$self->{'reentrant'};
1963 }
1964  
1965 return $self->_wrap($token, 'Quote') . $postamble;
1966  
1967 }
1968  
1969 }
1970  
1971  
1972  
1973 #
1974 # Test for quote instigator (string-start).
1975 #
1976 # Note: The forward-slash and question mark characters (/ and ?) as quote instigators are
1977 # pretty difficult to disambiguate and if it becomes a problem, I may just leave them
1978 # out. Though I'd rather not.
1979 #
1980 # The strategy we use here is to only make them operators (division or ternary conditional,
1981 # respectively) if they immediately follow a closing bracket-type-thingy ([\)\]\}]),
1982 # quote, variable, literal number, bareword, or a builtin or user function other than
1983 # print, split, grep, or map.
1984 #
1985 # Note that the function/subroutine situation is still ambiguous: some functions should
1986 # cause them to start a quote and some should cause them to be operators, depending on
1987 # the effective prototype of the function (eg: sin vs. time).
1988 #
1989 # There are also some other ambiguities, as well. Such as when they follow certain types
1990 # of symbols or operators.
1991 #
1992 # See http://perlmonks.org/index.pl?node=On+Parsing+Perl for more information.
1993 #
1994 if((not $self->{'in_string'}) and (not $self->{'in_heredoc'}) and ($token =~ /^$quotes_re/ or
1995 ($token =~ m{^[/?]} and
1996 $self->{'last_token'} !~ /[\)\]\}]$/ and # Closing bracket-type-thingy. Might add > later.
1997 $self->{'last_token_type'} ne 'Quote' and
1998 $self->{'last_token_type'} ne 'Variable' and
1999 $self->{'last_token_type'} ne 'Number' and
2000 $self->{'last_token_type'} ne 'Bareword' and
2001 $self->{'last_token_type'} ne 'Subroutine' and
2002 ($self->{'last_token_type'} ne 'Builtin' or
2003 $self->{'last_token'} eq 'print' or
2004 $self->{'last_token'} eq 'split' or
2005 $self->{'last_token'} eq 'grep' or
2006 $self->{'last_token'} eq 'map')))) {
2007  
2008 #
2009 # Create a copy of $token so we can modify it with impunity.
2010 # (Specifically for splitting up of '/='-like constructs (see below).)
2011 #
2012 my $quote = $token;
2013  
2014 #
2015 # In certain contexts (see above), the token '/=' or similar will be considered
2016 # the start of a pattern-match quote. We have to split off the extra and process
2017 # it as part of the string so we store it in $postamble and re-process it later
2018 # (see the return at the bottom of this block).
2019 #
2020 my $postamble = '';
2021 if($quote =~ m{^([/?])(.+)$}) {
2022  
2023 #
2024 # Store the split-off quote instigator (/) and the postamble.
2025 #
2026 $quote = $1;
2027 $postamble = $2;
2028  
2029 }
2030  
2031 #
2032 # The quote-instigator is the last character of the token.
2033 #
2034 $self->{'quote_instigator'} = substr $quote, -1;
2035  
2036 #
2037 # Define special terminators for bracketing characters.
2038 # Otherwise, use the same character as the instigator.
2039 #
2040 $self->{'quote_terminator'} =
2041 { '{' => '}',
2042 '(' => ')',
2043 '[' => ']',
2044 '<' => '>',
2045 }->{$self->{'quote_instigator'}} || $self->{'quote_instigator'};
2046  
2047 #
2048 # The type will be matched later on like so:
2049 #
2050 # if($self->{'quote_type'} =~ /Interpolated/) { # Is it interpolated?
2051 #
2052 # Note: Some of the quoted constructs are interpolated or non-interpolated
2053 # depending on the instigator/terminator character used. Specifically,
2054 # if a single quote is used as the delimter for qx, m, qr, and s, they
2055 # are *not* interpolated. Otherwise, they *are* interpolated.
2056 #
2057 $self->{'quote_type'} = (
2058 ($quote =~ /'|q[^qxwr]/ && 'Literal') or
2059  
2060 ($quote =~ /"|qq/ && 'Literal, Interpolated') or
2061  
2062 ($quote =~ /`|qx/ && 'Command' .
2063 ($self->{'quote_instigator'} eq q(') ? '' : ', Interpolated'))
2064 or
2065 ($quote =~ /qw/ && 'Word list') or
2066  
2067 ($quote =~ /m|^[\/?]/ && 'Pattern, Matching, Optioned' .
2068 ($self->{'quote_instigator'} eq q(') ? '' : ', Interpolated'))
2069 or
2070 ($quote =~ /qr/ && 'Pattern, Optioned' .
2071 ($self->{'quote_instigator'} eq q(') ? '' : ', Interpolated'))
2072 or
2073 ($quote =~ /s/ && 'Pattern, Substitution, Optioned, Multipart' .
2074 ($self->{'quote_instigator'} eq q(') ? '' : ', Interpolated'))
2075 or
2076 ($quote =~ /tr|y/ && 'Transliteration, Optioned, Multipart') or
2077  
2078 'Error: Quote-Type Not Found'
2079 );
2080  
2081 $self->{'in_string'} = TRUE;
2082  
2083 $self->{'last_token'} = $quote;
2084 $self->{'last_token_type'} = 'Quote';
2085  
2086 #
2087 # Post-process the $postamble if necessary.
2088 #
2089 if($postamble ne '') {
2090  
2091 ++$self->{'reentrant'};
2092 $postamble = $self->_format_line($postamble);
2093 --$self->{'reentrant'};
2094  
2095 }
2096  
2097 return
2098 $self->_wrap($quote, 'Quote')
2099 . $self->_start('String')
2100 . $postamble
2101 . ($self->{'unstable'} ? $self->_end('String') : '');
2102  
2103 } # End test for Quote Instigator
2104  
2105  
2106 #
2107 # Check for Quote Terminator. Here-doc terminators are line-based
2108 # constructs, and thus are checked in _format_line().
2109 #
2110 if($self->{'in_string'} and
2111 $token =~ /^(.*?)(?<!\\)(\Q$self->{'quote_terminator'}\E)(.*)$/
2112 ) {{ # Note double braces. That's so we can use last (just below) to exit the if.
2113  
2114 #
2115 # If our quotes are bracketing and we've encountered another Instigator inside
2116 # the string, we require a matching terminator before we end the string.
2117 #
2118 # For example:
2119 #
2120 # qq{ Because of this -> { the string does not end here -> }, it ends here -> }
2121 #
2122 if($self->{'nested_quote'} > 0) {
2123 --$self->{'nested_quote'};
2124 last; # Exit enclosing if block (with double braces, not if nested_quote)
2125 }
2126  
2127 #
2128 # This block terminates (possibly just the first part of) a quoted structure.
2129 #
2130  
2131 $self->{'last_token'} = $token;
2132 $self->{'last_token_type'} = 'Quote';
2133  
2134 #
2135 # Note on $preamble and $postamble:
2136 #
2137 # There might be extra "stuff" attached to the beginning or end of
2138 # our quote terminator that should be formatted separately. This
2139 # is due to the fact that the tokenizer doesn't know when it's tokenizing
2140 # that it should be breaking things up based on quote teriminators.
2141 #
2142 # For example, the string '"$"' tokenizes as:
2143 #
2144 # '"' (quote-start), '$"' (built-in variable)
2145 #
2146 # It should actually be parsed:
2147 #
2148 # '"' (quote-start), '$' (error in Perl), '"' (quote-end)
2149 #
2150 # Another example; the string 's sjsjs' tokenizes as:
2151 #
2152 # 's s' (substitution-start), 'jsjs' (bareword)
2153 #
2154 # Instead of:
2155 #
2156 # 's s' (sub.-start), 'j' (string), 's' (sub.-end 1), 'j' (string), 's' (sub.-end)
2157 #
2158 my $preamble = $1;
2159 my $terminator = $2;
2160 my $new_string = ''; # May or may not be set; depends on if we find the second part of a multi-part.
2161 my $postamble = $3;
2162  
2163 ++$self->{'reentrant'};
2164 $preamble = $self->_format_line($preamble); # Must process preamble before doing multi-part logic.
2165 --$self->{'reentrant'}; # Otherwise, we wouldn't know if it should be in_string.
2166
2167 #
2168 # Handle multipart quotes.
2169 #
2170 if($self->{'found_multi'}) {
2171  
2172 #
2173 # If we've already found Multipart, we shouldn't look for it.
2174 #
2175 $self->{'found_multi'} = FALSE;
2176 $self->{'in_string'} = FALSE;
2177 $self->{'awaiting_options'} = TRUE if($self->{'quote_type'} =~ /Optioned/);
2178  
2179 } elsif($self->{'quote_type'} =~ /Multipart/) {
2180  
2181 #
2182 # If we haven't already found the Multipart, we should check for it immediately.
2183 #
2184  
2185 if($self->{'quote_instigator'} eq $self->{'quote_terminator'}) {
2186  
2187 #
2188 # We found it.
2189 #
2190  
2191 $self->{'found_multi'} = TRUE;
2192 $self->{'in_string'} = TRUE;
2193  
2194 $new_string = $self->_start('String'); # Unstable logic done below (when this is used).
2195  
2196 } else {
2197  
2198 #
2199 # We didn't find it so we start waiting.
2200 #
2201  
2202 $self->{'awaiting_multi'} = TRUE;
2203 $self->{'in_string'} = FALSE;
2204  
2205 }
2206  
2207 } else {
2208  
2209 #
2210 # This quoted construct isn't Multiparted so we just end it.
2211 #
2212  
2213 $self->{'in_string'} = FALSE;
2214 $self->{'awaiting_options'} = TRUE if($self->{'quote_type'} =~ /Optioned/);
2215 }
2216  
2217 #
2218 # Do the formatting and re-processing.
2219 #
2220 my $formatted_token;
2221 $formatted_token .= $preamble;
2222 $formatted_token .= $self->_end('String') unless($self->{'unstable'});
2223 $formatted_token .= $self->_wrap($terminator, 'Quote');
2224 $formatted_token .= $new_string unless($self->{'unstable'});
2225 #
2226 # Reprocess $postamble now that multi-part logic has been done.
2227 #
2228 ++$self->{'reentrant'};
2229 $formatted_token .= $self->_format_line($postamble);
2230 --$self->{'reentrant'};
2231  
2232 return $formatted_token;
2233  
2234 }} # End check for Quote Terminator. Note double braces.
2235  
2236  
2237 #
2238 # Test for Here-Doc (type of quote).
2239 #
2240 # Note: This maybe should be combined with the test
2241 # for Quote Instigator or possibly Operator. Maybe.
2242 #
2243 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and $token =~ /^$heredoc_re$/) {
2244  
2245 #
2246 # Here-Document instigators are ambiguous in some situations. The
2247 # rule used here is that it's considered a Here-Document instigator
2248 # unless it follows a Variable, Bareword, or Quote (string). If it
2249 # does follow one of those, then it's probably a binary left shift
2250 # operator and the token should be reprocessed.
2251 #
2252 # This rule doesn't cover every situation (functions with certain
2253 # prototypes should also cause it to be considered a shift operator,
2254 # and there's probably other situtations) but it should cover the
2255 # most common ones.
2256 #
2257 if($self->{'last_token_type'} =~ /Quote|Bareword|Variable/) {
2258  
2259 my ($preamble, $postamble) = $token =~ /^(<<)(.*)$/;
2260  
2261 $preamble = $self->_wrap($preamble, 'Operator');
2262  
2263 $self->{'last_token'} = $token;
2264 $self->{'last_token_type'} = 'Quote';
2265  
2266 ++$self->{'reentrant'};
2267 $postamble = $self->_format_line($postamble);
2268 --$self->{'reentrant'};
2269  
2270 return $preamble . $postamble;
2271  
2272 }
2273  
2274 $self->{'last_token'} = $token;
2275 $self->{'last_token_type'} = 'Quote';
2276  
2277 #
2278 # Push only the word part of the here-doc terminator (not any optional quotes) onto the list.
2279 # Note that this word part *can* be an empty string.
2280 #
2281 my ($term) = $token =~ /(\w+)/;
2282 $term = '' unless(defined $term);
2283 push @{$self->{'here_terminator'}}, $term;
2284  
2285 $self->{'quote_type'} = 'Here-Document, Interpolated';
2286  
2287 return $self->_wrap($token, 'Quote');
2288  
2289 }
2290  
2291 #
2292 # End of quote-related tests.
2293 #
2294  
2295  
2296 #
2297 # Test for keyword.
2298 #
2299 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and $token =~ /^(?:$keyword_list_re)$/) {
2300  
2301 $self->{'last_token'} = $token;
2302 $self->{'last_token_type'} = 'Keyword';
2303  
2304 #
2305 # Test for formats. Treat them like here-docs (they terminate w/ line =~ /^.$/).
2306 #
2307 if($token eq 'format') {
2308 push @{$self->{'here_terminator'}}, '.';
2309 }
2310  
2311 return $self->_wrap($token, 'Keyword');
2312  
2313 }
2314  
2315  
2316  
2317 #
2318 # Test for builtin functions.
2319 #
2320 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and $token =~ /^($builtin_list_re)(\s*)(\(?)$/ and
2321 $self->{'last_token'} ne 'sub' and $self->{'last_token'} ne '->') {
2322  
2323 $self->{'last_token'} = $token;
2324  
2325 if($3) {
2326 $self->{'last_token_type'} = 'Symbol';
2327 } else {
2328 $self->{'last_token_type'} = 'Builtin';
2329 }
2330  
2331 return
2332 $self->_wrap($1, 'Builtin', ($3 ? 'Function' : 'Operator')) .
2333 $self->_wrap($2, 'Whitespace') .
2334 $self->_wrap($3, 'Symbol');
2335  
2336 }
2337  
2338  
2339  
2340 #
2341 # Test for subroutine.
2342 #
2343 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and
2344 ($token =~ /^(\&$varchars?)(\s*)($identifier_re)?()((?(4)|\{))$/ or
2345 $token =~ /^()()($identifier_re)(\s*)(\()$/ or
2346 ($token =~ /^()()($identifier_re)()()$/ and
2347 ($self->{'last_token'} eq 'sub' or $self->{'last_token'} eq '->')))) {
2348  
2349 my $ampersand = $1 || '';
2350 my $whitespace1 = $2 || '';
2351 my $identifier = $3 || '';
2352 my $whitespace2 = $4 || '';
2353 my $parenthesis = $5 || '';
2354  
2355 #
2356 # Check for package imports with parameters (they accidentally get tokenized
2357 # as subroutine calls.
2358 #
2359 if($self->{'last_token'} =~ /^(require|use|no)$/ and $ampersand eq '') {
2360  
2361 $self->{'last_token'} = $token;
2362  
2363 if($parenthesis) {
2364 $self->{'last_token_type'} = 'Symbol';
2365 } else {
2366 $self->{'last_token_type'} = 'Subroutine';
2367 }
2368  
2369 return
2370 $self->_wrap($identifier, 'Package')
2371 . $self->_wrap($whitespace2, 'Whitespace')
2372 . $self->_wrap($parenthesis, 'Symbol');
2373  
2374 }
2375  
2376 $self->{'last_token'} = $token;
2377  
2378 if($parenthesis) {
2379 $self->{'last_token_type'} = 'Symbol';
2380 } else {
2381 $self->{'last_token_type'} = 'Subroutine';
2382 }
2383  
2384 return (
2385 (
2386 $self->{'unstable'}
2387 ?
2388 $self->_wrap($ampersand, 'Subroutine')
2389 . $self->_wrap($whitespace1, 'Whitespace')
2390 . $self->_wrap($identifier, 'Subroutine')
2391 :
2392 $self->_wrap($ampersand
2393 . $self->_wrap($whitespace1, 'Whitespace')
2394 . $identifier, 'Subroutine') # _wrap($amp . _wrap($white1, 'WS') . $ident, 'SR')
2395 )
2396 . $self->_wrap($whitespace2, 'Whitespace')
2397 . $self->_wrap($parenthesis, 'Symbol')
2398 );
2399  
2400 } # End tests for subroutine.
2401  
2402  
2403 #
2404 # Test for variable.
2405 #
2406 # Note in the conditional for built-in variables (1st line):
2407 # First we test for match, then, still in the conditional, we split it into the needed
2408 # positional variables ($[1-6]) with the next match (3rd line). Once the token matches
2409 # the first pattern (1st line), it should *always* match the second (2nd line) as well.
2410 #
2411 if(($token =~ /^$varchars?$builtin_vars_re$/ and
2412 $token =~ /^($varchars)()(.+?)()([\[\{]?)$/) or # Split after match built-in variables.
2413 $token =~ /^($varchars)(\s*)($identifier_re)(\s*)([\[\{]?)$/ or
2414 $token =~ /^($varchars)(\s*)(\{)$/ or
2415 $token =~ /^([\$\@\%\*])$/ or # Extra leading empty parens in regex to shift positional variables
2416 ($self->{'awaiting_variable'} and $token =~ /^()()($identifier_re)$/) ) {{ # Note double braces.
2417  
2418 #
2419 # The braces for this if() statement are doubled so that we can use the 'last'
2420 # loop control statement to break out of the entire conditional block in the
2421 # first conditional block below (the test for partial variables within Patterns).
2422 #
2423
2424 my $class = '';
2425 my $class_specifier = $1 || '';
2426 my $whitespace1 = $2 || '';
2427 my $identifier = (defined $3 ? $3 : ''); # Special case. ident. might be 0 ($0)
2428 my $whitespace2 = $4 || '';
2429 my $subscript = $5 || '';
2430 my $postamble = '';
2431  
2432 #
2433 # Don't format variables inside of non-interpolated strings.
2434 # Remember to do this _after_ using the positional variables!
2435 #
2436 last if($self->{'in_string'} and $self->{'quote_type'} !~ /Interpolated/);
2437  
2438 #
2439 # We don't format partial variables (ie, ones without identifiers) inside
2440 # of Regex (Pattern) strings.
2441 #
2442 if($identifier eq '' and $self->{'in_string'} and $self->{'quote_type'} =~ /Pattern/) {
2443  
2444 #
2445 # This 'last' only works because of the doubled braces around the larger conditional
2446 # block (if($token =~ ...) (see above). This trick is explained in the Perl syntax
2447 # documentation (perlsyn). See `man perlsyn` under the Loop Control section for
2448 # more information.
2449 #
2450  
2451 last;
2452  
2453 }
2454
2455  
2456 #
2457 # Hashes and typeglobs are not interpolated into strings
2458 # or here-documents, so we need to strip any leading hash
2459 # hash symbols (%) or typeglob symbols (*) and not consider
2460 # the token a variable.
2461 #
2462 if(($self->{'in_string'} or $self->{'in_heredoc'}) and $class_specifier =~ /^([\%\*]+)/) {
2463  
2464 my $preamble = $1;
2465  
2466 #
2467 # If the only thing there is the hash/typeglob character, just return.
2468 #
2469 if(length $token == length $preamble) {
2470 return (
2471 $self->{'unstable'}
2472 ?
2473 $self->_wrap($token, 'String') # Reinstate string formatting if unstable.
2474 :
2475 $self->_wrap($token, '') # Format as '' to ensure substitutions.
2476 );
2477 }
2478  
2479 #
2480 # Reinstate string formatting if we're unstable.
2481 #
2482 $preamble = $self->_wrap($preamble, $self->{'unstable'} ? 'String' : '');
2483  
2484 #
2485 # Otherwise, strip off the leading symbol and reprocess the rest (via recursive call).
2486 #
2487 ++$self->{'reentrant'};
2488 my $result = $preamble . $self->_format_line(substr($token,1));
2489 --$self->{'reentrant'};
2490  
2491 return $result;
2492  
2493 }
2494  
2495 if($self->{'awaiting_variable'}) {
2496  
2497 $self->{'awaiting_variable'} = FALSE;
2498  
2499 # Adjust the class for a subscript or use the class determined before.
2500 $class = (
2501 $subscript
2502 ?
2503 {'[' => 'Array', '{' => 'Hash'}->{$subscript}
2504 :
2505 $self->{'awaiting_class'}
2506 );
2507  
2508 ++$self->{'reentrant'};
2509 my $return_val =
2510 $self->_wrap($whitespace1, 'Whitespace')
2511 . $self->_wrap($class_specifier . $identifier, 'Variable', $class)
2512 . $self->_wrap($whitespace2, 'Whitespace')
2513 . $self->_format_line($subscript); # Reprocess it, in case we're in a string.
2514 --$self->{'reentrant'};
2515  
2516 $self->{'last_token'} = $token;
2517 $self->{'last_token_type'} = 'Variable';
2518  
2519 return $return_val;
2520  
2521 } elsif($class_specifier) {
2522  
2523 # Adjust the class for a subscript or use the class specifier.
2524 $class = (
2525 $subscript ?
2526 {'[' => 'Array', '{' => 'Hash'}->{$subscript}
2527 :
2528 {'$' => 'Scalar',
2529 '@' => 'Array',
2530 '%' => 'Hash',
2531 '*' => 'Typeglob',
2532 }->{substr($class_specifier,0,1)}
2533 );
2534  
2535 $class = 'INVALID CLASS' unless(defined $class);
2536  
2537 #
2538 # The special case of $identifier eq '{' (as in $token eq '${' for '${foo}')
2539 # has to be handled. In case we're inside a string (we want symbols formatted
2540 # consistantly, even if it's consistantly wrong), we need to recursively
2541 # re-process the identifier.
2542 #
2543 # The other special case of $identifier eq '' means that we should start waiting
2544 # for an identifier on a following line.
2545 #
2546 if($identifier eq '{') {
2547  
2548 ++$self->{'reentrant'};
2549 $postamble = $self->_format_line($identifier);
2550 --$self->{'reentrant'};
2551  
2552 $identifier = '';
2553  
2554 } elsif($identifier eq '') {
2555  
2556 $self->{'awaiting_variable'} = TRUE;
2557 $self->{'awaiting_class'} = $class;
2558  
2559 }
2560  
2561 ++$self->{'reentrant'};
2562 my $return_val =
2563 $self->_wrap($whitespace1, 'Whitespace')
2564 . $self->_wrap($class_specifier . $identifier, 'Variable', $class)
2565 . $self->_wrap($whitespace2, 'Whitespace')
2566 . $self->_format_line($subscript) # Reprocess in case we're in a string.
2567 . $postamble;
2568 --$self->{'reentrant'};
2569  
2570 $self->{'last_token'} = $token;
2571 $self->{'last_token_type'} = 'Variable';
2572  
2573 return $return_val;
2574  
2575 }
2576  
2577 #
2578 # If we're not awaiting a variable identifier and there's no class specifier
2579 # ($, @, or %), then it's probably actually a bareword or keyword or something.
2580 #
2581 # So we just let this if() statement fall through.
2582 #
2583  
2584 }} # End test for Variable. Note double braces.
2585  
2586  
2587  
2588 #
2589 # Test for whitespace.
2590 #
2591 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and $token =~ /^\s+$/) {
2592 return $self->_wrap($token, 'Whitespace');
2593 }
2594  
2595  
2596  
2597 #
2598 # Test for operator.
2599 #
2600 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and $token =~ /^(?:$operator_list_re)$/) {
2601  
2602 $self->{'last_token'} = $token;
2603 $self->{'last_token_type'} = 'Operator';
2604  
2605 return $self->_wrap($token, 'Operator');
2606  
2607 }
2608  
2609  
2610  
2611 #
2612 # Test for number.
2613 #
2614 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and $token =~ /^$number_re$/) {
2615  
2616 $self->{'last_token'} = $token;
2617 $self->{'last_token_type'} = 'Number';
2618  
2619 return $self->_wrap($token, 'Number');
2620  
2621 }
2622  
2623  
2624 #
2625 # Test for bareword (or package name, which is a type of bareword).
2626 #
2627 if(not($self->{'in_string'} or $self->{'in_heredoc'}) and
2628 $token =~ /^($identifier_re)((\s*)(=>|:(?!:)))?$/) {
2629  
2630 #
2631 # If $token =~ /BAREWORD\s*=>/, then BAREWORD should be formatted as a string.
2632 #
2633 # If $token =~ /BAREWORD\s*:/, then it's actually a label (elsif block).
2634 #
2635 if(defined $2 and substr($2, -1) ne ':') { # It's a string.
2636  
2637 $self->{'last_token'} = $token;
2638 $self->{'last_token_type'} = 'String';
2639  
2640 ++$self->{'reentrant'};
2641 my $result = $self->_wrap($1, 'String') . $self->_format_line($2);
2642 --$self->{'reentrant'};
2643  
2644 return $result;
2645  
2646 } elsif(defined $2) { # It's a label.
2647  
2648 $self->{'last_token'} = $token;
2649 $self->{'last_token_type'} = 'String';
2650  
2651 return $self->_wrap($1, 'Label') . $self->_wrap($3, 'Whitespace') . $self->_wrap($4, 'Label');
2652  
2653 }
2654  
2655 if($self->{'last_token'} =~ /^(require|package|use|no)$/) {
2656  
2657 #
2658 # This bareword is actually a package name.
2659 #
2660 $self->{'last_token'} = $token;
2661 $self->{'last_token_type'} = 'Package';
2662 return $self->_wrap($token, 'Package');
2663  
2664 } elsif(substr($self->{'last_token'}, -1) eq '{' and $self->{'last_token_type'} eq 'Variable') {
2665  
2666 #
2667 # This bareword is the subscript to a hash and should be formatted as a string.
2668 #
2669 $self->{'last_token'} = $token;
2670 $self->{'last_token_type'} = 'String';
2671 return $self->_wrap($token, 'String');
2672  
2673 } else {
2674  
2675 $self->{'last_token'} = $token;
2676 $self->{'last_token_type'} = 'Bareword';
2677 return $self->_wrap($token, 'Bareword');
2678  
2679 }
2680  
2681 }
2682  
2683  
2684 #
2685 # Test for symbol.
2686 #
2687 if(not($self->{'in_string'} or $self->{'in_heredoc'})) {
2688  
2689 #
2690 # The pound symbol (#) is special in that it starts a normal comment.
2691 #
2692 if($token eq '#') {
2693  
2694 $self->{'last_token'} = $token;
2695 $self->{'last_token_type'} = 'Comment';
2696  
2697 $self->{'in_comment'} = TRUE;
2698  
2699 return (
2700 $self->{'unstable'}
2701 ?
2702 $self->_wrap($token, 'Comment', 'Normal')
2703 :
2704 $self->_start('Comment', 'Normal') . $self->_wrap($token, '')
2705 );
2706  
2707 }
2708  
2709 $self->{'last_token'} = $token;
2710 $self->{'last_token_type'} = 'Symbol';
2711  
2712 return $self->_wrap($token, 'Symbol');
2713  
2714 }
2715  
2716  
2717 #
2718 # No matches were made so we just return the token (reinstating
2719 # string formatting if formatting is unstable).
2720 #
2721 return (
2722 (($self->{'in_string'} or $self->{'in_heredoc'}) and $self->{'unstable'})
2723 ?
2724 $self->_wrap($token, 'String') # If we're unstable, we need to reinstate string formatting.
2725 :
2726 $self->_wrap($token, '') # Format as '' to ensure substitutions; this causes no markup.
2727 );
2728  
2729 } # End _process_token()
2730  
2731  
2732  
2733  
2734 TRUE; # Return true to indicate success.
2735  
2736 =pod
2737  
2738 =back
2739  
2740 =head1 KNOWN ISSUES or LIMITATIONS
2741  
2742 =over 4
2743  
2744 =item *
2745  
2746 Barewords used as keys to a hash are formatted as strings. This is Good. They should
2747 not be, however, if they are not the only thing within the curly braces. That can be
2748 fixed.
2749  
2750 =item *
2751  
2752 This version does not handle formats (see L<perlform(1)|perlform/1>) very well. It treats them as
2753 Here-Documents and ignores the rules for comment lines, as well as the fact that picture
2754 lines are not supposed to be interpolated. Thus, your picture lines will look strange
2755 with the '@'s being formatted as array variables (albeit, invalid ones).
2756 Ideally, it would also treat value lines as normal Perl code and format accordingly.
2757 I think I'll get to the comment lines and non-interpolating picture lines first.
2758 If/When I do get this fixed, I will most likely add a format type of 'Format' or something,
2759 so that they can be formatted differently, if so desired.
2760  
2761 =item *
2762  
2763 This version does not handle Regular Expression significant characters. It simply treats
2764 Regular Expressions as interpolated strings.
2765  
2766 =item *
2767  
2768 User-defined subroutines, called without parentheses, are formatted as barewords.
2769 This is because there is no way to tell them apart from barewords without parsing
2770 the code, and would require us to go as far as perl does when doing the C<-c> check
2771 (ie, executing BEGIN and END blocks and the like). That's not going to happen.
2772  
2773 =item *
2774  
2775 If you are indexing (subscripting) an array or hash, the formatter tries to figure
2776 out the "real" variable class by looking at how you index the variable. However, if
2777 you do something funky (but legal in Perl) and put line-breaks or comments between
2778 the variable class character ($) and your identifier, the formatter will get confused
2779 and treat your variable as a scalar. Until it finds the index character. Then it
2780 will format the scalar class character ($) as a scalar and your identifier as
2781 the "correct" class.
2782  
2783 =item *
2784  
2785 If you put a line-break between your variable identifier and it's indexing character (see
2786 above), which is also legal in Perl, the formatter will never find it and treat your
2787 variable as a scalar.
2788  
2789 =item *
2790  
2791 If you put a line-break between a bareword hash-subscript and the hash variable, or between
2792 a bareword and its associated C<=E<gt>> operator, the bareword will not be formatted correctly
2793 (as a string). I<(Noticing a pattern here?)>
2794  
2795 =back
2796  
2797 =head1 AUTHOR
2798  
2799 Cory Johns B<darkness@yossman.net>
2800  
2801 Copyright (c) 2001 Cory Johns. This library is free software; you can
2802 redistribute and/or modify it under the same conditions as Perl itself.
2803  
2804 =head1 TO DO
2805  
2806 =over 4
2807  
2808 =item 1
2809  
2810 Improve handling of regular expressions. Add support for regexp-special characters.
2811 Recognize the /e option to the substitution operator (maybe).
2812  
2813 =item 2
2814  
2815 Improve handling of formats. Don't treat format definitions as interpolating. Handle
2816 format-comments. Possibly format value lines as normal Perl code.
2817  
2818 =item 3
2819  
2820 Create in-memory deep-copy routine to replace C<eval(Data::Dumper)> deep-copy.
2821  
2822 =item 4
2823  
2824 Generalize state transitions (C<reset()> and, in the future, C<copy_state()>) to use
2825 non-hard-coded keys and values for state variables. Probably will extrapolate them into an
2826 overloadable hash, and use the aforementioned deep-copy to assign them.
2827  
2828 =item 5
2829  
2830 Create a method to save or copy states between objects (C<copy_state()>). Would be useful for
2831 using this module in an editor.
2832  
2833 =item 6
2834  
2835 Add support for greater-than-one length special characters. Specifically, octal,
2836 hexidecimal, and control character codes. For example, C<\644>, C<\x1a4> or C<\c[>.
2837  
2838 =back
2839  
2840 =head1 REVISIONS
2841  
2842 =head2 04-04-2001 Cory Johns
2843  
2844 =over 4
2845  
2846 =item *
2847  
2848 Fixed problem with special characters not formatting inside of Here-Documents.
2849  
2850 =item *
2851  
2852 Fixed bug causing hash variables to format inside of Here-Documents.
2853  
2854 =back
2855  
2856 =head2 03-30-2001 Cory Johns
2857  
2858 =over 4
2859  
2860 =item *
2861  
2862 Fixed bug where quote-terminators were checked for inside of Here-Documents.
2863  
2864 =back
2865  
2866 =head2 03-29-2001 Cory Johns
2867  
2868 =over 4
2869  
2870 =item *
2871  
2872 Moved token processing tests from _format_line() into _process_token()
2873 (where they should've been all along), generally making _format_line()
2874 more logical. Contemplating extrapolating the tokenizing and token
2875 loop into its own subroutine to avoid all the recursive calls.
2876  
2877 =item *
2878  
2879 Fixed bug that caused special characters to be recognized outside of
2880 strings.
2881  
2882 =item *
2883  
2884 Added $VERSION variable.
2885  
2886 =item *
2887  
2888 Added support for different types of literal numbers: floating point,
2889 exponential notation (eg: 1.3e10), hexidecimal, and underscore-separated.
2890  
2891 =item *
2892  
2893 Added the C<CodeTerm> and C<DATA> Formats.
2894  
2895 =back
2896  
2897 =head2 03-27-2001 Cory Johns
2898  
2899 =over 4
2900  
2901 =item *
2902  
2903 Added was_pod() and updated the documentation for in_pod().
2904  
2905 =back
2906  
2907 =head2 03-20-2001 Cory Johns
2908  
2909 =over 4
2910  
2911 =item *
2912  
2913 Added support for Perl formats (ie, `C<format = ...>').
2914  
2915 =back