CL-PPCRE - portable Perl-compatible regular expressions for Common Lisp


 

Abstract

CL-PPCRE is a portable regular expression library for Common Lisp which has the following features:

 

Contents

  1. How to use CL-PPCRE
    1. create-scanner (for Perl regex strings)
    2. create-scanner (for parse trees)
    3. scan
    4. scan-to-strings
    5. do-scans
    6. do-matches
    7. do-matches-as-strings
    8. all-matches
    9. all-matches-as-strings
    10. split
    11. regex-replace
    12. regex-replace-all
    13. regex-apropos
    14. regex-apropos-list
    15. *regex-char-code-limit*
    16. *use-bmh-matchers*
  2. Download and installation
  3. Testing CL-PPCRE
  4. Compatibility with Perl
    1. Empty strings instead of undef in $1, $2, etc.
    2. Strange scoping of embedded modifiers
    3. Inconsistent capturing of $1, $2, etc.
    4. Captured groups not available outside of look-aheads and look-behinds
    5. Alternations don't always work from left to right
    6. "\r" doesn't work with MCL
    7. What about "\w"?
  5. Performance
    1. Benchmarking
    2. Other performance issues
  6. Bugs and problems
    1. Stack overflow
  7. Remarks
  8. Acknowledgements

 

How to use CL-PPCRE

CL-PPCRE exports the following symbols:


[Function]
create-scanner string &key case-insensitive-mode multi-line-mode single-line-mode extended-mode destructive => scanner


Accepts a string which is a regular expression in Perl syntax and returns a closure which will scan strings for this regular expression. The mode keyboard arguments are equivalent to the "imsx" modifiers in Perl. The destructive keyword will be ignored.

The function accepts most of the regex syntax of Perl 5 as described in man perlre including extended features like non-greedy repetitions, positive and negative look-ahead and look-behind assertions, "standalone" subexpressions, and conditional subpatterns. The following Perl features are (currently) not supported:

Note, however, that \t, \n, \r, \f, \a, \e, \033 (octal character codes), \x1B (hexadecimal character codes), \c[ (control characters), \w, \W, \s, \S, \d, \D, \b, \B, \A, \Z, and \z are supported.

The keyword arguments are just for your convenience. You can always use embedded modifiers like "(?i-s)" instead.


[Function]
create-scanner parse-tree &key case-insensitive-mode multi-line-mode single-line-mode extended-mode destructive => scanner


This is similar to CREATE-SCANNER above but accepts a parse tree as its first argument. A parse tree is an S-expression conforming to the following syntax: Because CREATE-SCANNER is defined as a generic function which dispatches on its first argument there's a certain ambiguity: Although strings are valid parse trees they will be interpreted as Perl regex strings when given to CREATE-SCANNER. To circumvent this you can always use the equivalent parse tree (:GROUP <string>) instead.

Note that currently CREATE-SCANNER doesn't always check for the well-formedness of its first argument, i.e. you are expected to provide correct parse trees. This will most likely change in future releases.

The usage of the keyword argument extended-mode obviously doesn't make sense if CREATE-SCANNER is applied to parse trees and will signal an error.

If destructive is not NIL (the default is NIL) the function is allowed to destructively modify parse-tree while creating the scanner.

If you want to find out how parse trees are related to Perl regex strings you should play around with CL-PPCRE::PARSE-STRING - a function which converts Perl regex strings to parse trees. Here are some examples: 

* (cl-ppcre::parse-string "(ab)*")
(:GREEDY-REPETITION 0 NIL (:REGISTER "ab"))

* (cl-ppcre::parse-string "(a(b))")
(:REGISTER (:SEQUENCE #\a (:REGISTER #\b)))

* (cl-ppcre::parse-string "(?:abc){3,5}")
(:GREEDY-REPETITION 3 5 (:GROUP "abc"))
;; (:GREEDY-REPETITION 3 5 "abc") would also be OK,

* (cl-ppcre::parse-string "a(?i)b(?-i)c")
(:SEQUENCE #\a
 (:SEQUENCE (:FLAGS :CASE-INSENSITIVE-P)
  (:SEQUENCE #\b (:SEQUENCE (:FLAGS :CASE-SENSITIVE-P) #\c))))
;; same as (:SEQUENCE #\a :CASE-INSENSITIVE-P #\b :CASE-SENSITIVE-P #\c)

* (cl-ppcre::parse-string "(?=a)b")
(:SEQUENCE (:POSITIVE-LOOKAHEAD #\a) #\b)


For the rest of this section regex can always be a string (which is interpreted as a Perl regular expression), a parse tree, or a scanner created by CREATE-SCANNER. The start and end keyword parameters are always used as in SCAN.


[Function]
scan regex target-string &key start end => match-start, match-end, reg-starts, reg-ends


Searches the string target-string from start (which defaults to 0) to end (which default to the length of target-string) and tries to match regex. On success returns four values - the start of the match, the end of the match, and two arrays denoting the beginnings and ends of register matches. On failure returns NIL. target-string will be coerced to a simple string if it isn't one already.

SCAN acts as if the part of target-string between start and end were a standalone string, i.e. look-aheads and look-behinds can't look beyond these boundaries.

Examples: 

* (cl-ppcre:scan "(a)*b" "xaaabd")
1
5
#(3)
#(4)

* (cl-ppcre:scan "(a)*b" "xaaabd" :start 1)
1
5
#(3)
#(4)

* (cl-ppcre:scan "(a)*b" "xaaabd" :start 2)
2
5
#(3)
#(4)

* (cl-ppcre:scan "(a)*b" "xaaabd" :end 4)
NIL

* (cl-ppcre:scan '(:GREEDY-REPETITION 0 NIL #\b) "bbbc")
0
3
#()
#()

* (cl-ppcre:scan '(:GREEDY-REPETITION 4 6 #\b) "bbbc")
NIL

* (let ((s (cl-ppcre:create-scanner "(([a-c])+)x")))
    (cl-ppcre:scan s "abcxy"))
0
4
#(0 2)
#(3 3)


[Function]
scan-to-strings regex target-string &key start end => match, regs


Like SCAN but returns substrings of target-string instead of positions, i.e. this function returns two values on success: the whole match as a string plus an array of substrings (or NILs) corresponding to the matched registers.

Examples: 

* (cl-ppcre:scan-to-strings "[^b]*b" "aaabd")
"aaab"
#()

* (cl-ppcre:scan-to-strings "([^b])*b" "aaabd")
"aaab"
#("a")

* (cl-ppcre:scan-to-strings "(([^b])*)b" "aaabd")
"aaab"
#("aaa" "a")


[Macro]
do-scans (match-start match-end reg-starts reg-ends regex target-string &optional result-form &key start end) declaration* statement* => result*


A macro which iterates over target-string and tries to match regex as often as possible evaluating statement* with match-start, match-end, reg-starts, and reg-ends bound to the four return values of each match (see SCAN) in turn. After the last match, returns result-form if provided or NIL otherwise. An implicit block named NIL surrounds DO-SCANS; RETURN may be used to terminate the loop immediately. If regex matches an empty string the scan is continued one position behind this match.

This is the most general macro to iterate over all matches in a target string. See the source code of DO-MATCHES, ALL-MATCHES, SPLIT, or REGEX-REPLACE-ALL for examples of its usage.


[Macro]
do-matches (match-start match-end regex target-string &optional result-form &key start end) declaration* statement* => result*


Like DO-SCANS but doesn't bind variables to the register arrays.

Example: 

* (defun foo (regex target-string &key (start 0) (end (length target-string)))
    (let ((sum 0))
      (cl-ppcre:do-matches (s e regex target-string nil :start start :end end)
        (incf sum (- e s)))
      (format t "~,2F% of the string was inside of a match~%"
                ;; note: doesn't check for division by zero
                (float (* 100 (/ sum (- end start)))))))

FOO

* (foo "a" "abcabcabc")
33.33% of the string was inside of a match
NIL
* (foo "aa|b" "aacabcbbc")
55.56% of the string was inside of a match
NIL


[Macro]
do-matches-as-strings (match-var regex target-string &optional result-form &key start end) declaration* statement* => result*


Like DO-MATCHES but binds match-var to the substring of target-string corresponding to each match in turn.

Example: 

* (defun crossfoot (target-string &key (start 0) (end (length target-string)))
    (let ((sum 0))
      (cl-ppcre:do-matches-as-strings (m :digit-class
                                         target-string nil
                                         :start start :end end)
        (incf sum (parse-integer m)))
      (if (< sum 10)
        sum
        (crossfoot (format nil "~A" sum)))))

CROSSFOOT

* (crossfoot "bar")
0

* (crossfoot "a3x")
3

* (crossfoot "12345")
6
Of course, in real life you would do this with DO-MATCHES and use the start and end keyword parameters of PARSE-INTEGER.


[Function]
all-matches regex target-string &key start end => list


Returns a list containing the start and end positions of all matches of regex against target-string, i.e. if there are N matches the list contains (* 2 N) elements. If regex matches an empty string the scan is continued one position behind this match.

Examples: 

* (cl-ppcre:all-matches "a" "foo bar baz")
(5 6 9 10)

* (cl-ppcre:all-matches "\\w*" "foo bar baz")
(0 3 3 3 4 7 7 7 8 11 11 11)


[Function]
all-matches-as-strings regex target-string &key start end => list


Like ALL-MATCHES but returns a list of substrings instead.

Examples: 

* (cl-ppcre:all-matches-as-strings "a" "foo bar baz")
("a" "a")

* (cl-ppcre:all-matches-as-strings "\\w*" "foo bar baz")
("foo" "" "bar" "" "baz" "")


[Function]
split regex target-string &key start end limit with-registers-p omit-unmatched-p => list


Matches regex against target-string as often as possible and returns a list of the substrings between the matches. If with-registers-p is true, substrings corresponding to matched registers are inserted into the list as well. If omit-unmatched-p is true, unmatched registers will simply be left out, otherwise they will show up as NIL. limit limits the number of elements returned - registers aren't counted. If limit is NIL (or 0 which is equivalent), trailing empty strings are removed from the result list. If regex matches an empty string the scan is continued one position behind this match.

Beginning with CL-PPCRE 0.2.0, this function also tries hard to be Perl-compatible - thus the somewhat peculiar behaviour. But note that it hasn't been as extensively tested as SCAN.

Examples: 

* (cl-ppcre:split "\\s+" "foo   bar baz
frob")
("foo" "bar" "baz" "frob")

* (cl-ppcre:split "\\s*" "foo bar   baz")
("f" "o" "o" "b" "a" "r" "b" "a" "z")

* (cl-ppcre:split "(\\s+)" "foo bar   baz")
("foo" "bar" "baz")

* (cl-ppcre:split "(\\s+)" "foo bar   baz" :with-registers-p t)
("foo" " " "bar" "   " "baz")

* (cl-ppcre:split "(\\s)(\\s*)" "foo bar   baz" :with-registers-p t)
("foo" " " "" "bar" " " "  " "baz")

* (cl-ppcre:split "(,)|(;)" "foo,bar;baz" :with-registers-p t)
("foo" "," NIL "bar" NIL ";" "baz")

* (cl-ppcre:split "(,)|(;)" "foo,bar;baz" :with-registers-p t :omit-unmatched-p t)
("foo" "," "bar" ";" "baz")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::")
("a" "b" "c" "d" "e" "f" "g")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 1)
("a:b:c:d:e:f:g::")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 2)
("a" "b:c:d:e:f:g::")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 3)
("a" "b" "c:d:e:f:g::")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 1000)
("a" "b" "c" "d" "e" "f" "g" "" "")


[Function]
regex-replace regex target-string replacement &key start end preserve-case => list


Try to match target-string between start and end against regex and replace the first match with replacement.

replacement can be a string which may contain the special substrings "\&" for the whole match, "\`" for the part of target-string before the match, "\'" for the part of target-string after the match, "\N" or "\{N}" for the Nth register where N is a positive integer.

replacement can also be a function designator in which case the match will be replaced with the result of calling the function designated by replacement with the arguments target-string, start, end, match-start, match-end, reg-starts, and reg-ends. (reg-starts and reg-ends are arrays holding the start and end positions of matched registers (or NIL) - the meaning of the other arguments should be obvious.)

Finally, replacement can be a list where each element is a string (which will be inserted verbatim), one of the symbols :match, :before-match, or :after-match (corresponding to "\&", "\`", and "\'" above), an integer N (representing register (1+ N)), or a function designator.

If preserve-case is true (default is NIL), the replacement will try to preserve the case (all upper case, all lower case, or capitalized) of the match. The result will always be a fresh string, even if regex doesn't match.

Examples: 

* (cl-ppcre:regex-replace "fo+" "foo bar" "frob")
"frob bar"

* (cl-ppcre:regex-replace "fo+" "FOO bar" "frob")
"FOO bar"

* (cl-ppcre:regex-replace "(?i)fo+" "FOO bar" "frob")
"frob bar"

* (cl-ppcre:regex-replace "(?i)fo+" "FOO bar" "frob" :preserve-case t)
"FROB bar"

* (cl-ppcre:regex-replace "(?i)fo+" "Foo bar" "frob" :preserve-case t)
"Frob bar"

* (cl-ppcre:regex-replace "bar" "foo bar baz" "[frob (was '\\&' between '\\`' and '\\'')]")
"foo [frob (was 'bar' between 'foo ' and ' baz')] baz"

* (cl-ppcre:regex-replace "bar" "foo bar baz"
                          '("[frob (was '" :match "' between '" :before-match "' and '" :after-match "')]"))
"foo [frob (was 'bar' between 'foo ' and ' baz')] baz"


[Function]
regex-replace-all regex target-string replacement &key start end preserve-case => list


Like REGEX-REPLACE but replaces all matches.

Examples: 

* (cl-ppcre:regex-replace-all "(?i)fo+" "foo Fooo FOOOO bar" "frob" :preserve-case t)
"frob Frob FROB bar"

* (cl-ppcre:regex-replace-all "(?i)f(o+)" "foo Fooo FOOOO bar" "fr\\1b" :preserve-case t)
"froob Frooob FROOOOB bar"

* (let ((qp-regex (cl-ppcre:create-scanner "[\\x80-\\xff]")))
    (defun encode-quoted-printable (string)
      "Convert 8-bit string to quoted-printable representation."
      ;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there
      (flet ((convert (target-string start end match-start match-end reg-starts reg-ends)
             (declare (ignore start end match-end reg-starts reg-ends))
             (format nil "=~2,'0x" (char-code (char target-string match-start)))))
        (cl-ppcre:regex-replace-all qp-regex string #'convert))))
Converted ENCODE-QUOTED-PRINTABLE.
ENCODE-QUOTED-PRINTABLE

* (encode-quoted-printable "Fête Sørensen naïve Hühner Straße")
"F=EAte S=F8rensen na=EFve H=FChner Stra=DFe"

* (let ((url-regex (cl-ppcre:create-scanner "[^a-zA-Z0-9_\\-.]")))
    (defun url-encode (string)
      "URL-encode a string."
      ;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there
      (flet ((convert (target-string start end match-start match-end reg-starts reg-ends)
             (declare (ignore start end match-end reg-starts reg-ends))
             (format nil "%~2,'0x" (char-code (char target-string match-start)))))
        (cl-ppcre:regex-replace-all url-regex string #'convert))))
Converted URL-ENCODE.
URL-ENCODE

* (url-encode "Fête Sørensen naïve Hühner Straße")
"F%EAte%20S%F8rensen%20na%EFve%20H%FChner%20Stra%DFe"

* (defun how-many (target-string start end match-start match-end reg-starts reg-ends)
    (declare (ignore start end match-start match-end))
    (format nil "~A" (- (svref reg-ends 0)
                        (svref reg-starts 0))))
HOW-MANY

* (cl-ppcre:regex-replace-all "{(.+?)}"
                              "foo{...}bar{.....}{..}baz{....}frob"
                              (list "[" 'how-many " dots]"))
"foo[3 dots]bar[5 dots][2 dots]baz[4 dots]frob"


[Function]
regex-apropos regex &optional packages &key case-insensitive => list


Like APROPOS but searches for interned symbols which match the regular expression regex. The output is implementation-dependent. If case-insensitive is true (which is the default) and regex isn't already a scanner, a case-insensitive scanner is used.

Here are examples for CMUCL: 

* *package*
#<The COMMON-LISP-USER package, 16/21 internal, 0/9 external>

* (defun foo (n &optional (k 0)) (+ 3 n k))
FOO

* (defparameter foo "bar")
FOO

* (defparameter |foobar| 42)
|foobar|

* (defparameter fooboo 43)
FOOBOO

* (defclass frobar () ())
#<STANDARD-CLASS FROBAR {4874E625}>

* (cl-ppcre:regex-apropos "foo(?:bar)?")
FOO [variable] value: "bar"
    [compiled function] (N &OPTIONAL (K 0))
FOOBOO [variable] value: 43
|foobar| [variable] value: 42

* (cl-ppcre:regex-apropos "(?:foo|fro)bar")
PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]
FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
|foobar| [variable] value: 42

* (cl-ppcre:regex-apropos "(?:foo|fro)bar" 'cl-user)
FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
|foobar| [variable] value: 42

* (cl-ppcre:regex-apropos "(?:foo|fro)bar" '(pcl ext))
PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]

* (cl-ppcre:regex-apropos "foo")
FOO [variable] value: "bar"
    [compiled function] (N &OPTIONAL (K 0))
FOOBOO [variable] value: 43
|foobar| [variable] value: 42

* (cl-ppcre:regex-apropos "foo" nil :case-insensitive nil)
|foobar| [variable] value: 42


[Function]
regex-apropos-list regex &optional packages &key upcase => list


Like APROPOS-LIST but searches for interned symbols which match the regular expression regex. If case-insensitive is true (which is the default) and regex isn't already a scanner, a case-insensitive scanner is used.

Example (continued from above): 

* (cl-ppcre:regex-apropos-list "foo(?:bar)?")
(|foobar| FOOBOO FOO)


[Special variable]
*regex-char-code-limit*


This variable controls whether scanners take into account all characters of your CL implementation or only those the CHAR-CODE of which is not larger than its value. It is only relevant if the regular expression contains certain character classes. The default is CHAR-CODE-LIMIT, and you might see significant speed and space improvements during scanner creation if, say, your target strings only contain ISO-8859-1 characters and you're using an implementation like AllegroCL, LispWorks, or CLISP where CHAR-CODE-LIMIT has a value much higher than 255. The test suite will automatically set *REGEX-CHAR-CODE-LIMIT* to 255 while you're running the default test.

Here's an example with LispWorks: 

CL-USER 23 > (time (cl-ppcre:create-scanner "[3\\D]"))
Timing the evaluation of (CL-PPCRE:CREATE-SCANNER "[3\\D]")

user time    =      0.443
system time  =      0.001
Elapsed time =   0:00:01
Allocation   = 546600 bytes standard / 2162611 bytes fixlen
0 Page faults
#<closure 20654AF2>

CL-USER 24 > (time (let ((cl-ppcre:*regex-char-code-limit* 255)) (cl-ppcre:create-scanner "[3\\D]")))
Timing the evaluation of (LET ((CL-PPCRE:*REGEX-CHAR-CODE-LIMIT* 255)) (CL-PPCRE:CREATE-SCANNER "[3\\D]"))

user time    =      0.000
system time  =      0.000
Elapsed time =   0:00:00
Allocation   = 3336 bytes standard / 8338 bytes fixlen
0 Page faults
#<closure 206569DA>

Note: Due to the nature of LOAD-TIME-VALUE and the compiler macro for SCAN some scanners might be created in a null lexical environment at load time or at compile time so be careful to which value *REGEX-CHAR-CODE-LIMIT* is bound at that time. The default value should always yield correct results unless you play dirty tricks with implementation-dependent behaviour, though.


[Special variable]
*use-bmh-matchers*


Usually, the scanners created by CREATE-SCANNER (or implicitely by other functions and macros) will use fast Boyer-Moore-Horspool matchers to check for constant strings at the start or end of the regular expression. If *USE-BMH-MATCHERS* is NIL (the default is T), the standard function SEARCH will be used instead. This will usually be a bit slower but can save lots of space if you're storing many scanners. The test suite will automatically set *USE-BMH-MATCHERS* to NIL while you're running the default test.

Note: Due to the nature of LOAD-TIME-VALUE and the compiler macro for SCAN some scanners might be created in a null lexical environment at load time or at compile time so be careful to which value *USE-BMH-MATCHERS* is bound at that time.


 

Download and installation

CL-PPCRE together with this documentation can be downloaded from http://weitz.de/files/cl-ppcre.tgz. The current version is 0.5.4 - older versions, beginning from 0.1.1, are available for download at URLs like http://weitz.de/files/cl-ppcre-<version>.tgz. A CHANGELOG is available.

If you're on Debian you should probably use the CL-PPCRE Debian package which is available thanks to Kevin Rosenberg.

CL-PPCRE comes with simple system definitions for MK:DEFSYSTEM and asdf so you can either adapt it to your needs or just unpack the archive and from within the CL-PPCRE directory start your Lisp image and evaluate the form (mk:compile-system "cl-ppcre") (or the equivalent one for asdf) which should compile and load the whole system.

If for the some reason you don't want to use MK:DEFSYSTEM or asdf you can just LOAD the file load.lisp or you can also get away with something like this: 

(loop for name in '("packages" "specials" "util" "lexer"
                    "parser" "regex-class" "convert" "optimize"
                    "closures" "repetition-closures" "scanner" "api")
      do (compile-file (make-pathname :name name
                                      :type "lisp"))
         (load name))
Note that on CL implementations which use the Python compiler (i.e. CMUCL, SBCL, SCL) you can concatenate the compiled object files to create one single object file which you can load afterwards: 
cat {packages,specials,util,lexer,parser,regex-class,convert,optimize,closures,repetition-closures,scanner,api}.x86f > cl-ppcre.x86f
(Replace ".x86f" with the correct suffix for your platform.)
 

Testing CL-PPCRE

CL-PPCRE comes with a comprehensive test suite most of which is stolen from the PCRE library. You can use it like this: 
* (mk:compile-system "cl-ppcre-test")
; Loading #p"/home/edi/cl-ppcre/cl-ppcre.system".
; Loading #p"/home/edi/cl-ppcre/packages.x86f".
; Loading #p"/home/edi/cl-ppcre/specials.x86f".
; Loading #p"/home/edi/cl-ppcre/util.x86f".
; Loading #p"/home/edi/cl-ppcre/lexer.x86f".
; Loading #p"/home/edi/cl-ppcre/parser.x86f".
; Loading #p"/home/edi/cl-ppcre/regex-class.x86f".
; Loading #p"/home/edi/cl-ppcre/convert.x86f".
; Loading #p"/home/edi/cl-ppcre/optimize.x86f".
; Loading #p"/home/edi/cl-ppcre/closures.x86f".
; Loading #p"/home/edi/cl-ppcre/repetition-closures.x86f".
; Loading #p"/home/edi/cl-ppcre/scanner.x86f".
; Loading #p"/home/edi/cl-ppcre/api.x86f".
; Loading #p"/home/edi/cl-ppcre/ppcre-tests.x86f".
NIL

* (cl-ppcre-test:test)

;; ....
;; (a list of incompatibilities with Perl)
(If you're not using MK:DEFSYSTEM or asdf it suffices to build CL-PPCRE and then compile and load the file ppcre-tests.lisp.)

With LispWorks and SCL you can also call CL-PPCRE-TEST:TEST with a keyword argument argument THREADED which - in addition to the usual tests - will also check whether the scanners created by CL-PPCRE are thread-safe.

Note that the file testdata provided with CL-PPCRE was created on a Linux system with Perl 5.8.0. You can (and you should if you're on Mac OS or Windows) create your own testdata with the Perl script perltest.pl: 

edi@bird:~/cl-ppcre > perl perltest.pl < testinput > testdata
Of course you can also create your own tests - the format accepted by perltest.pl should be rather clear from looking at the file testinput. Note that the target strings are wrapped in double quotes and then fed to Perl's eval so you can use ugly Perl constructs like, say, a@{['b' x 10]}c which will result in the target string "abbbbbbbbbbc".
 

Compatibility with Perl

Depending on your Perl version you might encounter a couple of small incompatibilities with Perl most of which aren't due to CL-PPCRE:

Empty strings instead of undef in $1, $2, etc.

(Cf. case #629 of testdata.) This is a bug in Perl 5.6.1 and earlier which has been fixed in 5.8.0.

Strange scoping of embedded modifiers

(Cf. case #430 of testdata.) This is a bug in Perl 5.6.1 and earlier which has been fixed in 5.8.0.

Inconsistent capturing of $1, $2, etc.

(Cf. case #662 of testdata.) This is a bug in Perl which hasn't been fixed yet.

Captured groups not available outside of look-aheads and look-behinds

(Cf. case #1439 of testdata.) Well, OK, this ain't a Perl bug. I just can't quite understand why captured groups should only be seen within the scope of a look-ahead or look-behind. For the moment, CL-PPCRE and Perl agree to disagree... :)

Alternations don't always work from left to right

(Cf. case #790 of testdata.) I also think this a Perl bug but I currently have lost the drive to report it.

"\r" doesn't work with MCL

(Cf. case #9 of testdata.) For some strange reason that I don't understand MCL translates #\Return to (CODE-CHAR 10) while MacPerl translates "\r" to (CODE-CHAR 13). Hmmm...

What about "\w"?

CL-PPCRE uses ALPHANUMERICP to decide whether a character matches Perl's "\w", so depending on your CL implementation you might encounter differences between Perl and CL-PPCRE when matching non-ASCII characters.
 

Performance

Benchmarking

The CL-PPCRE test suite can also be used for benchmarking purposes: If you call perltest.pl with a command line argument it will be interpreted as the number of seconds each test should run. Perl will time its tests accordingly and create output which, when fed to CL-PPCRE-TEST:TEST, will result in a benchmark. Here's an example: 
edi@bird:~/cl-ppcre > echo "/((a{0,5}){0,5})*[c]/
aaaaaaaaaaaac

/((a{0,5})*)*[c]/
aaaaaaaaaaaac" | perl perltest.pl .5 > timedata.lisp
1
2

edi@bird:~/cl-ppcre > cmucl -quiet
; Loading #p"/home/edi/.cmucl-init".

* (mk:compile-system "cl-ppcre-test")
; Loading #p"/home/edi/cl-ppcre/cl-ppcre.system".
; Loading #p"/home/edi/cl-ppcre/packages.x86f".
; Loading #p"/home/edi/cl-ppcre/specials.x86f".
; Loading #p"/home/edi/cl-ppcre/util.x86f".
; Loading #p"/home/edi/cl-ppcre/lexer.x86f".
; Loading #p"/home/edi/cl-ppcre/parser.x86f".
; Loading #p"/home/edi/cl-ppcre/regex-class.x86f".
; Loading #p"/home/edi/cl-ppcre/convert.x86f".
; Loading #p"/home/edi/cl-ppcre/optimize.x86f".
; Loading #p"/home/edi/cl-ppcre/closures.x86f".
; Loading #p"/home/edi/cl-ppcre/repetition-closures.x86f".
; Loading #p"/home/edi/cl-ppcre/scanner.x86f".
; Loading #p"/home/edi/cl-ppcre/api.x86f".
; Loading #p"/home/edi/cl-ppcre/ppcre-tests.x86f".
NIL

* (cl-ppcre-test:test "/home/edi/cl-ppcre/timedata")
   1: 0.5559 (1000000 repetitions, Perl: 4.5330 seconds, CL-PPCRE: 2.5200 seconds)
   2: 0.4573 (1000000 repetitions, Perl: 4.5922 seconds, CL-PPCRE: 2.1000 seconds)
NIL
We gave two test cases to perltest.pl and asked it to repeat those tests often enough so that it takes at least 0.5 seconds to run each of them. In both cases, CMUCL was about twice as fast as Perl.

Here are some more benchmarks (done with Perl 5.6.1 and CMUCL 18d+):

Test caseRepetitionsPerl (sec)CL-PPCRE (sec)Ratio CL-PPCRE/Perl
"@{['x' x 100]}" =~ /(.)*/s1000000.13940.07000.5022
"@{['x' x 1000]}" =~ /(.)*/s1000000.16280.06000.3685
"@{['x' x 10000]}" =~ /(.)*/s1000000.50710.06000.1183
"@{['x' x 100000]}" =~ /(.)*/s100000.39020.00000.0000
"@{['x' x 100]}" =~ /.*/1000000.15200.08000.5262
"@{['x' x 1000]}" =~ /.*/1000000.37860.54001.4263
"@{['x' x 10000]}" =~ /.*/100000.27090.51001.8826
"@{['x' x 100000]}" =~ /.*/10000.27340.51001.8656
"@{['x' x 100]}" =~ /.*/s1000000.13200.03000.2274
"@{['x' x 1000]}" =~ /.*/s1000000.16340.03000.1836
"@{['x' x 10000]}" =~ /.*/s1000000.53040.03000.0566
"@{['x' x 100000]}" =~ /.*/s100000.39660.00000.0000
"@{['x' x 100]}" =~ /x*/1000000.15070.09000.5970
"@{['x' x 1000]}" =~ /x*/1000000.37820.63001.6658
"@{['x' x 10000]}" =~ /x*/100000.27300.60002.1981
"@{['x' x 100000]}" =~ /x*/10000.27080.59002.1790
"@{['x' x 100]}" =~ /[xy]*/1000000.26370.15000.5688
"@{['x' x 1000]}" =~ /[xy]*/100000.14490.12000.8282
"@{['x' x 10000]}" =~ /[xy]*/10000.13440.11000.8185
"@{['x' x 100000]}" =~ /[xy]*/1000.13550.12000.8857
"@{['x' x 100]}" =~ /(.)*/1000000.15230.11000.7220
"@{['x' x 1000]}" =~ /(.)*/1000000.37350.57001.5262
"@{['x' x 10000]}" =~ /(.)*/100000.27350.51001.8647
"@{['x' x 100000]}" =~ /(.)*/10000.25980.50001.9242
"@{['x' x 100]}" =~ /(x)*/1000000.15650.13000.8307
"@{['x' x 1000]}" =~ /(x)*/1000000.37830.66001.7446
"@{['x' x 10000]}" =~ /(x)*/100000.27200.60002.2055
"@{['x' x 100000]}" =~ /(x)*/10000.27250.60002.2020
"@{['x' x 100]}" =~ /(y|x)*/100000.24110.10000.4147
"@{['x' x 1000]}" =~ /(y|x)*/10000.23130.09000.3891
"@{['x' x 10000]}" =~ /(y|x)*/1000.23360.09000.3852
"@{['x' x 100000]}" =~ /(y|x)*/100.41650.09000.2161
"@{['x' x 100]}" =~ /([xy])*/1000000.26780.18000.6721
"@{['x' x 1000]}" =~ /([xy])*/100000.14590.12000.8227
"@{['x' x 10000]}" =~ /([xy])*/10000.13720.11000.8017
"@{['x' x 100000]}" =~ /([xy])*/1000.13580.11000.8098
"@{['x' x 100]}" =~ /((x){2})*/100000.10730.04000.3727
"@{['x' x 1000]}" =~ /((x){2})*/100000.91460.24000.2624
"@{['x' x 10000]}" =~ /((x){2})*/10000.90200.23000.2550
"@{['x' x 100000]}" =~ /((x){2})*/1000.89830.23000.2560
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/1000000.28290.23000.8129
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/100000.18590.17000.9143
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/10000.14200.17001.1968
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /[a-z]*FOOBARBAZ/10000000.91960.46000.5002
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /[a-z]*FOOBARBAZ/1000000.21660.25001.1542
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /[a-z]*FOOBARBAZ/100000.14650.23001.5696
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/1000000.29170.26000.8915
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/100000.18110.18000.9942
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/10000.14240.16001.1233
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /([a-z])*FOOBARBAZ/10000000.91540.74000.8083
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /([a-z])*FOOBARBAZ/1000000.21700.28001.2901
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /([a-z])*FOOBARBAZ/100000.14970.23001.5360
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/100000.43590.15000.3441
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/10000.54560.15000.2749
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/100.20390.06000.2943
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/10000000.93110.74000.7947
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/1000000.21620.27001.2489
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/100000.14880.23001.5455
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /[a-z]*FOOBARBAZ/i10000.15550.00000.0000
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /[a-z]*FOOBARBAZ/i100.14410.00000.0000
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /[a-z]*FOOBARBAZ/i1013.71500.01000.0007

As you might have noticed, Perl shines if it can reduce significant parts of the matching process to cases where it can advance through the target string one character at a time. This leads to C code where you can very efficiently test and increment a pointer into a string in a tight loop and can hardly be beaten with CL. In almost all other cases, the CMUCL/CL-PPCRE combination is usually faster than Perl - sometimes a lot faster.

As most of the examples above were chosen to make Perl look good here's another benchmark - the result of running perltest.pl against the full testdata file with a time limit of 0.1 seconds, CL-PPCRE 0.1.2 on CMUCL 18e-pre vs. Perl 5.6.1. CL-PPCRE is faster than Perl in 1511 of 1545 cases - in 1045 cases it's more than twice as fast.

Note that Perl as well as CL-PPCRE keep the rightmost matches in registers - keep that in mind if you benchmark against other regex implementations. Also note that CL-PPCRE-TEST:TEST automatically skips test cases where Perl and CL-PPCRE don't agree.

Other performance issues

While the scanners created by CL-PPCRE are pretty fast, the process which creates scanners from Perl regex strings and parse trees isn't that speedy and conses a lot. It is recommended that you store and re-use scanners if possible. The DO-macros will do this for you automatically.

However, beginning with version 0.5.2, CL-PPCRE uses a compiler macro and LOAD-TIME-VALUE to make sure that the scanner is only built once if the first argument to SCAN, SCAN-TO-STRINGS, or REGEX-REPLACE is a constant form. (But see the notes for *REGEX-CHAR-CODE-LIMIT* and *USE-BMH-MATCHERS*.)

Here's an example of its effect 

* (trace cl-ppcre::convert)
(CL-PPCRE::CONVERT)
* (defun foo (string) (cl-ppcre:scan "(?s).*" string))
FOO
* (time (foo "The quick brown fox"))
Compiling LAMBDA NIL: 
Compiling Top-Level Form: 

  0: (CL-PPCRE::CONVERT #<lambda-list-unavailable>)
  0: CL-PPCRE::CONVERT returned
       #<CL-PPCRE::SEQ {48B033C5}>
       0
       #<CL-PPCRE::EVERYTHING {48B031D5}>
Evaluation took:
  0.0 seconds of real time
  0.00293 seconds of user run time
  9.77e-4 seconds of system run time
  0 page faults and
  11,408 bytes consed.
0
19
#()
#()
* (time (foo "The quick brown fox"))
Compiling LAMBDA NIL: 
Compiling Top-Level Form: 

  0: (CL-PPCRE::CONVERT #<lambda-list-unavailable>)
  0: CL-PPCRE::CONVERT returned
       #<CL-PPCRE::SEQ {48B14C4D}>
       0
       #<CL-PPCRE::EVERYTHING {48B14B65}>
Evaluation took:
  0.0 seconds of real time
  0.00293 seconds of user run time
  0.0 seconds of system run time
  0 page faults and
  10,960 bytes consed.
0
19
#()
#()
* (compile 'foo)
  0: (CL-PPCRE::CONVERT #<lambda-list-unavailable>)
  0: CL-PPCRE::CONVERT returned
       #<CL-PPCRE::SEQ {48B1FEC5}>
       0
       #<CL-PPCRE::EVERYTHING {48B1FDDD}>
Compiling LAMBDA (STRING): 
Compiling Top-Level Form: 
FOO
NIL
NIL
* (time (foo "The quick brown fox"))
Compiling LAMBDA NIL: 
Compiling Top-Level Form: 

Evaluation took:
  0.0 seconds of real time
  0.0 seconds of user run time
  0.0 seconds of system run time
  0 page faults and
  0 bytes consed.
0
19
#()
#()
* (time (foo "The quick brown fox"))
Compiling LAMBDA NIL: 
Compiling Top-Level Form: 

Evaluation took:
  0.0 seconds of real time
  0.0 seconds of user run time
  0.0 seconds of system run time
  0 page faults and
  0 bytes consed.
0
19
#()
#()
*

Of course, the usual rules for creating efficient regular expressions apply to CL-PPCRE as well although it can optimize a couple of cases itself. The most important rule is probably that you shouldn't use capturing groups if you don't need the captured information, i.e. use "(?:a|b)*" instead of "(a|b)*" if you don't need to refer to the register. (In fact, in this particular case CL-PPCRE will be able to optimize away the register group, but it won't if you replace "a|b" with, say, "a|bc".)

Another point worth mentioning is that you definitely should use single-line mode if you have long strings without #\Newline (or where you don't care about the line breaks) and plan to use regular expressions like ".*". See the benchmarks for comparisons between single-line mode and normal mode with such target strings.

Another thing to consider is that, for performance reasons, CL-PPCRE assumes that most of the target strings you're trying to match are simple strings and coerces non-simple strings to simple strings before scanning them. If you plan on working with non-simple strings mostly you might consider modifying the CL-PPCRE source code. This is easy: Change all occurences of SCHAR to CHAR and redefine the macro in util.lisp where the coercion takes place - that's all.
 

Bugs and problems

Stack overflow

CL-PPCRE can optimize away a lot of unnecessary backtracking but sometimes this simply isn't possible. With complicated regular expressions and long strings this might lead to stack overflows depending on your machine and your CL implementation.

Here's one example with CLISP: 

[1]> (defun target (n) (concatenate 'string (make-string n :initial-element #\a) "b"))
TARGET

[2]> (cl-ppcre:scan "a*" (target 1000))
0 ;
1000 ;
#() ;
#()

[3]> (cl-ppcre:scan "(?:a|b)*" (target 1000))
0 ;
1001 ;
#() ;
#()

[4]> (cl-ppcre:scan "(a|b)*" (target 1000))
0 ;
1001 ;
#(1000) ;
#(1001)

[5]> (cl-ppcre:scan "(a|b)*" (target 10000))
0 ;
10001 ;
#(10000) ;
#(10001)

[6]> (cl-ppcre:scan "(a|b)*" (target 100000))
0 ;
100001 ;
#(100000) ;
#(100001)

[7]> (cl-ppcre:scan "(a|b)*" (target 1000000))
0 ;
1000001 ;
#(1000000) ;
#(1000001)

;; No problem until now - but...

[8]> (cl-ppcre:scan "(a|)*" (target 100000))
*** - Lisp stack overflow. RESET

[9]> (cl-ppcre:scan "(a|)*" (target 3200))
*** - Lisp stack overflow. RESET

With CMUCL the situation is better and worse at the same time. It will take a lot longer until CMUCL gives up but if it gives up the whole Lisp image will silently die (at least on my machine): 

* (defun target (n) (concatenate 'string (make-string n :initial-element #\a) "b"))
TARGET

* (cl-ppcre:scan "(a|)*" (target 3200))
0
3200
#(3200)
#(3200)

* (cl-ppcre:scan "(a|)*" (target 10000))
0
10000
#(10000)
#(10000)

* (cl-ppcre:scan "(a|)*" (target 100000))
0
100000
#(100000)
#(100000)

* (cl-ppcre:scan "(a|)*" (target 1000000))
0
1000000
#(1000000)
#(1000000)

;; No problem until now - but...

* (cl-ppcre:scan "(a|)*" (target 10000000))
edi@bird:~ >
This behaviour can be changed with very conservative optimization settings but that'll make CL-PPCRE crawl compared to Perl.

You might want to compare this to the way Perl handles the same situation. It might lie to you: 

edi@bird:~ > perl -le '$_="a" x 32766 . "b"; /(a|)*/; print $1'

edi@bird:~ > perl -le '$_="a" x 32767 . "b"; /(a|)*/; print $1'
a
Or it might warn you before it's lying to you: 
edi@bird:~ > perl -lwe '$_="a" x 32767 . "b"; /(a|)*/; print $1'
Complex regular subexpression recursion limit (32766) exceeded at -e line 1.
a
Or it might simply die: 
edi@bird:~ > /opt/perl-5.8/bin/perl -lwe '$_="a" x 32767 . "b"; /(a|)*/; print $1'
Segmentation fault
Your mileage may vary, of course...
 

Remarks

The sample output from CMUCL and CLISP has been slightly edited to increase readability.

All test cases and benchmarks in this document where performed on an IBM Thinkpad T23 laptop (Pentium III 1.2 GHz, 768 MB RAM) running Gentoo Linux 1.1a.
 

Acknowledgements

Although I didn't use their code I was heavily inspired by looking at the Scheme/CL regex implementations of Dorai Sitaram and Michael Parker. Also, the nice folks from CMUCL's mailing list as well as the output of Perl's use re "debug" pragma have been very helpful in optimizing the scanners created by CL-PPCRE.

The asdf system definitions were kindly provided by Marco Baringer. Hannu Koivisto provided patches to make the .system files more usable. Thanks to Kevin Rosenberg and Douglas Crosher for pointing out how to be friendly to case-sensitive ACL images. Thanks to Karsten Poeck and JP Massar for their help in making CL-PPCRE work with Corman Lisp. JP Massar and Kent M. Pitman also helped to improve/fix the test suite and the compiler macro.

Thanks to the guys at "Café Olé" in Hamburg where I wrote most of the code and thanks to my wife for lending me her PowerBook to test CL-PPCRE with MCL and OpenMCL.

$Header: /usr/local/cvsrep/cl-ppcre/doc/index.html,v 1.37 2003/04/09 12:15:05 edi Exp $

BACK TO MY HOMEPAGE

viewable with any browser valid HTML 4.0