Recursive pattern in regex
Question:
This is very much related to Regular Expression to match outer brackets however, I specifically want to know how or whether it’s possible to do this regex’s recursive pattern? I’m yet to find a python example using this strategy so think this ought to be a useful question!
I’ve seen some claims that recursive patterns can be used to match balanced parenthesis, but no examples using python’s regex package (Note: re does not support recursive pattern, you need to use regex).
One claim is that syntax is b(?:m|(?R))*e where:
b is what begins the construct, m is what can occur in the middle of the construct, and e is what can occur at the end of the construct
I want to extract matches for the outer braces in the following:
"{1, {2, 3}} {4, 5}"
["1, {2, 3}", "4, 5"] # desired
Note that this is easy to do the same for inner braces:
re.findall(r"{([^{}]*)}", "{1, {2, 3}} {4, 5}")
['2, 3', '4, 5']
(In my example I was using finditer (over match objects), see here.)
So I had hoped that the following, or some variation, would work:
regex.findall(r"{(:[^{}]*|?R)}", "{1, {2, 3}} {4, 5}")
regex.findall(r"({(:[^{}]*|?R)})", "{1, {2, 3}} {4, 5}")
regex.findall(r"({(:.*|(?R))*})", "{1, {2, 3}} {4, 5}")
regex.findall(r"({(:.*)|(?R)*})", "{1, {2, 3}} {4, 5}")
regex.findall(r"({(:[^{}])|(?R)})", "{1, {2, 3}} {4, 5}")
but I’m scuppered by either [] or error: too much backtracking.
Is it possible to extract match objects for the outer parenthesis using regex’s recursion?
Obviously, I run the risk of being shot down with:
- don’t parse html with regex
- do this with pyparse
- write a proper lexer & parser e.g. using ply
I want to emphasis this is about how to use the recursive pattern (which if my understanding is correct, takes us outside of regular language parsing, so may can actually be possible!). If it can be done, this ought to be a cleaner solution.
Answers:
The pattern is:
{((?>[^{}]+|(?R))*)}
You can see this works for your example:
regex.findall("{((?>[^{}]+|(?R))*)}", "{1, {2, 3}} {4, 5}")
# ['1, {2, 3}', '4, 5']
Explanation:
The m part needs to exclude the brackets. The use of an atomic group is needed if you want at the same time to allow a quantifier for [^{}] and to repeat the group without catastropic backtracking problems. To be more clear, if the last closing curly bracket is missing this regex engine will backtrack atomic group by atomic group instead of character by character. To drive home this point, you can make the quantifier possessive like that: {((?>[^{}]+|(?R))*+)} (or {((?:[^{}]+|(?R))*+)} since the atomic group is no more useful).
The atomic group (?>....) and the possessive quantifier ?+, *+, ++ are the two sides of the same feature. This feature forbids the regex engine to backtrack inside the group of characters that becomes an “atom” (something you can’t divide in smaller parts).
The basic examples are the following two patterns that always fail for the string aaaaaaaaaab:
(?>a+)ab
a++ab
that is:
regex.match("a++ab", "aaaaaaaaaab")
regex.match("(?>a+)ab", "aaaaaaaaaab")
When you use (?:a+) or a+ the regex engine (by default) records (in prevision) all backtracking positions for all characters. But when you use an atomic group or a possessive quantifier, theses backtracking positions are no more recorded (except for the begining of the group). So when the backtracking mechanism occurs the last “a” character can’t be given back. Only the entire group can be given back.
[EDIT]: the pattern can be written in a more efficient way if you use an “unrolled” subpattern to describe the content between brackets:
{([^{}]*+(?:(?R)[^{}]*)*+)}
I was able to do this no problem with the b(?:m|(?R))*e syntax:
{((?:[^{}]|(?R))*)}
I think the key from what you were attempting is that the repetition doesn’t go on m, but the entire (?:m|(?R)) group. This is what allows the recursion with the (?R) reference.
This is very much related to Regular Expression to match outer brackets however, I specifically want to know how or whether it’s possible to do this regex’s recursive pattern? I’m yet to find a python example using this strategy so think this ought to be a useful question!
I’ve seen some claims that recursive patterns can be used to match balanced parenthesis, but no examples using python’s regex package (Note: re does not support recursive pattern, you need to use regex).
One claim is that syntax is b(?:m|(?R))*e where:
bis what begins the construct,mis what can occur in the middle of the construct, andeis what can occur at the end of the construct
I want to extract matches for the outer braces in the following:
"{1, {2, 3}} {4, 5}"
["1, {2, 3}", "4, 5"] # desired
Note that this is easy to do the same for inner braces:
re.findall(r"{([^{}]*)}", "{1, {2, 3}} {4, 5}")
['2, 3', '4, 5']
(In my example I was using finditer (over match objects), see here.)
So I had hoped that the following, or some variation, would work:
regex.findall(r"{(:[^{}]*|?R)}", "{1, {2, 3}} {4, 5}")
regex.findall(r"({(:[^{}]*|?R)})", "{1, {2, 3}} {4, 5}")
regex.findall(r"({(:.*|(?R))*})", "{1, {2, 3}} {4, 5}")
regex.findall(r"({(:.*)|(?R)*})", "{1, {2, 3}} {4, 5}")
regex.findall(r"({(:[^{}])|(?R)})", "{1, {2, 3}} {4, 5}")
but I’m scuppered by either [] or error: too much backtracking.
Is it possible to extract match objects for the outer parenthesis using regex’s recursion?
Obviously, I run the risk of being shot down with:
- don’t parse html with regex
- do this with pyparse
- write a proper lexer & parser e.g. using ply
I want to emphasis this is about how to use the recursive pattern (which if my understanding is correct, takes us outside of regular language parsing, so may can actually be possible!). If it can be done, this ought to be a cleaner solution.
The pattern is:
{((?>[^{}]+|(?R))*)}
You can see this works for your example:
regex.findall("{((?>[^{}]+|(?R))*)}", "{1, {2, 3}} {4, 5}")
# ['1, {2, 3}', '4, 5']
Explanation:
The m part needs to exclude the brackets. The use of an atomic group is needed if you want at the same time to allow a quantifier for [^{}] and to repeat the group without catastropic backtracking problems. To be more clear, if the last closing curly bracket is missing this regex engine will backtrack atomic group by atomic group instead of character by character. To drive home this point, you can make the quantifier possessive like that: {((?>[^{}]+|(?R))*+)} (or {((?:[^{}]+|(?R))*+)} since the atomic group is no more useful).
The atomic group (?>....) and the possessive quantifier ?+, *+, ++ are the two sides of the same feature. This feature forbids the regex engine to backtrack inside the group of characters that becomes an “atom” (something you can’t divide in smaller parts).
The basic examples are the following two patterns that always fail for the string aaaaaaaaaab:
(?>a+)ab
a++ab
that is:
regex.match("a++ab", "aaaaaaaaaab")
regex.match("(?>a+)ab", "aaaaaaaaaab")
When you use (?:a+) or a+ the regex engine (by default) records (in prevision) all backtracking positions for all characters. But when you use an atomic group or a possessive quantifier, theses backtracking positions are no more recorded (except for the begining of the group). So when the backtracking mechanism occurs the last “a” character can’t be given back. Only the entire group can be given back.
[EDIT]: the pattern can be written in a more efficient way if you use an “unrolled” subpattern to describe the content between brackets:
{([^{}]*+(?:(?R)[^{}]*)*+)}
I was able to do this no problem with the b(?:m|(?R))*e syntax:
{((?:[^{}]|(?R))*)}
I think the key from what you were attempting is that the repetition doesn’t go on m, but the entire (?:m|(?R)) group. This is what allows the recursion with the (?R) reference.