- regex is used to manipulate text data via pattern language
- there are a few key functions under regex that we can use
- we first have to use
import re
Functions
re.sub()
- this function replaces matches with a string and returns the new string
re.findall()
- this function is to return a list of every match in the string
re.search()
- this function returns a match object for the first match anywhere in the string (or None)
re.split()
- returns a list of the string split at each match
Patterns
What a pattern finds in the text is a match. There are 3 types of pattern pieces:
- metacharacters are characters with a special function in the pattern
- special sequences are a
\followed by a character, and each one matches a specific group of characters - sets are characters inside square brackets
[ ], and each set matches one character
Metacharacters
| Character | Meaning | Example |
|---|---|---|
[] | A set of characters | [a-m] |
\ | Special sequence or escape | \. |
. | Any character except newline | c.t |
^ | Starts with | ^hello |
$ | Ends with | planet$ |
* | Zero or more times | he.*o |
+ | One or more times | he.+o |
? | Zero or one time | colou?r |
{} | Exactly this number of times | he.{2}o |
| | Either or | start|stop |
() | Group | (hello)+ |
Special Sequences
| Sequence | Meaning | Example |
|---|---|---|
\A | At the start of the string | \AThe |
\b | At the start or end of a word | \bain, ain\b |
\B | Not at the start or end of a word | \Bain, ain\B |
\d / \D | Digit / not a digit | \d |
\s / \S | Space character / not a space character | \s |
\w / \W | Word character (a to z, A to Z, 0 to 9, _) / not a word character | \w |
\Z | At the end of the string | Spain\Z |
Sets
| Set | Meaning |
|---|---|
[ntu] | One of n, t or u |
[a-d] | One lower-case letter from a to d |
[^ntu] | Any character except n, t and u |
[0123], [0-9] | One of these digits |
[0-5][0-9] | A two-digit number from 00 to 59 |
[a-zA-Z] | One letter, lower case or upper case |
[+] | The character + with no special meaning |
Examples
These are the 2 test strings that most of the examples use.
import re
string = '''
One ring to rule them all,
One ring to find them, One ring to bring them all,
and in the darkness, bind them.
'''
test_string = '''cat sat on mat
dog ran and run in park
dogg flew over hat flew
who let the doogs out
ooi hello'''Remove punctuation and spaces
[^\w\s]removes only the punctuation, because it matches a character that is not a word character and not a space\Walso matches spaces and\n, so it joins all the words together\sremoves only the spaces and\n, so the punctuation stays
sentence = 'Harper is a good girl!'
print(re.sub(r'[^\w\s]', '', sentence))
print(re.sub(r'\W', '', sentence))
print(re.sub(r'\s', '', sentence))Harper is a good girl
Harperisagoodgirl
Harperisagoodgirl!
Inside [ ], the characters ., * and ? lose their special meaning. Each one is a normal character.
print(re.sub(r'[<.,*?>]', ' ', string))
One ring to rule them all
One ring to find them One ring to bring them all
and in the darkness bind them
These 3 cleaning steps are done one after the other to keep only the letters.
clean = re.sub(r'<.*?>', ' ', string) # remove HTML tags such as <b>
clean = re.sub(r'[,\.!?:()"]', '', clean) # remove these punctuation marks
clean = re.sub(r'[^a-zA-Z"]', ' ', clean) # change each character that is not a letter or " into a space
print(clean) One ring to rule them all One ring to find them One ring to bring them all and in the darkness bind them
Sets
print(re.sub(r'[aeiou]', '*', 'Harper is a good girl!')) # change each vowel
print(re.sub(r'[A-Z]', '*', 'NTU contact number')) # change each capital letter
print(re.sub(r'[a-zA-Z]', '', 'abc, def.')) # remove each letterH*rp*r *s * g**d g*rl!
*** contact number
, .
Any character with . and *
.matches any character except\n.*matches as much text as possible, up to the last possible match.*?matches as little text as possible, up to the first possible match
# 2 characters, then g, then as much as possible up to the last d
print(re.sub(r'..g.*d', 'the best', 'Harper is a good girl!'))
html = '<b>bold</b> text'
print(re.sub(r'<.*?>', '', html)) # removes each tag alone
print(re.sub(r'<.*>', '', html)) # goes from the first < to the last >, so the word bold is removed tooHarper is the best girl!
bold text
text
Quantifiers ?, {n} and +
print(re.findall(r'dog{2}', test_string)) # {2} applies only to g, so it finds 'dogg'
print(re.findall(r'do{2}g', test_string)) # {2} applies only to o, so it finds 'doog'
print(re.findall(r'o{2}', test_string)) # finds 'oo' in 'doogs' and 'oo' in 'ooi'
print(re.findall(r'do?g', test_string)) # 'dog' in line 2, and 'dog' inside 'dogg' in line 3['dogg']
['doog']
['oo', 'oo']
['dog', 'dog']
do?g and do{1}g are different patterns. ? also allows zero o, so do?g also matches 'dg'.
print(re.findall(r'do?g', 'dg dog'))
print(re.findall(r'do{1}g', 'dg dog'))['dg', 'dog']
['dog']
+ means one or more times, so \s+ matches a sequence of spaces as one match.
number = 'NTU contact number is 6791 1744.'
print(re.sub(r'\s+', ' ', number)) # a sequence of spaces becomes 1 space
print(re.sub(r'\d', 'X', number)) # each digit becomes XNTU contact number is 6791 1744.
NTU contact number is XXXX XXXX.
Either or with |
| splits the whole pattern into 2 parts. cowboy hat|cap means 'cowboy hat' or 'cap', not 'cowboy hat' or 'cowboy cap'.
print(re.findall(r'ran|run', test_string))
print(re.findall(r'flew|fly', test_string))
print(re.findall(r'cowboy hat|cap', 'a cowboy hat, a cap, a cowboy cap'))['ran', 'run']
['flew', 'flew']
['cowboy hat', 'cap', 'cap']
Start and end with ^, $, \A and \Z
- without
MULTILINE,^matches only at the start of the whole string - without
MULTILINE,$matches only at the end of the whole string, or before 1 final\n - with
re.MULTILINE,^and$match at the start and end of each line \Aand\Zalways match only at the start and end of the whole string, even withMULTILINE
string starts with \n, so ^One ring finds nothing until strip() removes that \n.
print(re.sub(r'^One ring', 'REPLACED', string.strip()))
print('=========')
print(re.sub(r'^One ring', 'REPLACED', string, flags=re.MULTILINE))REPLACED to rule them all,
One ring to find them, One ring to bring them all,
and in the darkness, bind them.
=========
REPLACED to rule them all,
REPLACED to find them, One ring to bring them all,
and in the darkness, bind them.
\. matches only a full stop, but . matches any character.
string2 = '''
abcdefg,
hijklmnop,
qrstuvwxyz.
'''
print(re.findall(r'xyz\.$', string2)) # works without the flag, because 'xyz.' is before the final \n
print(re.findall(r'hijklmnop,$', string2)) # finds nothing, because this line is not the last line
print(re.findall(r'hijklmnop,$', string2, flags=re.MULTILINE))
print(re.findall(r'^abc', string2)) # finds nothing, because string2 starts with \n
print(re.findall(r'^abc', string2, flags=re.MULTILINE))['xyz.']
[]
['hijklmnop,']
[]
['abc']
multi = 'line one\nline two\nline three'
print(re.findall(r'\w+$', multi, flags=re.MULTILINE)) # the last word of each line
print(re.findall(r'\w+\Z', multi, flags=re.MULTILINE)) # \Z ignores MULTILINE, so only the last word of the string
print(re.search(r'\Acat', test_string)) # 'cat' is at the start of the string
print(re.search(r'\Adog', test_string)) # 'dog' is at the start of a line, not of the string['one', 'two', 'three']
['three']
<re.Match object; span=(0, 3), match='cat'>
None
Word boundaries with \b and \B
\bmatches at the start or end of a word\Bmatches only inside a word
sentence = 'Harper is a good girl!'
print(re.findall(r'\bflew\b', test_string))
print(re.sub(r'\bgood\b', 'nice', sentence))
print(re.sub(r'\b\w{2}\b', '**', sentence)) # a full word with exactly 2 letters
print(re.sub(r'\B\w{2}\B', '**', sentence)) # 2 letters inside a word, not at its start or end['flew', 'flew']
Harper is a nice girl!
Harper ** a good girl!
H****r is a g**d g**l!
Groups and back-references
( ) makes a group. In the replacement of re.sub, \1, \2 and \3 are the text that group 1, 2 and 3 matched.
print(re.sub(r'(\d{4})(\s+)(\d{4})', r'\3\2\1', number)) # swap the 2 numbers and keep the spaces
print(re.sub(r'(\d{4})\s+(\d{4})', r'\2 \1', number)) # swap the 2 numbers with 1 space between themNTU contact number is 1744 6791.
NTU contact number is 1744 6791.
When the pattern has a group, findall returns only the text inside the group.
print(re.findall(r'\d{4}\s+\d{4}', number))
print(re.findall(r'(\d{4})\s+\d{4}', number))['6791 1744']
['6791']
Digits and whitespace with \d, \D and \s
testing123 = '''testioh 123
fgewfewf 45r6
bfnewugr78.'''
print(re.findall(r'\d', testing123)) # each digit
print(re.findall(r'\d+', testing123)) # each sequence of digits
print(re.findall(r'\D', 'a1 b2')) # each character that is not a digit
print(re.split(r'\s+', testing123)) # split at each sequence of spaces or \n['1', '2', '3', '4', '5', '6', '7', '8']
['123', '45', '6', '78']
['a', ' ', 'b']
['testioh', '123', 'fgewfewf', '45r6', 'bfnewugr78.']
search and split
re.search()returns a match object for the first match only, orNonere.split()removes the match and splits the string at that place
print(re.search(r'flew', test_string))
print(re.search(r'fly', test_string))
print(re.split(r'ran', test_string))<re.Match object; span=(44, 48), match='flew'>
None
['cat sat on mat\ndog ', ' and run in park\ndogg flew over hat flew\nwho let the doogs out\nooi hello']
/ is not \
/W is a forward slash and the letter W, so it matches only the text '/W'. \W is a backslash sequence, so it matches each non-word character.
text = '/W Never willing to see you fall. /s Never hesitating.'
print(re.sub(r'/W', ' ', text))
print(re.sub(r'\W', ' ', text)) Never willing to see you fall. /s Never hesitating.
W Never willing to see you fall s Never hesitating
Regex is the first cleaning step, and the steps that follow it are listed in Preprocessing Techniques.