• regex is used to manipulate text data via pattern language
  • there are a few key functions under regex that we can use
  • we first have to use import re

Functions

re.sub()

  • this function replaces matches with a string and returns the new string

re.findall()

  • this function is to return a list of every match in the string

re.search()

  • this function returns a match object for the first match anywhere in the string (or None)

re.split()

  • returns a list of the string split at each match

Patterns

What a pattern finds in the text is a match. There are 3 types of pattern pieces:

  • metacharacters are characters with a special function in the pattern
  • special sequences are a \ followed by a character, and each one matches a specific group of characters
  • sets are characters inside square brackets [ ], and each set matches one character

Metacharacters

CharacterMeaningExample
[]A set of characters[a-m]
\Special sequence or escape\.
.Any character except newlinec.t
^Starts with^hello
$Ends withplanet$
*Zero or more timeshe.*o
+One or more timeshe.+o
?Zero or one timecolou?r
{}Exactly this number of timeshe.{2}o
|Either orstart|stop
()Group(hello)+

Special Sequences

SequenceMeaningExample
\AAt the start of the string\AThe
\bAt the start or end of a word\bain, ain\b
\BNot at the start or end of a word\Bain, ain\B
\d / \DDigit / not a digit\d
\s / \SSpace character / not a space character\s
\w / \WWord character (a to z, A to Z, 0 to 9, _) / not a word character\w
\ZAt the end of the stringSpain\Z

Sets

SetMeaning
[ntu]One of n, t or u
[a-d]One lower-case letter from a to d
[^ntu]Any character except n, t and u
[0123], [0-9]One of these digits
[0-5][0-9]A two-digit number from 00 to 59
[a-zA-Z]One letter, lower case or upper case
[+]The character + with no special meaning

Examples

These are the 2 test strings that most of the examples use.

import re
 
string = '''
One ring to rule them all,
One ring to find them, One ring to bring them all,
and in the darkness, bind them.
'''
 
test_string = '''cat sat on mat
dog ran and run in park
dogg flew over hat flew
who let the doogs out
ooi hello'''

Remove punctuation and spaces

  • [^\w\s] removes only the punctuation, because it matches a character that is not a word character and not a space
  • \W also matches spaces and \n, so it joins all the words together
  • \s removes only the spaces and \n, so the punctuation stays
sentence = 'Harper is a good girl!'
 
print(re.sub(r'[^\w\s]', '', sentence))
print(re.sub(r'\W', '', sentence))
print(re.sub(r'\s', '', sentence))
Harper is a good girl
Harperisagoodgirl
Harperisagoodgirl!

Inside [ ], the characters ., * and ? lose their special meaning. Each one is a normal character.

print(re.sub(r'[<.,*?>]', ' ', string))

One ring to rule them all 
One ring to find them  One ring to bring them all 
and in the darkness  bind them 

These 3 cleaning steps are done one after the other to keep only the letters.

clean = re.sub(r'<.*?>', ' ', string)       # remove HTML tags such as <b>
clean = re.sub(r'[,\.!?:()"]', '', clean)   # remove these punctuation marks
clean = re.sub(r'[^a-zA-Z"]', ' ', clean)   # change each character that is not a letter or " into a space
print(clean)
 One ring to rule them all One ring to find them One ring to bring them all and in the darkness bind them 

Sets

print(re.sub(r'[aeiou]', '*', 'Harper is a good girl!'))   # change each vowel
print(re.sub(r'[A-Z]', '*', 'NTU contact number'))         # change each capital letter
print(re.sub(r'[a-zA-Z]', '', 'abc, def.'))                 # remove each letter
H*rp*r *s * g**d g*rl!
*** contact number
, .

Any character with . and *

  • . matches any character except \n
  • .* matches as much text as possible, up to the last possible match
  • .*? matches as little text as possible, up to the first possible match
# 2 characters, then g, then as much as possible up to the last d
print(re.sub(r'..g.*d', 'the best', 'Harper is a good girl!'))
 
html = '<b>bold</b> text'
print(re.sub(r'<.*?>', '', html))   # removes each tag alone
print(re.sub(r'<.*>', '', html))    # goes from the first < to the last >, so the word bold is removed too
Harper is the best girl!
bold text
 text

Quantifiers ?, {n} and +

print(re.findall(r'dog{2}', test_string))   # {2} applies only to g, so it finds 'dogg'
print(re.findall(r'do{2}g', test_string))   # {2} applies only to o, so it finds 'doog'
print(re.findall(r'o{2}', test_string))     # finds 'oo' in 'doogs' and 'oo' in 'ooi'
print(re.findall(r'do?g', test_string))     # 'dog' in line 2, and 'dog' inside 'dogg' in line 3
['dogg']
['doog']
['oo', 'oo']
['dog', 'dog']

do?g and do{1}g are different patterns. ? also allows zero o, so do?g also matches 'dg'.

print(re.findall(r'do?g', 'dg dog'))
print(re.findall(r'do{1}g', 'dg dog'))
['dg', 'dog']
['dog']

+ means one or more times, so \s+ matches a sequence of spaces as one match.

number = 'NTU contact number is 6791       1744.'
 
print(re.sub(r'\s+', ' ', number))   # a sequence of spaces becomes 1 space
print(re.sub(r'\d', 'X', number))    # each digit becomes X
NTU contact number is 6791 1744.
NTU contact number is XXXX       XXXX.

Either or with |

| splits the whole pattern into 2 parts. cowboy hat|cap means 'cowboy hat' or 'cap', not 'cowboy hat' or 'cowboy cap'.

print(re.findall(r'ran|run', test_string))
print(re.findall(r'flew|fly', test_string))
print(re.findall(r'cowboy hat|cap', 'a cowboy hat, a cap, a cowboy cap'))
['ran', 'run']
['flew', 'flew']
['cowboy hat', 'cap', 'cap']

Start and end with ^, $, \A and \Z

  • without MULTILINE, ^ matches only at the start of the whole string
  • without MULTILINE, $ matches only at the end of the whole string, or before 1 final \n
  • with re.MULTILINE, ^ and $ match at the start and end of each line
  • \A and \Z always match only at the start and end of the whole string, even with MULTILINE

string starts with \n, so ^One ring finds nothing until strip() removes that \n.

print(re.sub(r'^One ring', 'REPLACED', string.strip()))
print('=========')
print(re.sub(r'^One ring', 'REPLACED', string, flags=re.MULTILINE))
REPLACED to rule them all,
One ring to find them, One ring to bring them all,
and in the darkness, bind them.
=========

REPLACED to rule them all,
REPLACED to find them, One ring to bring them all,
and in the darkness, bind them.

\. matches only a full stop, but . matches any character.

string2 = '''
abcdefg,
hijklmnop,
qrstuvwxyz.
'''
 
print(re.findall(r'xyz\.$', string2))                      # works without the flag, because 'xyz.' is before the final \n
print(re.findall(r'hijklmnop,$', string2))                 # finds nothing, because this line is not the last line
print(re.findall(r'hijklmnop,$', string2, flags=re.MULTILINE))
print(re.findall(r'^abc', string2))                        # finds nothing, because string2 starts with \n
print(re.findall(r'^abc', string2, flags=re.MULTILINE))
['xyz.']
[]
['hijklmnop,']
[]
['abc']
multi = 'line one\nline two\nline three'
 
print(re.findall(r'\w+$', multi, flags=re.MULTILINE))   # the last word of each line
print(re.findall(r'\w+\Z', multi, flags=re.MULTILINE))  # \Z ignores MULTILINE, so only the last word of the string
print(re.search(r'\Acat', test_string))                  # 'cat' is at the start of the string
print(re.search(r'\Adog', test_string))                  # 'dog' is at the start of a line, not of the string
['one', 'two', 'three']
['three']
<re.Match object; span=(0, 3), match='cat'>
None

Word boundaries with \b and \B

  • \b matches at the start or end of a word
  • \B matches only inside a word
sentence = 'Harper is a good girl!'
 
print(re.findall(r'\bflew\b', test_string))
print(re.sub(r'\bgood\b', 'nice', sentence))
print(re.sub(r'\b\w{2}\b', '**', sentence))   # a full word with exactly 2 letters
print(re.sub(r'\B\w{2}\B', '**', sentence))   # 2 letters inside a word, not at its start or end
['flew', 'flew']
Harper is a nice girl!
Harper ** a good girl!
H****r is a g**d g**l!

Groups and back-references

( ) makes a group. In the replacement of re.sub, \1, \2 and \3 are the text that group 1, 2 and 3 matched.

print(re.sub(r'(\d{4})(\s+)(\d{4})', r'\3\2\1', number))   # swap the 2 numbers and keep the spaces
print(re.sub(r'(\d{4})\s+(\d{4})', r'\2 \1', number))      # swap the 2 numbers with 1 space between them
NTU contact number is 1744       6791.
NTU contact number is 1744 6791.

When the pattern has a group, findall returns only the text inside the group.

print(re.findall(r'\d{4}\s+\d{4}', number))
print(re.findall(r'(\d{4})\s+\d{4}', number))
['6791       1744']
['6791']

Digits and whitespace with \d, \D and \s

testing123 = '''testioh 123
fgewfewf 45r6
bfnewugr78.'''
 
print(re.findall(r'\d', testing123))    # each digit
print(re.findall(r'\d+', testing123))   # each sequence of digits
print(re.findall(r'\D', 'a1 b2'))       # each character that is not a digit
print(re.split(r'\s+', testing123))     # split at each sequence of spaces or \n
['1', '2', '3', '4', '5', '6', '7', '8']
['123', '45', '6', '78']
['a', ' ', 'b']
['testioh', '123', 'fgewfewf', '45r6', 'bfnewugr78.']

search and split

  • re.search() returns a match object for the first match only, or None
  • re.split() removes the match and splits the string at that place
print(re.search(r'flew', test_string))
print(re.search(r'fly', test_string))
print(re.split(r'ran', test_string))
<re.Match object; span=(44, 48), match='flew'>
None
['cat sat on mat\ndog ', ' and run in park\ndogg flew over hat flew\nwho let the doogs out\nooi hello']

/ is not \

/W is a forward slash and the letter W, so it matches only the text '/W'. \W is a backslash sequence, so it matches each non-word character.

text = '/W Never willing to see you fall. /s Never hesitating.'
 
print(re.sub(r'/W', ' ', text))
print(re.sub(r'\W', ' ', text))
  Never willing to see you fall. /s Never hesitating.
 W Never willing to see you fall   s Never hesitating 

Regex is the first cleaning step, and the steps that follow it are listed in Preprocessing Techniques.