- regex is used to manipulate text data via pattern language
- there are a few key functions under regex that we can use
- we first have to use
import re
Functions
re.sub()
- this function replaces matches with a string and returns the new string
re.findall()
- this function is to return a list of every match in the string
re.search()
- this function returns a match object for the first match anywhere in the string (or None)
re.split()
- returns a list of the string split at each match
Patterns
The individual special characters are metacharacters. What a pattern finds in the text is a match.
| Piece | Category | Description |
|---|---|---|
. | Metacharacter | Matches any single character except newline |
\. | Escaped literal | Matches an actual full stop . |
[] | Character class | Matches any one character listed inside the brackets |
\[ \] | Escaped literal | Matches an actual square bracket |
[a-d] | Character class | Matches any one character in the range a to d |
\ | Escape | Removes the special meaning of the next character, or starts a special sequence |
\\ | Escaped literal | Matches an actual backslash \ |
| | Alternation | Matches the expression on the left or the right |
| | Escaped literal | |
() | Capture group | Groups characters into one unit and stores what it matched |
\( \) | Escaped literal | Matches an actual round bracket |
* | Quantifier | Zero or more of the preceding element |
\* | Escaped literal | Matches an actual asterisk * |
+ | Quantifier | One or more of the preceding element |
\+ | Escaped literal | Matches an actual plus sign + |
? | Quantifier | Zero or one of the preceding element |
\? | Escaped literal | Matches an actual question mark ? |
{n} | Quantifier | Exactly n of the preceding element |
\{ \} | Escaped literal | Matches an actual curly brace |
^abc | Anchor | ^ outside brackets: matches at the start of the string |
[^abc] | Negated class | ^ inside brackets, in first position: matches any character except a, b or c |
[a^b] | Character class | ^ inside brackets, not first: loses its meaning, matches a literal ^ |
(^abc) | Capture group | ^ inside round brackets still anchors, it does not negate |
\^ | Escaped literal | Matches an actual caret ^ |
$ | Anchor | Matches at the end of the string |
\$ | Escaped literal | Matches an actual dollar sign $ |
\A | Anchor | Matches only at the very start of the string |
\Z | Anchor | Matches only at the very end of the string |
\b | Anchor | Matches at a word boundary (start or end of a word) |
\B | Anchor | Matches where there is not a word boundary |
\d | Shorthand class | Matches any digit, equivalent to [0-9] |
\D | Shorthand class | Matches any non-digit |
\w | Shorthand class | Matches any word character: letter, digit, or underscore |
\W | Shorthand class | Matches any non-word character |
\s | Shorthand class | Matches any whitespace character |
\S | Shorthand class | Matches any non-whitespace character |
Examples
string='''
One ring to rule them all,
One ring to find them, One ring to bring them all,
and in the darkness, bind them.
'''
# the () is being used to group these elements together, such that we will only replace a group of <.*?> altogether
a = re.sub('(<.*?>)', ' ', string)
print(a)One ring to rule them all,
One ring to find them, One ring to bring them all,
and in the darkness, bind them.
# to remove full stop, comma, etc, we need to treat each element as itself and not group them, therefore we are using the set function, i.e. []
b = re.sub('[<.,*?>]', ' ', string)
print(b)One ring to rule them all
One ring to find them One ring to bring them all
and in the darkness bind them
# to remove all non-word characters like (, . ' ' \n)
c = re.sub(r'\W', '', string)
print(c)OneringtorulethemallOneringtofindthemOneringtobringthemallandinthedarknessbindthem
# to remove whitespace characters, \n and ' '
d = re.sub(r'\s', '', string)
print(d)Oneringtorulethemall,Oneringtofindthem,Oneringtobringthemall,andinthedarkness,bindthem.
g1 = re.sub(r'^One ring', 'REPLACED', string.strip()) # strip removes the leading \n, which we are then allowed to replace the first 'One Ring' in this string
print(g1)
print('=========')
g2 = re.sub(r'^One ring', 'REPLACED', string, flags=re.MULTILINE) # this using a flag then allows for multiline, which we are allowed to change it for every line
print(g2)REPLACED to rule them all,
One ring to find them, One ring to bring them all,
and in the darkness, bind them.
=========
REPLACED to rule them all,
REPLACED to find them, One ring to bring them all,
and in the darkness, bind them.
string2 = '''
abcdefg,
hijklmnop,
qrstuvwxyz.
'''
e1 = re.sub('(abcde)', '', string2) # removing abcde as 1 group
print(e1)
e2 = re.sub(r'\W', '', e1) # removing all newline characters, fullstops and commas
print(e2)
fg,
hijklmnop,
qrstuvwxyz.
fghijklmnopqrstuvwxyz
# What [a-zA-Z] does is it replaces any of these a-z and A-Z occurances in the string with an empty string, resulting in ,.\n
e3 = re.sub('[a-zA-Z]', '', string2)
print(e3)
,
,
.
'''
IMPORTANT:
findall is just used to find the match, and then return the match, no replacement argument needed unlike re.sub
escape . is used as a literal dot, that matches an actual dot. But if we use a . then it is representing ANY character
MULTILINE flag can be omitted when the word we are trying to match is not in the middle of a line in a string, where ...\n...{something here}\n...\n.
This is true for both ^ and $.
'''
e4 = re.findall(r'xyz\.$', string2, flags=re.MULTILINE)
print(e4)
print('========')
e5 = re.findall(r'xyz.$', string2)
print(e5)['xyz.']
========
['xyz.']
test_string = '''cat sat on mat
dog ran and run in park
dogg flew over hat flew
who let the doogs out
ooi hello'''a = re.findall(r'dog{2}', test_string) # {2} just means the number of occurances of g = 2. meaning we will print dogg since it has 2 occurances of g
print(f'a = {a}')
b = re.findall(r'do{2}g', test_string) # {2} right after o, just means that we are finding all with d, 2 o, and a g.
print(f'b = {b}')
c = re.findall(r'o{2}', test_string) # this just means we are finding all instances where there is 'oo', in any sentence. We get the first 'oo' from 'doog' and second 'oo' from 'ooi'
print(f'c = {c}')
d = re.findall(r'do?g', test_string) # ? means zero or one occurance. so 1 occurance of 'o' means 'dog'. Therefore, we have returned 'dog' from line 2 and 'dogg' as 'dog' from line 3.
print(f'd = {d} is the same as...')
e = re.findall(r'do{1}g', test_string)
print(f'e = {e}')a = ['dogg']
b = ['doog']
c = ['oo', 'oo']
d = ['dog', 'dog'] is the same as...
e = ['dog', 'dog']
f = re.findall(r'ran|run', test_string) # either or command |, to find either run or ran. it found both,so it will return both
print(f)
g = re.findall(r'flew|fly', test_string) # finding either flew or fly, it found 2 occurances of flew, so it will return both occurances of flew.
print(g)
h = re.findall(r'cowboy hat|cap', test_string)
print(h)['ran', 'run']
['flew', 'flew']
[]
a = re.search(r'\Acat', test_string) # matches cat, which is at the start of a string. will return nothing if we try to match dog since the start of a newline but not start of a string.
print(a)<re.Match object; span=(0, 3), match='cat'>
b = re.findall(r'\bflew\b', test_string) # \b ..... \b is used to match the word within the boundary. In this case its 'flew', and it will print both occurances.
print(b)['flew', 'flew']
re.findall(r'\D', test_string) # finds everything that is NOT a digit \D, and returns it
# if we wanna find everything that IS a digit, we use \d['c',
'a',
't',
' ',
's',
'a',
't',
' ',
'o',
'n',
' ',
'm',
'a',
't',
'\n',
'd',
'o',
'g',
' ',
'r',
'a',
'n',
' ',
'a',
'n',
'd',
' ',
'r',
'u',
'n',
' ',
'i',
'n',
' ',
'p',
'a',
'r',
'k',
'\n',
'd',
'o',
'g',
'g',
' ',
'f',
'l',
'e',
'w',
' ',
'o',
'v',
'e',
'r',
' ',
'h',
'a',
't',
' ',
'f',
'l',
'e',
'w',
'\n',
'w',
'h',
'o',
' ',
'l',
'e',
't',
' ',
't',
'h',
'e',
' ',
'd',
'o',
'o',
'g',
's',
' ',
'o',
'u',
't',
'\n',
'o',
'o',
'i',
' ',
'h',
'e',
'l',
'l',
'o']
testing123 = '''testioh 123456789
fgewfewf 2132r176483264
bfnewugr27182173.'''
re.findall(r'\d', testing123) # this returns all the digits['1',
'2',
'3',
'4',
'5',
'6',
'7',
'8',
'9',
'2',
'1',
'3',
'2',
'1',
'7',
'6',
'4',
'8',
'3',
'2',
'6',
'4',
'2',
'7',
'1',
'8',
'2',
'1',
'7',
'3']
testing123 = '''testioh 123456789
fgewfewf 2132r176483264
bfnewugr27182173.'''
parts = re.split(r'\s+', testing123) # this splits the texts wherever there is whitespace
print(parts)['testioh', '123456789', 'fgewfewf', '2132r176483264', 'bfnewugr27182173.']
test_string = '''cat sat on mat
dog ran and run in park
dogg flew over hat flew
who let the doogs out
ooi hello'''
print(re.split(r'ran', test_string))['cat sat on mat\ndog ', ' and run in park\ndogg flew over hat flew\nwho let the doogs out\nooi hello']
multi = 'line one\nline two\nline three'
print(re.findall(r'\w+$', multi, flags=re.MULTILINE)) # \w finds a word character and + means one or more occurances['one', 'two', 'three']
print(re.findall(r'\w+\Z', multi, flags=re.MULTILINE)) # \Z only matches at the very end of the WHOLE string, MULTIlINE or not['three']
Regex is the first cleaning step, and the steps that follow it are listed in Preprocessing Techniques.