Definition
- Part-of-speech is the process of reading a sentence and automatically labelling each word with its corresponding grammatical role based on its context
Purpose & Function
- POS allows us to understand structure and semantic context of words in speech
- POS tagging allows computers to contextualise these words for a more accurate processing and comprehension
- Previously we did this for entities, but now under POS tagging, we are doing it for all tokens
Tagging Methods
- using linguistic patterns, context and predefined dictionaries
- probability models, such as Hidden Markov Models
# Load spacy's smal english model, define our sentence, run sentence thru model
nlp = spacy.load("en_core_web_sm")
sentence = "This course is lectured by Dr. S. Supraja, and Simon Liu at NTU, Singapore."
doc = nlp(sentence)
# Process the text for coarse-grained POS tagging
for token in doc:
print(f"{token.text}[{token.pos_}]")This[DET]
course[NOUN]
is[AUX]
lectured[VERB]
by[ADP]
Dr.[PROPN]
S.[PROPN]
Supraja[PROPN]
,[PUNCT]
and[CCONJ]
Simon[PROPN]
Liu[PROPN]
at[ADP]
NTU[PROPN]
,[PUNCT]
Singapore[PROPN]
.[PUNCT]
# Process the text for fine-grained POS tagging
for token in doc:
print(f"{token.text}[{token.tag_}]")This[DT]
course[NN]
is[VBZ]
lectured[VBN]
by[IN]
Dr.[NNP]
S.[NNP]
Supraja[NNP]
,[,]
and[CC]
Simon[NNP]
Liu[NNP]
at[IN]
NTU[NNP]
,[,]
Singapore[NNP]
.[.]
token.pos_gives course-grained tag- it comes from Universal POS tagset (UPOS) that contains a small, language-independent set of about 17 broad categories
token.tag_gives fine-grained tag- it come from detailed, language-specific tagset (Penn Treebank tagset) with 40+ tags that encode morphological detail like tense, number, and degree
Hidden Markov Model (HMM)
HMM is a stochastic technique for POS tagging. It works by leveraging state transitions and observations to determine the most likely sequence for POS tags in a sentence.
In HMM, we treat the grammar system like a machine with Hidden States and visible Observations.
Components of a HHM:
Hidden States (POS Tags): Specific tags like 'noun', 'verb', 'adjective' hidden behind the text
Observations: Each observation corresponds to a physical word in a sentence
2 types of probabilities used in HHM:
Transition Probability: The probability of transitioning from one POS tag to another
Emission Probability: The probability of a certain word being emitted from a POS tag
"Emitted" just means "outputs" or "generates". Emission Probability tells us if the hidden tag is really a "Verb", what is the likelihood that it produces a word "spot" in this sentence. If it is likely means that the hidden tag is predicted to be a "Verb".
Emission Probability is represented as:
The Final Formula:
Example
Ok imagine we are trying to create a simple HHM-based POS tagger with 3 tags: {Noun, Modal, Verb}.
Our Sentence: John can see Will
In terms of their True Tags: Noun, Modal, Verb, Noun
The transition probability is the probability that a POS tag will be followed by another. Example, the probability of tag Noun given tag Verb.
The emission probability represents the probability that the noun tag will emit the word John.
If our training data consist of 4 sentences:
- Mary Jane can see Will.
- Spot will see Mary.
- Will Jane spot Mary.
- Mary will pat Spot.
First, we calculate total number of occurances of each word in the training dataset:
Next, we calculate emission probabilities:
To calculate transition probability, we need to add 2 more tags, <s> and <e>, representing start and end of a sentence.
Next, we see the co-occurance probability, for example, <s> followed by a noun is .
The above shows how we can represent the co-occurance matrix. The entries show the co-occurance probabilities of a tag co-occuring with another tag.
From the above, by multiplying these probabilities together, we can determine the most likely tags for words in the sentence.
However, as sentences get longer, when we calculate probabilities one by one, the number of permuntations is gonna be exponentially higher. So we use probability trees.
Viterbi algorithm and backtrace
The Viterbi algorithm finds the most likely complete POS-tag sequence. It stores intermediate path scores so that the same partial paths can be reused.
- Start at
<s>and calculate the score for each possible tag of the first word - Move to the next word and consider each possible current tag
- For each previous tag, combine the previous path score, the transition probability, and the current word's emission probability
- Keep the highest score for each current tag, together with a backpointer to the previous tag that produced it
- Continue to the final word, include the transition to
<e>, and select the best complete path - Backtrace from the endpoint through the stored backpointers to the start
- Reverse the recovered tag sequence to read it in sentence order
A backpointer records which earlier state belongs to the best path for a current state. The forward pass calculates scores. The backtrace recovers the tag sequence.
For Will can spot Mary, the recovered sequence is:
<s> -> Noun -> Modal -> Verb -> Noun -> <e>
Will can spot MaryThe best complete path can differ from the tag with the highest local score at one word. Keep the best path to each state until the final decision.
nlp.pipe() is used to process text in batches rather than one by one which improves the efficiency of text processing.
from nltk import Tree
def tok_format(tok, coarse=False):
if coarse:
return "[".join([tok.orth_, tok.pos_]) + "]"
return "[".join([tok.orth_, tok.tag_]) + "]"
def to_nltk_tree(node, coarse=False):
if node.n_lefts + node.n_rights > 0:
return Tree(tok_format(node), [to_nltk_tree(child, coarse=coarse) for child in node.children])
else:
return tok_format(node, coarse=coarse)
[to_nltk_tree(sent.root, coarse=True).pretty_print() for sent in doc.sents];
[to_nltk_tree(sent.root).pretty_print() for sent in doc.sents];
Process
- Tokenize
- then POS tag them
Applications
A tag is what Lemmatization needs before it can pick the right dictionary form.