JJ

Part-of-Speech (POS) Tagging

4 min read

Definition

  • Part-of-speech is the process of reading a sentence and automatically labelling each word with its corresponding grammatical role based on its context

Purpose & Function

  • POS allows us to understand structure and semantic context of words in speech
  • POS tagging allows computers to contextualise these words for a more accurate processing and comprehension
  • Previously we did this for entities, but now under POS tagging, we are doing it for all tokens

Tagging Methods

  1. using linguistic patterns, context and predefined dictionaries
  2. probability models, such as Hidden Markov Models
# Load spacy's smal english model, define our sentence, run sentence thru model
nlp = spacy.load("en_core_web_sm")
sentence = "This course is lectured by Dr. S. Supraja, and Simon Liu at NTU, Singapore."
 
doc = nlp(sentence)
 
# Process the text for coarse-grained POS tagging
for token in doc:
	print(f"{token.text}[{token.pos_}]"
This[DET]
course[NOUN]
is[AUX]
lectured[VERB]
by[ADP]
Dr.[PROPN]
S.[PROPN]
Supraja[PROPN]
,[PUNCT]
and[CCONJ] 
Simon[PROPN]
Liu[PROPN]
at[ADP]
NTU[PROPN]
,[PUNCT]
Singapore[PROPN]
.[PUNCT]

# Process the text for fine-grained POS tagging
for token in doc:
print(f"{token.text}[{token.tag_}]")
This[DT]
course[NN]
is[VBZ]
lectured[VBN]
by[IN]
Dr.[NNP]
S.[NNP]
Supraja[NNP]
,[,]
and[CC] 
Simon[NNP]
Liu[NNP] 
at[IN]
NTU[NNP]
,[,] 
Singapore[NNP]
.[.]
  • token.pos_ gives course-grained tag
  • it comes from Universal POS tagset (UPOS) that contains a small, language-independent set of about 17 broad categories
  • token.tag_ gives fine-grained tag
  • it come from detailed, language-specific tagset (Penn Treebank tagset) with 40+ tags that encode morphological detail like tense, number, and degree

Hidden Markov Model (HMM)

HMM is a stochastic technique for POS tagging. It works by leveraging state transitions and observations to determine the most likely sequence for POS tags in a sentence.

In HMM, we treat the grammar system like a machine with Hidden States and visible Observations.

Components of a HHM:

Hidden States (POS Tags): Specific tags like 'noun', 'verb', 'adjective' hidden behind the text
Observations: Each observation corresponds to a physical word in a sentence

2 types of probabilities used in HHM:

Transition Probability: The probability of transitioning from one POS tag to another
Emission Probability: The probability of a certain word being emitted from a POS tag

"Emitted" just means "outputs" or "generates". Emission Probability tells us if the hidden tag is really a "Verb", what is the likelihood that it produces a word "spot" in this sentence. If it is likely means that the hidden tag is predicted to be a "Verb".

Emission Probability is represented as:

The Final Formula:

Example

Ok imagine we are trying to create a simple HHM-based POS tagger with 3 tags: {Noun, Model, Verb}.

Our Sentence: John can see Will
In terms of their True Tags: Noun ,Model, Verb, Noun

The transition probability is the probability that a POS tag will be followed by another. Example, the probability of tag Noun given tag Verb.

The emission probability represents the probability that the noun tag will emit the word John.

If our training data consist of 4 sentences:

  1. Mary Jane can see Will.
  2. Spot will see Mary.
  3. Will Jane spot Mary.
  4. Mary will pat Spot.

First, we calculate total number of occurances of each word in the training dataset:

Next, we calculate emission probabilities:

To calculate transition probability, we need to add 2 more tags, <s> and <e>, representing start and end of a sentence.

Next, we see the co-occurance probability, for example, <s> followed by a noun is .

The above shows how we can represent the co-occurance matrix. The entries show the co-occurance probabilities of a tag co-occuring with another tag.

From the above, by multiplying these probabilities together, we can determine the most likely tags for words in the sentence.

However, as sentences get longer, when we calculate probabilities one by one, the number of permuntations is gonna be exponentially higher. So we use probability trees.

We optimize with this Viteribi Algorithm and trace the pace with the highest probabilities.

nlp.pipe() is used to process text in batches rather than one by one which improves the efficiency of text processing.

from nltk import Tree
 
def tok_format(tok, coarse=False):
	if coarse:
		return "[".join([tok.orth_, tok.pos_]) + "]"
	return "[".join([tok.orth_, tok.tag_]) + "]"
 
def to_nltk_tree(node, coarse=False):
	if node.n_lefts + node.n_rights > 0:
		return Tree(tok_format(node), [to_nltk_tree(child, coarse=coarse) for child in node.children])
	else:
		return tok_format(node, coarse=coarse)
		
[to_nltk_tree(sent.root, coarse=True).pretty_print() for sent in doc.sents];
[to_nltk_tree(sent.root).pretty_print() for sent in doc.sents];

Process

  1. Tokenize
  2. then POS tag them

Applications

A tag is what Lemmatization needs before it can pick the right dictionary form.

Graph View

Backlinks