Definition

  • Part-of-speech is the process of reading a sentence and automatically labelling each word with its corresponding grammatical role based on its context

Purpose & Function

  • POS allows us to understand structure and semantic context of words in speech
  • POS tagging allows computers to contextualise these words for a more accurate processing and comprehension
  • Previously we did this for entities, but now under POS tagging, we are doing it for all tokens

Tagging Methods

  1. using linguistic patterns, context and predefined dictionaries
  2. probability models, such as Hidden Markov Models
# Load spacy's smal english model, define our sentence, run sentence thru model
nlp = spacy.load("en_core_web_sm")
sentence = "This course is lectured by Dr. S. Supraja, and Simon Liu at NTU, Singapore."
 
doc = nlp(sentence)
 
# Process the text for coarse-grained POS tagging
for token in doc:
	print(f"{token.text}[{token.pos_}]")
This[DET]
course[NOUN]
is[AUX]
lectured[VERB]
by[ADP]
Dr.[PROPN]
S.[PROPN]
Supraja[PROPN]
,[PUNCT]
and[CCONJ] 
Simon[PROPN]
Liu[PROPN]
at[ADP]
NTU[PROPN]
,[PUNCT]
Singapore[PROPN]
.[PUNCT]
# Process the text for fine-grained POS tagging
for token in doc:
    print(f"{token.text}[{token.tag_}]")
This[DT]
course[NN]
is[VBZ]
lectured[VBN]
by[IN]
Dr.[NNP]
S.[NNP]
Supraja[NNP]
,[,]
and[CC] 
Simon[NNP]
Liu[NNP] 
at[IN]
NTU[NNP]
,[,] 
Singapore[NNP]
.[.]
  • token.pos_ gives course-grained tag
  • it comes from Universal POS tagset (UPOS) that contains a small, language-independent set of about 17 broad categories
  • token.tag_ gives fine-grained tag
  • it come from detailed, language-specific tagset (Penn Treebank tagset) with 40+ tags that encode morphological detail like tense, number, and degree

nlp.pipe() is used to process text in batches rather than one by one which improves the efficiency of text processing.

from nltk import Tree
 
def tok_format(tok, coarse=False):
	if coarse:
		return "[".join([tok.orth_, tok.pos_]) + "]"
	return "[".join([tok.orth_, tok.tag_]) + "]"
 
def to_nltk_tree(node, coarse=False):
	if node.n_lefts + node.n_rights > 0:
		return Tree(tok_format(node), [to_nltk_tree(child, coarse=coarse) for child in node.children])
	else:
		return tok_format(node, coarse=coarse)
		
[to_nltk_tree(sent.root, coarse=True).pretty_print() for sent in doc.sents];
[to_nltk_tree(sent.root).pretty_print() for sent in doc.sents];

Process

  1. Tokenize
  2. then POS tag them

Applications

A tag is what Lemmatization needs before it can pick the right dictionary form.