Pipeline

The NLP pipeline is the order in which text is cleaned, labelled, and turned into numbers. Every technique in this folder sits at exactly one layer of it, and the layer tells you what the technique consumes and what it produces.

The Five Layers

Layer	What is this layer about?	Techniques
0. Tools	The tools we can use for this pipeline	Regular Expression (RegEx), SpaCy
1. Preprocessing	The preprocessing techniques we can use	Tokenization, Normalization, Stop Word Removal, Stemming, Lemmatization
2. Linguistic Annotation	How we can understand each token and how they relate to one another	Part-of-Speech (POS) Tagging, Dependency Parsing, Named Entity Recognition (NER), Word Sense Disambiguation (WSD)
3. Representation	Changing words into numbers	Vectorization, N-gram Language Models
4. Application	What we are trying to do	Information Extraction

Two of these are not techniques at all:

• Preprocessing Techniques is a category name that covers the whole of Layer 1
• SpaCy is a library that implements Layers 1 and 2 for you

The Pipeline

graph TD
    RAW["Raw Text"] --> RE["Regular Expression"]
    RE --> TOK["Tokenization"]

    subgraph L2["Layer 2: Linguistic Annotation"]
        POS["POS Tagging"]
        DEP["Dependency Parsing"]
        NER["Named Entity Recognition"]
        WSD["Word Sense Disambiguation"]
        POS --> DEP
        POS --> WSD
    end

    subgraph L1["Layer 1: Preprocessing"]
        NORM["Normalization"]
        STOP["Stop Word Removal"]
        STEM["Stemming"]
        LEM["Lemmatization"]
        NORM --> STOP
        NORM --> STEM
    end

    subgraph L3["Layer 3: Representation"]
        VEC["Vectorization"]
        NGRAM["N-gram Language Model"]
    end

    TOK --> POS
    TOK --> NER
    TOK --> NORM
    TOK --> NGRAM
    POS --> LEM
    STOP --> VEC
    STEM --> VEC
    LEM --> VEC

    DEP --> IE["Information Extraction"]
    NER --> IE
    WSD --> IE
    RE -.-> IE
    VEC --> ML["Downstream Model"]
    NGRAM --> ML

There are two routes through the pipeline. The left route strips the sentence down for a bag-of-words model. The right route keeps the sentence intact so a parser can read its grammar.

What Needs What

Technique	Requirement	Reason
Everything below Layer 0	Tokenization	you need to split before being able to label anything
Lemmatization	POS Tagging	the lemma of saw depends on whether it is a verb or a noun
Dependency Parsing	POS Tagging	the parser uses tags as input features
Word Sense Disambiguation	POS Tagging and a context window	Lesk compares the dictionary gloss against the neighbouring words
Vectorization	Tokenization, optionally stems or lemmas	stems and lemmas shrink the vocabulary and the matrix
N-gram Language Model	Tokenization only	it works directly on the raw token sequence
Stemming	nothing	it is pure suffix rules, with no dictionary and no POS

That last row is the whole point of stemming which is fast and crude. Lemmatization on the otherhand is slow but more accurate.

Order Traps

These are the sequencing errors that break the pipeline:

• stop word removal before dependency parsing destroys the tree, because the parser needs is, by and the
• stemming or lemmatization before POS tagging or NER breaks both taggers, because they read the surface form and the capitalisation
• lemmatization before POS tagging is backwards, since the lemma depends on the tag
• stop word removal before an N-gram model or sentiment work deletes not, and the meaning flips

The safe default order:

regex clean --> tokenize --> POS tag --> parse / NER / WSD
            --> normalize --> stop words --> lemmatize --> vectorize

Aggressive preprocessing belongs to the bag-of-words route only. The linguistic annotation route wants the sentence intact. Picking the wrong route for the task is the most common mistake.

What SpaCy Covers

Task	In spaCy?	Note
Tokenization	Yes	rules plus exception lists, not a plain regex split
POS Tagging	Yes	token.pos_ for coarse tags, token.tag_ for fine tags
Dependency Parsing	Yes	token.dep_, token.head, token.children
NER	Yes	doc.ents
Lemmatization	Yes	token.lemma_, uses the POS tag
Stop Word Removal	Yes	token.is_stop
Vectorization	Yes	word vectors in the medium and large models
Stemming	No	spaCy has no stemmer by design, use NLTK PorterStemmer

The last row is easy to get wrong. For stemming the answer is NLTK, never spaCy.

Layer 3 ends with a matrix of counts, and the course carries on from there with Term Weighting Schemes.