Preprocessing Techniques

The main goal of preprocessing text data, is so that we can convert it into a vector for machine learning models to be able to process it.

Here are 5 key reasons why we use preprocessing:

1. Noise Reduction
   • removing special characters, punctuations, irrelevant infomation for cleaning of data
2. Tokenization
   • break texts into tokens (words/subwords), used in transformer architecture
3. Normalization
   • standardize words (lowercase, stemming, lemmatiation) for reduced vocabulary size
4. Stop Word Removal
   • eliminate common words
5. Handling of OOV words
   • replace out-of-vocabulary (OOV) words with unknown tokens
6. Sentence Segmentation
   • seperate text into sentences for individual analysis
7. Feature Engineering
   • extract linguistic features (n-grams, POS tags) to enhance understanding

There are 3 main ways we can preprocess textual data.

1. Regular Expression (RegEx)
2. Tokenization
3. Stop Word Removal
4. Normalization

All these are performed for Vectorization

Order matters as much as the choice of step, and the safe order is set out in Pipeline.