{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "af3b7839-37f7-43ce-a12e-ccb723c1774e",
   "metadata": {},
   "source": [
    "# Feature Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d477afd-c2ab-46e4-9472-9f2000595d20",
   "metadata": {},
   "source": [
    "In this chapter, we will carefully examine our pre-processed training dataset and select the best features for machine learning algorithms. I already processed the training dataset and saved it as joblib file. See [here](Pipeline.ipynb) if you want to know the whole process. Since the dataframe was massive, I had to break it down to three different dataframes to save it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c1dff93c-c6c2-4ad3-bfdc-13a96a43d5a2",
   "metadata": {
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import pickle\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9dcad22e-cbb9-48d8-bdae-6675cbb2935e",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_features_train = joblib.load('./data/text_features_train_jlib')\n",
    "OHE_features_train = joblib.load('./data/OHE_features_train_jlib')\n",
    "processed_train = joblib.load('./data/processed_train_jlib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e641e902-6fa2-4b2b-8333-22dfea106c0a",
   "metadata": {},
   "source": [
    "However, here we face a big problem. If we try to combine these three dataframes into single dataframe by doing \n",
    "\n",
    "```python\n",
    "train_features = pd.concat([text_features_train, OHE_features_train, processed_train], axis = 1)\n",
    "```\n",
    "\n",
    "Then we get the **MemeoryError** because **text_features_train is massive dataframe with 89527 columns (10 GB).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "76f5de60-ba66-4b3c-9fbf-615f05fedf6c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aa_desc</th>\n",
       "      <th>aaa_desc</th>\n",
       "      <th>aaab_desc</th>\n",
       "      <th>aab_desc</th>\n",
       "      <th>aabc_desc</th>\n",
       "      <th>aabd_desc</th>\n",
       "      <th>aabf_desc</th>\n",
       "      <th>aac_desc</th>\n",
       "      <th>aaccd_desc</th>\n",
       "      <th>aachen_desc</th>\n",
       "      <th>...</th>\n",
       "      <th>zodat_benefits</th>\n",
       "      <th>zollman_benefits</th>\n",
       "      <th>zombi_benefits</th>\n",
       "      <th>zone_benefits</th>\n",
       "      <th>zoo_benefits</th>\n",
       "      <th>zowel_benefits</th>\n",
       "      <th>zu_benefits</th>\n",
       "      <th>zult_benefits</th>\n",
       "      <th>zutrifft_benefits</th>\n",
       "      <th>zweig_benefits</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.165596</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14299</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14300</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14301</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14302</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14303</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14304 rows × 89527 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        aa_desc  aaa_desc  aaab_desc  aab_desc  aabc_desc  aabd_desc  \\\n",
       "0      0.165596       0.0        0.0       0.0        0.0        0.0   \n",
       "1      0.000000       0.0        0.0       0.0        0.0        0.0   \n",
       "2      0.000000       0.0        0.0       0.0        0.0        0.0   \n",
       "3      0.000000       0.0        0.0       0.0        0.0        0.0   \n",
       "4      0.000000       0.0        0.0       0.0        0.0        0.0   \n",
       "...         ...       ...        ...       ...        ...        ...   \n",
       "14299  0.000000       0.0        0.0       0.0        0.0        0.0   \n",
       "14300  0.000000       0.0        0.0       0.0        0.0        0.0   \n",
       "14301  0.000000       0.0        0.0       0.0        0.0        0.0   \n",
       "14302  0.000000       0.0        0.0       0.0        0.0        0.0   \n",
       "14303  0.000000       0.0        0.0       0.0        0.0        0.0   \n",
       "\n",
       "       aabf_desc  aac_desc  aaccd_desc  aachen_desc  ...  zodat_benefits  \\\n",
       "0            0.0       0.0         0.0          0.0  ...             0.0   \n",
       "1            0.0       0.0         0.0          0.0  ...             0.0   \n",
       "2            0.0       0.0         0.0          0.0  ...             0.0   \n",
       "3            0.0       0.0         0.0          0.0  ...             0.0   \n",
       "4            0.0       0.0         0.0          0.0  ...             0.0   \n",
       "...          ...       ...         ...          ...  ...             ...   \n",
       "14299        0.0       0.0         0.0          0.0  ...             0.0   \n",
       "14300        0.0       0.0         0.0          0.0  ...             0.0   \n",
       "14301        0.0       0.0         0.0          0.0  ...             0.0   \n",
       "14302        0.0       0.0         0.0          0.0  ...             0.0   \n",
       "14303        0.0       0.0         0.0          0.0  ...             0.0   \n",
       "\n",
       "       zollman_benefits  zombi_benefits  zone_benefits  zoo_benefits  \\\n",
       "0                   0.0             0.0            0.0           0.0   \n",
       "1                   0.0             0.0            0.0           0.0   \n",
       "2                   0.0             0.0            0.0           0.0   \n",
       "3                   0.0             0.0            0.0           0.0   \n",
       "4                   0.0             0.0            0.0           0.0   \n",
       "...                 ...             ...            ...           ...   \n",
       "14299               0.0             0.0            0.0           0.0   \n",
       "14300               0.0             0.0            0.0           0.0   \n",
       "14301               0.0             0.0            0.0           0.0   \n",
       "14302               0.0             0.0            0.0           0.0   \n",
       "14303               0.0             0.0            0.0           0.0   \n",
       "\n",
       "       zowel_benefits  zu_benefits  zult_benefits  zutrifft_benefits  \\\n",
       "0                 0.0          0.0            0.0                0.0   \n",
       "1                 0.0          0.0            0.0                0.0   \n",
       "2                 0.0          0.0            0.0                0.0   \n",
       "3                 0.0          0.0            0.0                0.0   \n",
       "4                 0.0          0.0            0.0                0.0   \n",
       "...               ...          ...            ...                ...   \n",
       "14299             0.0          0.0            0.0                0.0   \n",
       "14300             0.0          0.0            0.0                0.0   \n",
       "14301             0.0          0.0            0.0                0.0   \n",
       "14302             0.0          0.0            0.0                0.0   \n",
       "14303             0.0          0.0            0.0                0.0   \n",
       "\n",
       "       zweig_benefits  \n",
       "0                 0.0  \n",
       "1                 0.0  \n",
       "2                 0.0  \n",
       "3                 0.0  \n",
       "4                 0.0  \n",
       "...               ...  \n",
       "14299             0.0  \n",
       "14300             0.0  \n",
       "14301             0.0  \n",
       "14302             0.0  \n",
       "14303             0.0  \n",
       "\n",
       "[14304 rows x 89527 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_features_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "761884f7-fcea-4c36-abe6-8771929d1b0e",
   "metadata": {},
   "source": [
    "This means that we are not able to perform any supervised feature selection until we select features from `text_features_train`. We must reduce its dimension significantly. \n",
    "\n",
    "**In this section, we will discuss how we can reduce `text_features_train`'s dimension significantly, and what features should we select to make the most efficient and precise machine learning outcomes.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}