7. Data Preprocessing using PySpark (1).ipynb

{"cells":[{"cell_type":"markdown","source":["## Data Preprocessing using PySpark\n### Agenda\n<hr>\n* Understanding How DataFrames are stored internally\n* Why Preprocessing ?\n* Vectorizing Data\n* Scaling Techniques\n* Binning Techniques\n* Encoding Techniques\n* Text Processing\n* Dimensionality Reduction\n* Dealing with missing data using Imputer\n* Converting data to higher derivatives\n* Custom Transformer\n\n<hr>"],"metadata":{}},{"cell_type":"markdown","source":["### 1. Understanding How DataFrames are stored internally\n<hr>\n* Spark 1.0 to 1.3: It started with RDD’s where data is represented as Java Objects.\n* Spark 1.4 to 1.6: Deprioritised Java objects. DataSet and DataFrame evolved where data is stored in row-based format.\n* Spark 2.x: Support for Vectorized Parquet which is columnar in-memory data is added."],"metadata":{}},{"cell_type":"markdown","source":["<img src=\"https://github.com/awantik/machine-learning-slides/blob/master/ML-Pipeline.png?raw=true\">"],"metadata":{}},{"cell_type":"markdown","source":["### 2. Why Preprocessing \n<hr>\n* Learning algorithms have affinity towards certain pattern of data.\n* Unscaled or unstandardized data have might have unacceptable prediction\n* Learning algorithms understands only number, converting text image to number is required\n* Preprocessing refers to transformation before feeding data to learning models\n* PySpark wants training data to be vectorized. One column will be holding the feature data.\n* DataFrame is columner datastructure & thus, same column data will be in same partition. \n\n<hr>\n\n<img src=\"https://github.com/awantik/machine-learning-slides/blob/master/pipeline-ml.png?raw=true\" width=\"800px\">\n\n* Format for Machine Learning\n\n<img src=\"https://github.com/zekelabs/machine-learning-using-pyspark/blob/master/vec.PNG?raw=true\">"],"metadata":{}},{"cell_type":"markdown","source":["### 2. Vectorizing Data\n<hr>\n* Preprocessing transformers can be applied on columns of vectors.\n* Data of neumeric columns can be fed to VectorAssembler which can do the vectorization"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import VectorAssembler\ndataFrame = spark.createDataFrame(zip([1,2,3],[5,6,7],[7,8,9]),['A','B','C'])\ndisplay(dataFrame)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>7</td></tr><tr><td>2</td><td>6</td><td>8</td></tr><tr><td>3</td><td>7</td><td>9</td></tr></tbody></table></div>"]}}],"execution_count":6},{"cell_type":"code","source":["ve = VectorAssembler(inputCols=['A','B','C'], outputCol='vec_feature')\nout = ve.transform(dataFrame)\ndisplay(out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th><th>vec_feature</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>7</td><td>List(1, 3, List(), List(1.0, 5.0, 7.0))</td></tr><tr><td>2</td><td>6</td><td>8</td><td>List(1, 3, List(), List(2.0, 6.0, 8.0))</td></tr><tr><td>3</td><td>7</td><td>9</td><td>List(1, 3, List(), List(3.0, 7.0, 9.0))</td></tr></tbody></table></div>"]}}],"execution_count":7},{"cell_type":"markdown","source":["### 3. Scaling\n<hr> \n* A transformer for bringing data to same scale. Linear scling techniques like StandardScaler & MinMaxScaler\n* If data is normally distributed, StandardScaling works well. \n* Otherwise, MinMaxScaler comes very handy\n* StandardScaler - Subtract mean of column & div by standard deviation\n* MinMaxScaler - Subtract min of column & div by difference between max & min"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import StandardScaler, MinMaxScaler\n\nmm_scaler = MinMaxScaler(inputCol='vec_feature', outputCol='minmax_scaled')\nss_scaler = StandardScaler(inputCol='vec_feature', outputCol='standard_scaled', withMean=True, withStd=True)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":9},{"cell_type":"code","source":["mm = mm_scaler.fit(out)\nss = ss_scaler.fit(out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":10},{"cell_type":"code","source":["res = mm.transform(out)\nres = ss.transform(res)\ndisplay(res)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th><th>vec_feature</th><th>minmax_scaled</th><th>standard_scaled</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>7</td><td>List(1, 3, List(), List(1.0, 5.0, 7.0))</td><td>List(1, 3, List(), List(0.0, 0.0, 0.0))</td><td>List(1, 3, List(), List(-1.0, -1.0, -1.0))</td></tr><tr><td>2</td><td>6</td><td>8</td><td>List(1, 3, List(), List(2.0, 6.0, 8.0))</td><td>List(1, 3, List(), List(0.5, 0.5, 0.5))</td><td>List(1, 3, List(), List(0.0, 0.0, 0.0))</td></tr><tr><td>3</td><td>7</td><td>9</td><td>List(1, 3, List(), List(3.0, 7.0, 9.0))</td><td>List(1, 3, List(), List(1.0, 1.0, 1.0))</td><td>List(1, 3, List(), List(1.0, 1.0, 1.0))</td></tr></tbody></table></div>"]}}],"execution_count":11},{"cell_type":"markdown","source":["#### Normalization across vector\n* Transforms each vector to a unit norm.\n* Reduces complexity of model & make model more generalized.\n* Spark allows to configure p-norm\n\n<img src=\"https://d2vlcm61l7u1fs.cloudfront.net/media%2Fb14%2Fb140f0cd-b2e2-4f62-a3fe-31c1021f5ae3%2FphpIOG6p7.png\">"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import Normalizer"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":13},{"cell_type":"code","source":["normalizer = Normalizer(inputCol=\"vec_feature\", outputCol=\"normFeatures\", p=1.0)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":14},{"cell_type":"code","source":["norm_out = normalizer.transform(out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":15},{"cell_type":"code","source":["display(norm_out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th><th>vec_feature</th><th>normFeatures</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>7</td><td>List(1, 3, List(), List(1.0, 5.0, 7.0))</td><td>List(1, 3, List(), List(0.07692307692307693, 0.38461538461538464, 0.5384615384615384))</td></tr><tr><td>2</td><td>6</td><td>8</td><td>List(1, 3, List(), List(2.0, 6.0, 8.0))</td><td>List(1, 3, List(), List(0.125, 0.375, 0.5))</td></tr><tr><td>3</td><td>7</td><td>9</td><td>List(1, 3, List(), List(3.0, 7.0, 9.0))</td><td>List(1, 3, List(), List(0.15789473684210525, 0.3684210526315789, 0.47368421052631576))</td></tr></tbody></table></div>"]}}],"execution_count":16},{"cell_type":"markdown","source":["### 4. Binning Techniques\n<hr>\n* There are bunch of transformers for converting continues values to categorical.\n* There are many learning algorithms which expects categorical values & \n* Binerizer converts continues values into 0 or 1.\n* Bucketizer converts continues values into configued buckets.\n* QuantileDiscretizer converts continues values into configured number of buckets"],"metadata":{}},{"cell_type":"markdown","source":["#### Binarizer\n* Some learning algorithms like Bernoulli naive Bayes expects feature data to be in 0/1.\n* We need to transform data using BInarizer before feeding it to learning algorithm"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import Binarizer"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":19},{"cell_type":"code","source":["from pyspark.ml.feature import VectorAssembler\ndataFrame = spark.createDataFrame(zip([1,2,3],[5,6,7],[17,-3,89]),['A','B','C'])\ndisplay(dataFrame)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>17</td></tr><tr><td>2</td><td>6</td><td>-3</td></tr><tr><td>3</td><td>7</td><td>89</td></tr></tbody></table></div>"]}}],"execution_count":20},{"cell_type":"code","source":["bin = Binarizer(inputCol='C',outputCol='bin_C')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":21},{"cell_type":"code","source":["dataFrame = dataFrame.withColumn('C', dataFrame.C.cast('double'))\nbin_out = bin.transform(dataFrame)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":22},{"cell_type":"code","source":["display(bin_out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th><th>bin_C</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>17.0</td><td>1.0</td></tr><tr><td>2</td><td>6</td><td>-3.0</td><td>0.0</td></tr><tr><td>3</td><td>7</td><td>89.0</td><td>1.0</td></tr></tbody></table></div>"]}}],"execution_count":23},{"cell_type":"markdown","source":["##### Bucketizer\n* Used for converting Continues data into categorical values.\n* Learning algorithm like Multinomial Naive Bayes expects counts & this algorithm becomes handy"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import Bucketizer"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":25},{"cell_type":"code","source":["splits = [-float(\"inf\"), -20, -10, 0.0, 10, 20, float(\"inf\")]"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":26},{"cell_type":"code","source":["bucket = Bucketizer(splits=splits, inputCol='C',outputCol='bucket_out')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":27},{"cell_type":"code","source":["bucket_out = bucket.transform(dataFrame)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":28},{"cell_type":"code","source":["display(bucket_out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th><th>bucket_out</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>17.0</td><td>4.0</td></tr><tr><td>2</td><td>6</td><td>-3.0</td><td>2.0</td></tr><tr><td>3</td><td>7</td><td>89.0</td><td>5.0</td></tr></tbody></table></div>"]}}],"execution_count":29},{"cell_type":"markdown","source":["##### QuantileDiscretizer\n* Configuration involves number of buckets.\n* What goes in which bucket is decided by the algorithm rather than configuration"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import QuantileDiscretizer"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":31},{"cell_type":"code","source":["quantiledis = QuantileDiscretizer(numBuckets=2, inputCol='C', outputCol='quant_out')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":32},{"cell_type":"code","source":["quant = quantiledis.fit(dataFrame)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":33},{"cell_type":"code","source":["quant_out = quant.transform(dataFrame)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":34},{"cell_type":"code","source":["display(quant_out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th><th>quant_out</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>17.0</td><td>1.0</td></tr><tr><td>2</td><td>6</td><td>-3.0</td><td>0.0</td></tr><tr><td>3</td><td>7</td><td>89.0</td><td>1.0</td></tr></tbody></table></div>"]}}],"execution_count":35},{"cell_type":"markdown","source":["### 5. Encoding Techniques\n<hr>\n* Learning algorithms only understands number.\n* Categorical columns containing string needs to be mapped into numbers.\n* Nominal values should be one hot encoded.\n* Ordinal values should be assigned values as per order if expected by learning algorithm."],"metadata":{}},{"cell_type":"markdown","source":["##### StringIndexer\n* Converts string to numbers.\n* Numbers are assigned based either on frequency or alphabetical order, stringOrderType can be configured for this.\n* Handling unseen or invalid data can be controlled by handleInvalid\n  - throw an exception (which is the default)\n  - skip the row containing the unseen label entirely\n  - put unseen labels in a special additional bucket, at index numLabels"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import StringIndexer\nstring_indexer = StringIndexer(inputCol='category', outputCol='outcategory', stringOrderType=\"frequencyDesc\", handleInvalid=\"keep\")"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":38},{"cell_type":"code","source":["dataFrame = spark.createDataFrame(zip([1,2,22],[5,6,22],['orange','mango','orange']),['A','B','category'])"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":39},{"cell_type":"code","source":["display(dataFrame)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>category</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>orange</td></tr><tr><td>2</td><td>6</td><td>mango</td></tr><tr><td>22</td><td>22</td><td>orange</td></tr></tbody></table></div>"]}}],"execution_count":40},{"cell_type":"code","source":["string_indexer = string_indexer.fit(dataFrame)\nstring_indexer_out = string_indexer.transform(dataFrame)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":41},{"cell_type":"code","source":["display(string_indexer_out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>category</th><th>outcategory</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>orange</td><td>0.0</td></tr><tr><td>2</td><td>6</td><td>mango</td><td>1.0</td></tr><tr><td>22</td><td>22</td><td>orange</td><td>0.0</td></tr></tbody></table></div>"]}}],"execution_count":42},{"cell_type":"markdown","source":["##### OneHotEstimator\n* Nominal values are required to be converted to one hot vector.\n* OneHotEstimator is a transformer designed to do this.\n* String data first have to converted to number & then OneHotEstimator can work upon it."],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import OneHotEncoderEstimator"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":44},{"cell_type":"code","source":["ohe = OneHotEncoderEstimator(inputCols=['outcategory'], outputCols=['ohe_outcategory'])"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":45},{"cell_type":"code","source":["ohe = ohe.fit(string_indexer_out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":46},{"cell_type":"code","source":["ohe_dataframe = ohe.transform(string_indexer_out)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":47},{"cell_type":"code","source":["display(ohe_dataframe)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>category</th><th>outcategory</th><th>ohe_outcategory</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>orange</td><td>0.0</td><td>List(0, 1, List(0), List(1.0))</td></tr><tr><td>2</td><td>6</td><td>mango</td><td>1.0</td><td>List(0, 1, List(), List())</td></tr><tr><td>22</td><td>22</td><td>orange</td><td>0.0</td><td>List(0, 1, List(0), List(1.0))</td></tr></tbody></table></div>"]}}],"execution_count":48},{"cell_type":"markdown","source":["### 6. Text Processing\n<hr>\n* Again, machine learning models don't understand text.\n* We need to convert text into vectors.\n* First step is tokenization - Converting text to sequence of words\n* Followed by tokenization, minimization of word space happens.\n* Tokens with multiple words using NGrams\n* The vector representation can be word frequencies using CountVectorizer.\n* Also, vector representation can be importance of word using TFIDF\n* Using HashingVectorizer, words can be hashed into buckets."],"metadata":{}},{"cell_type":"markdown","source":["##### Tokenizer\n* Convert text into seq of words\n* Tokenizer splits' sentences bases on white spaces only.\n* For much more robust behaviour RegexTokenizer can be used."],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import Tokenizer"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":51},{"cell_type":"code","source":["sentenceData = spark.createDataFrame([\n    (1.0, \"The food is very tasty. I would recommand this to all my friends\"),\n    (0.0, \"I wish they could be more organized. Services seems very much laid back.\"),\n    (1.0, \"Loved the ambiance. Good place to be in weekdays.\")\n], [\"label\", \"sentence\"])"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":52},{"cell_type":"code","source":["tokenizer = Tokenizer(inputCol='sentence',outputCol='tokenized_sentence')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":53},{"cell_type":"code","source":["tokenized_sentence = tokenizer.transform(sentenceData)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":54},{"cell_type":"code","source":["display(tokenized_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>label</th><th>sentence</th><th>tokenized_sentence</th></tr></thead><tbody><tr><td>1.0</td><td>The food is very tasty. I would recommand this to all my friends</td><td>List(the, food, is, very, tasty., i, would, recommand, this, to, all, my, friends)</td></tr><tr><td>0.0</td><td>I wish they could be more organized. Services seems very much laid back.</td><td>List(i, wish, they, could, be, more, organized., services, seems, very, much, laid, back.)</td></tr><tr><td>1.0</td><td>Loved the ambiance. Good place to be in weekdays.</td><td>List(loved, the, ambiance., good, place, to, be, in, weekdays.)</td></tr></tbody></table></div>"]}}],"execution_count":55},{"cell_type":"markdown","source":["##### StopWordsRemover\n* Not all words are important.\n* StopWordsRemover transformer allows to drop words.\n* This technique also reduces dimensionality of data."],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import StopWordsRemover"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":57},{"cell_type":"code","source":["stwr = StopWordsRemover(inputCol='tokenized_sentence', outputCol='stopwordsremoved_sentence', stopWords=['the','is','i','in','to','be'], caseSensitive=False)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":58},{"cell_type":"code","source":["stopwords_removed_sentence = stwr.transform(tokenized_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":59},{"cell_type":"code","source":["display(stopwords_removed_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>label</th><th>sentence</th><th>tokenized_sentence</th><th>stopwordsremoved_sentence</th></tr></thead><tbody><tr><td>1.0</td><td>The food is very tasty. I would recommand this to all my friends</td><td>List(the, food, is, very, tasty., i, would, recommand, this, to, all, my, friends)</td><td>List(food, very, tasty., would, recommand, this, all, my, friends)</td></tr><tr><td>0.0</td><td>I wish they could be more organized. Services seems very much laid back.</td><td>List(i, wish, they, could, be, more, organized., services, seems, very, much, laid, back.)</td><td>List(wish, they, could, more, organized., services, seems, very, much, laid, back.)</td></tr><tr><td>1.0</td><td>Loved the ambiance. Good place to be in weekdays.</td><td>List(loved, the, ambiance., good, place, to, be, in, weekdays.)</td><td>List(loved, ambiance., good, place, weekdays.)</td></tr></tbody></table></div>"]}}],"execution_count":60},{"cell_type":"markdown","source":["##### NGrams\n* Sometimes multiple words together gives the meaning & not singluar words.\n* Based on configuration NGrams creates token of multiple words"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import NGram"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":62},{"cell_type":"code","source":["ngram = NGram(inputCol='stopwordsremoved_sentence', outputCol='ngram_sentence', n=2)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":63},{"cell_type":"code","source":["ngram_sentence = ngram.transform(stopwords_removed_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":64},{"cell_type":"code","source":["display(ngram_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>label</th><th>sentence</th><th>tokenized_sentence</th><th>stopwordsremoved_sentence</th><th>ngram_sentence</th></tr></thead><tbody><tr><td>1.0</td><td>The food is very tasty. I would recommand this to all my friends</td><td>List(the, food, is, very, tasty., i, would, recommand, this, to, all, my, friends)</td><td>List(food, very, tasty., would, recommand, this, all, my, friends)</td><td>List(food very, very tasty., tasty. would, would recommand, recommand this, this all, all my, my friends)</td></tr><tr><td>0.0</td><td>I wish they could be more organized. Services seems very much laid back.</td><td>List(i, wish, they, could, be, more, organized., services, seems, very, much, laid, back.)</td><td>List(wish, they, could, more, organized., services, seems, very, much, laid, back.)</td><td>List(wish they, they could, could more, more organized., organized. services, services seems, seems very, very much, much laid, laid back.)</td></tr><tr><td>1.0</td><td>Loved the ambiance. Good place to be in weekdays.</td><td>List(loved, the, ambiance., good, place, to, be, in, weekdays.)</td><td>List(loved, ambiance., good, place, weekdays.)</td><td>List(loved ambiance., ambiance. good, good place, place weekdays.)</td></tr></tbody></table></div>"]}}],"execution_count":65},{"cell_type":"markdown","source":["##### CountVectorizer\n* Assigns one word to each column.\n* Value of the column for a row is fequency of the word in the text"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import CountVectorizer"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":67},{"cell_type":"code","source":["cv = CountVectorizer(inputCol='stopwordsremoved_sentence', outputCol='countvectorizer_sentence')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":68},{"cell_type":"code","source":["cv = cv.fit(stopwords_removed_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":69},{"cell_type":"code","source":["cv_sentence = cv.transform(stopwords_removed_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":70},{"cell_type":"code","source":["display(cv_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>label</th><th>sentence</th><th>tokenized_sentence</th><th>stopwordsremoved_sentence</th><th>countvectorizer_sentence</th></tr></thead><tbody><tr><td>1.0</td><td>The food is very tasty. I would recommand this to all my friends</td><td>List(the, food, is, very, tasty., i, would, recommand, this, to, all, my, friends)</td><td>List(food, very, tasty., would, recommand, this, all, my, friends)</td><td>List(0, 24, List(0, 2, 4, 6, 7, 10, 11, 16, 20), List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td></tr><tr><td>0.0</td><td>I wish they could be more organized. Services seems very much laid back.</td><td>List(i, wish, they, could, be, more, organized., services, seems, very, much, laid, back.)</td><td>List(wish, they, could, more, organized., services, seems, very, much, laid, back.)</td><td>List(0, 24, List(0, 5, 8, 9, 12, 13, 14, 17, 18, 21, 22), List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td></tr><tr><td>1.0</td><td>Loved the ambiance. Good place to be in weekdays.</td><td>List(loved, the, ambiance., good, place, to, be, in, weekdays.)</td><td>List(loved, ambiance., good, place, weekdays.)</td><td>List(0, 24, List(1, 3, 15, 19, 23), List(1.0, 1.0, 1.0, 1.0, 1.0))</td></tr></tbody></table></div>"]}}],"execution_count":71},{"cell_type":"markdown","source":["##### TFIDF \n* Returns a sparse vector, where value corresponding to each column is in range of 0 & 1.\n* Importance of word is derived from how many times it appears in document verses entire corpus."],"metadata":{}},{"cell_type":"code","source":["sentenceData = spark.createDataFrame([\n    (1.0, \"The food is very tasty . I would recommand this to all my friends\"),\n    (0.0, \"I wish they could be more organized. Food is tasty. Services seems very much laid back.\"),\n    (1.0, \"Loved the ambiance. Good place to be in weekdays. Tasty Food.\")\n], [\"label\", \"sentence\"])"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":73},{"cell_type":"code","source":["tokenized_sentence = tokenizer.transform(sentenceData)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":74},{"cell_type":"code","source":["display(tokenized_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>label</th><th>sentence</th><th>tokenized_sentence</th></tr></thead><tbody><tr><td>1.0</td><td>The food is very tasty . I would recommand this to all my friends</td><td>List(the, food, is, very, tasty, ., i, would, recommand, this, to, all, my, friends)</td></tr><tr><td>0.0</td><td>I wish they could be more organized. Food is tasty. Services seems very much laid back.</td><td>List(i, wish, they, could, be, more, organized., food, is, tasty., services, seems, very, much, laid, back.)</td></tr><tr><td>1.0</td><td>Loved the ambiance. Good place to be in weekdays. Tasty Food.</td><td>List(loved, the, ambiance., good, place, to, be, in, weekdays., tasty, food.)</td></tr></tbody></table></div>"]}}],"execution_count":75},{"cell_type":"code","source":["from pyspark.ml.feature import HashingTF,IDF"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":76},{"cell_type":"code","source":["hashingtf = HashingTF(inputCol='tokenized_sentence', outputCol='hashed_sentence')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":77},{"cell_type":"code","source":["hashed_sentence = hashingtf.transform(tokenized_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":78},{"cell_type":"code","source":["display(hashed_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>label</th><th>sentence</th><th>tokenized_sentence</th><th>hashed_sentence</th></tr></thead><tbody><tr><td>1.0</td><td>The food is very tasty . I would recommand this to all my friends</td><td>List(the, food, is, very, tasty, ., i, would, recommand, this, to, all, my, friends)</td><td>List(0, 262144, List(1536, 15889, 24417, 37852, 68867, 103838, 108541, 121133, 135560, 194490, 205044, 210040, 222394, 227406), List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td></tr><tr><td>0.0</td><td>I wish they could be more organized. Food is tasty. Services seems very much laid back.</td><td>List(i, wish, they, could, be, more, organized., food, is, tasty., services, seems, very, much, laid, back.)</td><td>List(0, 262144, List(4333, 15889, 20719, 24417, 76764, 100620, 121133, 131149, 147765, 151536, 158129, 167152, 195807, 210040, 224471, 239029), List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td></tr><tr><td>1.0</td><td>Loved the ambiance. Good place to be in weekdays. Tasty Food.</td><td>List(loved, the, ambiance., good, place, to, be, in, weekdays., tasty, food.)</td><td>List(0, 262144, List(33933, 61231, 66865, 103838, 113432, 167152, 176674, 205044, 216731, 222453, 227406), List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td></tr></tbody></table></div>"]}}],"execution_count":79},{"cell_type":"code","source":["idf = IDF(inputCol='hashed_sentence', outputCol='idf_sentence')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":80},{"cell_type":"code","source":["idf = idf.fit(hashed_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":81},{"cell_type":"code","source":["idf_sentence = idf.transform(hashed_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":82},{"cell_type":"code","source":["display(idf_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>label</th><th>sentence</th><th>tokenized_sentence</th><th>hashed_sentence</th><th>idf_sentence</th></tr></thead><tbody><tr><td>1.0</td><td>The food is very tasty . I would recommand this to all my friends</td><td>List(the, food, is, very, tasty, ., i, would, recommand, this, to, all, my, friends)</td><td>List(0, 262144, List(1536, 15889, 24417, 37852, 68867, 103838, 108541, 121133, 135560, 194490, 205044, 210040, 222394, 227406), List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>List(0, 262144, List(1536, 15889, 24417, 37852, 68867, 103838, 108541, 121133, 135560, 194490, 205044, 210040, 222394, 227406), List(0.6931471805599453, 0.28768207245178085, 0.28768207245178085, 0.6931471805599453, 0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.6931471805599453, 0.28768207245178085, 0.28768207245178085, 0.6931471805599453, 0.28768207245178085))</td></tr><tr><td>0.0</td><td>I wish they could be more organized. Food is tasty. Services seems very much laid back.</td><td>List(i, wish, they, could, be, more, organized., food, is, tasty., services, seems, very, much, laid, back.)</td><td>List(0, 262144, List(4333, 15889, 20719, 24417, 76764, 100620, 121133, 131149, 147765, 151536, 158129, 167152, 195807, 210040, 224471, 239029), List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>List(0, 262144, List(4333, 15889, 20719, 24417, 76764, 100620, 121133, 131149, 147765, 151536, 158129, 167152, 195807, 210040, 224471, 239029), List(0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.6931471805599453, 0.6931471805599453, 0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.6931471805599453))</td></tr><tr><td>1.0</td><td>Loved the ambiance. Good place to be in weekdays. Tasty Food.</td><td>List(loved, the, ambiance., good, place, to, be, in, weekdays., tasty, food.)</td><td>List(0, 262144, List(33933, 61231, 66865, 103838, 113432, 167152, 176674, 205044, 216731, 222453, 227406), List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>List(0, 262144, List(33933, 61231, 66865, 103838, 113432, 167152, 176674, 205044, 216731, 222453, 227406), List(0.6931471805599453, 0.6931471805599453, 0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.28768207245178085, 0.6931471805599453, 0.6931471805599453, 0.28768207245178085))</td></tr></tbody></table></div>"]}}],"execution_count":83},{"cell_type":"markdown","source":["##### FeatureHasher\n* Projects a set of categorical or numerical values into a feature vector of fixed dimension.\n* FeatureHasher works on multiple columns\n* Columns can be neumeric columns, string columns or binary columns."],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import FeatureHasher\n\ndataset = spark.createDataFrame([\n    (2.2, True, \"1\", \"foo\"),\n    (3.3, False, \"2\", \"bar\"),\n    (4.4, False, \"3\", \"baz\"),\n    (5.5, False, \"4\", \"foo\")\n], [\"real\", \"bool\", \"stringNum\", \"string\"])\n\nhasher = FeatureHasher(inputCols=[\"real\", \"bool\", \"stringNum\", \"string\"],\n                       outputCol=\"features\")\n\nfeaturized = hasher.transform(dataset)\nfeaturized.show(truncate=False)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\">+----+-----+---------+------+--------------------------------------------------------+\nreal|bool |stringNum|string|features                                                |\n+----+-----+---------+------+--------------------------------------------------------+\n2.2 |true |1        |foo   |(262144,[174475,247670,257907,262126],[2.2,1.0,1.0,1.0])|\n3.3 |false|2        |bar   |(262144,[70644,89673,173866,174475],[1.0,1.0,1.0,3.3])  |\n4.4 |false|3        |baz   |(262144,[22406,70644,174475,187923],[1.0,1.0,4.4,1.0])  |\n5.5 |false|4        |foo   |(262144,[70644,101499,174475,257907],[1.0,1.0,5.5,1.0]) |\n+----+-----+---------+------+--------------------------------------------------------+\n\n</div>"]}}],"execution_count":85},{"cell_type":"markdown","source":["##### Word2Vec\n* Returns a dense representation of text.\n* Vector size is configurable\n* An efficeint way to reduce vectorspace & reduce computation."],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import Word2Vec"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":87},{"cell_type":"code","source":["wc = Word2Vec(vectorSize=3, minCount=0, inputCol='stopwordsremoved_sentence', outputCol='word2vec_sentence')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":88},{"cell_type":"code","source":["wc = wc.fit(stopwords_removed_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":89},{"cell_type":"code","source":["word2vec_stentence = wc.transform(stopwords_removed_sentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":90},{"cell_type":"code","source":["display(word2vec_stentence)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>label</th><th>sentence</th><th>tokenized_sentence</th><th>stopwordsremoved_sentence</th><th>word2vec_sentence</th></tr></thead><tbody><tr><td>1.0</td><td>The food is very tasty. I would recommand this to all my friends</td><td>List(the, food, is, very, tasty., i, would, recommand, this, to, all, my, friends)</td><td>List(food, very, tasty., would, recommand, this, all, my, friends)</td><td>List(1, 3, List(), List(0.001430635547472371, 0.028169848904427554, 0.06262729689478874))</td></tr><tr><td>0.0</td><td>I wish they could be more organized. Services seems very much laid back.</td><td>List(i, wish, they, could, be, more, organized., services, seems, very, much, laid, back.)</td><td>List(wish, they, could, more, organized., services, seems, very, much, laid, back.)</td><td>List(1, 3, List(), List(0.021398486019196836, -0.024643760555508463, 0.02174479878422889))</td></tr><tr><td>1.0</td><td>Loved the ambiance. Good place to be in weekdays.</td><td>List(loved, the, ambiance., good, place, to, be, in, weekdays.)</td><td>List(loved, ambiance., good, place, weekdays.)</td><td>List(1, 3, List(), List(-0.01969942003488541, -0.020766012882813812, -0.022866672463715078))</td></tr></tbody></table></div>"]}}],"execution_count":91},{"cell_type":"markdown","source":["### 7. Dimensionality Reduction\n<hr>\n* Reduce noise (unimportant columns) is an important step in getting data ready for machine learning.\n* We can use feature selecting techniques as well as feature deriving techniques like pca etc."],"metadata":{}},{"cell_type":"markdown","source":["##### VectorSlicer\n* This takes a feature vector and outputs a new feature vector with a subarray of the original features.\n* We can pass the columns of our interest & those columns will be retained."],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import VectorSlicer"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":94},{"cell_type":"code","source":["dataFrame = spark.createDataFrame(zip([1,2,3],[5,6,7],[17,-3,89]),['A','B','C'])\nva = VectorAssembler(inputCols=['A','B','C'], outputCol='vec_sentence')\nvec_sentences = va.transform(dataFrame)\ndisplay(vec_sentences)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th><th>vec_sentence</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>17</td><td>List(1, 3, List(), List(1.0, 5.0, 17.0))</td></tr><tr><td>2</td><td>6</td><td>-3</td><td>List(1, 3, List(), List(2.0, 6.0, -3.0))</td></tr><tr><td>3</td><td>7</td><td>89</td><td>List(1, 3, List(), List(3.0, 7.0, 89.0))</td></tr></tbody></table></div>"]}}],"execution_count":95},{"cell_type":"code","source":["vec_slicer = VectorSlicer(inputCol='vec_sentence', outputCol='vec_sliced_sentence',indices=[1])"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":96},{"cell_type":"code","source":["vec_slice_sentences = vec_slicer.transform(vec_sentences)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":97},{"cell_type":"code","source":["display(vec_slice_sentences)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th><th>C</th><th>vec_sentence</th><th>vec_sliced_sentence</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>17</td><td>List(1, 3, List(), List(1.0, 5.0, 17.0))</td><td>List(1, 1, List(), List(5.0))</td></tr><tr><td>2</td><td>6</td><td>-3</td><td>List(1, 3, List(), List(2.0, 6.0, -3.0))</td><td>List(1, 1, List(), List(6.0))</td></tr><tr><td>3</td><td>7</td><td>89</td><td>List(1, 3, List(), List(3.0, 7.0, 89.0))</td><td>List(1, 1, List(), List(7.0))</td></tr></tbody></table></div>"]}}],"execution_count":98},{"cell_type":"markdown","source":["##### ChiSeqSelector\n* Transformer for selecting catagorical features for selecting catagorical label"],"metadata":{}},{"cell_type":"code","source":["income_data = spark.read.csv('/FileStore/tables/adult.csv',header=True, inferSchema=True)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":100},{"cell_type":"code","source":["cat_cols = map(lambda x:x[0] ,filter(lambda x:x[1] == 'string', income_data.dtypes))\ncat_cols"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"><span class=\"ansired\">Out[</span><span class=\"ansired\">160</span><span class=\"ansired\">]: </span>\n[&apos;workclass&apos;,\n &apos;education&apos;,\n &apos;marital-status&apos;,\n &apos;occupation&apos;,\n &apos;relationship&apos;,\n &apos;race&apos;,\n &apos;gender&apos;,\n &apos;native-country&apos;,\n &apos;income&apos;]\n</div>"]}}],"execution_count":101},{"cell_type":"code","source":["from pyspark.ml.feature import ChiSqSelector"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":102},{"cell_type":"code","source":["for col in cat_cols:\n  st = StringIndexer(inputCol=col, outputCol=col+'tf').fit(income_data)\n  income_data = st.transform(income_data)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":103},{"cell_type":"code","source":["display(income_data.head(5))"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>age</th><th>workclass</th><th>fnlwgt</th><th>education</th><th>educational-num</th><th>marital-status</th><th>occupation</th><th>relationship</th><th>race</th><th>gender</th><th>capital-gain</th><th>capital-loss</th><th>hours-per-week</th><th>native-country</th><th>income</th><th>workclasstf</th><th>educationtf</th><th>marital-statustf</th><th>occupationtf</th><th>relationshiptf</th><th>racetf</th><th>gendertf</th><th>native-countrytf</th><th>incometf</th></tr></thead><tbody><tr><td>25</td><td>Private</td><td>226802</td><td>11th</td><td>7</td><td>Never-married</td><td>Machine-op-inspct</td><td>Own-child</td><td>Black</td><td>Male</td><td>0</td><td>0</td><td>40</td><td>United-States</td><td><=50K</td><td>0.0</td><td>5.0</td><td>1.0</td><td>6.0</td><td>2.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>38</td><td>Private</td><td>89814</td><td>HS-grad</td><td>9</td><td>Married-civ-spouse</td><td>Farming-fishing</td><td>Husband</td><td>White</td><td>Male</td><td>0</td><td>0</td><td>50</td><td>United-States</td><td><=50K</td><td>0.0</td><td>0.0</td><td>0.0</td><td>10.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>28</td><td>Local-gov</td><td>336951</td><td>Assoc-acdm</td><td>12</td><td>Married-civ-spouse</td><td>Protective-serv</td><td>Husband</td><td>White</td><td>Male</td><td>0</td><td>0</td><td>40</td><td>United-States</td><td>>50K</td><td>2.0</td><td>6.0</td><td>0.0</td><td>12.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>1.0</td></tr><tr><td>44</td><td>Private</td><td>160323</td><td>Some-college</td><td>10</td><td>Married-civ-spouse</td><td>Machine-op-inspct</td><td>Husband</td><td>Black</td><td>Male</td><td>7688</td><td>0</td><td>40</td><td>United-States</td><td>>50K</td><td>0.0</td><td>1.0</td><td>0.0</td><td>6.0</td><td>0.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>1.0</td></tr><tr><td>18</td><td>?</td><td>103497</td><td>Some-college</td><td>10</td><td>Never-married</td><td>?</td><td>Own-child</td><td>White</td><td>Female</td><td>0</td><td>0</td><td>30</td><td>United-States</td><td><=50K</td><td>3.0</td><td>1.0</td><td>1.0</td><td>7.0</td><td>2.0</td><td>0.0</td><td>1.0</td><td>0.0</td><td>0.0</td></tr></tbody></table></div>"]}}],"execution_count":104},{"cell_type":"code","source":["vec_cols = map(lambda x:x+'tf',cat_cols)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":105},{"cell_type":"code","source":["vec_cols"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"><span class=\"ansired\">Out[</span><span class=\"ansired\">180</span><span class=\"ansired\">]: </span>\n[&apos;workclasstf&apos;,\n &apos;educationtf&apos;,\n &apos;marital-statustf&apos;,\n &apos;occupationtf&apos;,\n &apos;relationshiptf&apos;,\n &apos;racetf&apos;,\n &apos;gendertf&apos;,\n &apos;native-countrytf&apos;,\n &apos;incometf&apos;]\n</div>"]}}],"execution_count":106},{"cell_type":"code","source":["vec_cols.remove('incometf')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":107},{"cell_type":"code","source":["vec_cols"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"><span class=\"ansired\">Out[</span><span class=\"ansired\">182</span><span class=\"ansired\">]: </span>\n[&apos;workclasstf&apos;,\n &apos;educationtf&apos;,\n &apos;marital-statustf&apos;,\n &apos;occupationtf&apos;,\n &apos;relationshiptf&apos;,\n &apos;racetf&apos;,\n &apos;gendertf&apos;,\n &apos;native-countrytf&apos;]\n</div>"]}}],"execution_count":108},{"cell_type":"code","source":["ve = VectorAssembler(inputCols=vec_cols, outputCol='vec_out')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":109},{"cell_type":"code","source":["income_data = ve.transform(income_data)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":110},{"cell_type":"code","source":["chi2 = ChiSqSelector(numTopFeatures=4, featuresCol='vec_out', outputCol='chi2_features', labelCol='incometf')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":111},{"cell_type":"code","source":["chi2 = chi2.fit(income_data)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":112},{"cell_type":"code","source":["chi2_data = chi2.transform(income_data)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":113},{"cell_type":"code","source":["display(chi2_data.head(5))"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>age</th><th>workclass</th><th>fnlwgt</th><th>education</th><th>educational-num</th><th>marital-status</th><th>occupation</th><th>relationship</th><th>race</th><th>gender</th><th>capital-gain</th><th>capital-loss</th><th>hours-per-week</th><th>native-country</th><th>income</th><th>workclasstf</th><th>educationtf</th><th>marital-statustf</th><th>occupationtf</th><th>relationshiptf</th><th>racetf</th><th>gendertf</th><th>native-countrytf</th><th>incometf</th><th>vec_out</th><th>chi2_features</th></tr></thead><tbody><tr><td>25</td><td>Private</td><td>226802</td><td>11th</td><td>7</td><td>Never-married</td><td>Machine-op-inspct</td><td>Own-child</td><td>Black</td><td>Male</td><td>0</td><td>0</td><td>40</td><td>United-States</td><td><=50K</td><td>0.0</td><td>5.0</td><td>1.0</td><td>6.0</td><td>2.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>List(1, 8, List(), List(0.0, 5.0, 1.0, 6.0, 2.0, 1.0, 0.0, 0.0))</td><td>List(1, 4, List(), List(0.0, 5.0, 1.0, 6.0))</td></tr><tr><td>38</td><td>Private</td><td>89814</td><td>HS-grad</td><td>9</td><td>Married-civ-spouse</td><td>Farming-fishing</td><td>Husband</td><td>White</td><td>Male</td><td>0</td><td>0</td><td>50</td><td>United-States</td><td><=50K</td><td>0.0</td><td>0.0</td><td>0.0</td><td>10.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>List(0, 8, List(3), List(10.0))</td><td>List(0, 4, List(3), List(10.0))</td></tr><tr><td>28</td><td>Local-gov</td><td>336951</td><td>Assoc-acdm</td><td>12</td><td>Married-civ-spouse</td><td>Protective-serv</td><td>Husband</td><td>White</td><td>Male</td><td>0</td><td>0</td><td>40</td><td>United-States</td><td>>50K</td><td>2.0</td><td>6.0</td><td>0.0</td><td>12.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>1.0</td><td>List(0, 8, List(0, 1, 3), List(2.0, 6.0, 12.0))</td><td>List(0, 4, List(0, 1, 3), List(2.0, 6.0, 12.0))</td></tr><tr><td>44</td><td>Private</td><td>160323</td><td>Some-college</td><td>10</td><td>Married-civ-spouse</td><td>Machine-op-inspct</td><td>Husband</td><td>Black</td><td>Male</td><td>7688</td><td>0</td><td>40</td><td>United-States</td><td>>50K</td><td>0.0</td><td>1.0</td><td>0.0</td><td>6.0</td><td>0.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>1.0</td><td>List(0, 8, List(1, 3, 5), List(1.0, 6.0, 1.0))</td><td>List(0, 4, List(1, 3), List(1.0, 6.0))</td></tr><tr><td>18</td><td>?</td><td>103497</td><td>Some-college</td><td>10</td><td>Never-married</td><td>?</td><td>Own-child</td><td>White</td><td>Female</td><td>0</td><td>0</td><td>30</td><td>United-States</td><td><=50K</td><td>3.0</td><td>1.0</td><td>1.0</td><td>7.0</td><td>2.0</td><td>0.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>List(1, 8, List(), List(3.0, 1.0, 1.0, 7.0, 2.0, 0.0, 1.0, 0.0))</td><td>List(1, 4, List(), List(3.0, 1.0, 1.0, 7.0))</td></tr></tbody></table></div>"]}}],"execution_count":114},{"cell_type":"markdown","source":["##### PCA\n* PCA is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. \n* A PCA class trains a model to project vectors to a low-dimensional space using PCA.\n* Data should be standardized before subjecting to PCA"],"metadata":{}},{"cell_type":"code","source":["from sklearn.datasets import load_iris\nimport pandas as pd"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":116},{"cell_type":"code","source":["iris_data = load_iris()"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":117},{"cell_type":"code","source":["df = pd.DataFrame(iris_data.data)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":118},{"cell_type":"code","source":["df.columns = ['PL','PW','SL','SW']"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":119},{"cell_type":"code","source":["dataFrame = spark.createDataFrame(df)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":120},{"cell_type":"code","source":["ve = VectorAssembler(inputCols=list(df.columns), outputCol='vec_data')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":121},{"cell_type":"code","source":["ve_data = ve.transform(dataFrame)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":122},{"cell_type":"code","source":["display(ve_data)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>PL</th><th>PW</th><th>SL</th><th>SW</th><th>vec_data</th></tr></thead><tbody><tr><td>5.1</td><td>3.5</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.1, 3.5, 1.4, 0.2))</td></tr><tr><td>4.9</td><td>3.0</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(4.9, 3.0, 1.4, 0.2))</td></tr><tr><td>4.7</td><td>3.2</td><td>1.3</td><td>0.2</td><td>List(1, 4, List(), List(4.7, 3.2, 1.3, 0.2))</td></tr><tr><td>4.6</td><td>3.1</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(4.6, 3.1, 1.5, 0.2))</td></tr><tr><td>5.0</td><td>3.6</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.6, 1.4, 0.2))</td></tr><tr><td>5.4</td><td>3.9</td><td>1.7</td><td>0.4</td><td>List(1, 4, List(), List(5.4, 3.9, 1.7, 0.4))</td></tr><tr><td>4.6</td><td>3.4</td><td>1.4</td><td>0.3</td><td>List(1, 4, List(), List(4.6, 3.4, 1.4, 0.3))</td></tr><tr><td>5.0</td><td>3.4</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.4, 1.5, 0.2))</td></tr><tr><td>4.4</td><td>2.9</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(4.4, 2.9, 1.4, 0.2))</td></tr><tr><td>4.9</td><td>3.1</td><td>1.5</td><td>0.1</td><td>List(1, 4, List(), List(4.9, 3.1, 1.5, 0.1))</td></tr><tr><td>5.4</td><td>3.7</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.4, 3.7, 1.5, 0.2))</td></tr><tr><td>4.8</td><td>3.4</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(4.8, 3.4, 1.6, 0.2))</td></tr><tr><td>4.8</td><td>3.0</td><td>1.4</td><td>0.1</td><td>List(1, 4, List(), List(4.8, 3.0, 1.4, 0.1))</td></tr><tr><td>4.3</td><td>3.0</td><td>1.1</td><td>0.1</td><td>List(1, 4, List(), List(4.3, 3.0, 1.1, 0.1))</td></tr><tr><td>5.8</td><td>4.0</td><td>1.2</td><td>0.2</td><td>List(1, 4, List(), List(5.8, 4.0, 1.2, 0.2))</td></tr><tr><td>5.7</td><td>4.4</td><td>1.5</td><td>0.4</td><td>List(1, 4, List(), List(5.7, 4.4, 1.5, 0.4))</td></tr><tr><td>5.4</td><td>3.9</td><td>1.3</td><td>0.4</td><td>List(1, 4, List(), List(5.4, 3.9, 1.3, 0.4))</td></tr><tr><td>5.1</td><td>3.5</td><td>1.4</td><td>0.3</td><td>List(1, 4, List(), List(5.1, 3.5, 1.4, 0.3))</td></tr><tr><td>5.7</td><td>3.8</td><td>1.7</td><td>0.3</td><td>List(1, 4, List(), List(5.7, 3.8, 1.7, 0.3))</td></tr><tr><td>5.1</td><td>3.8</td><td>1.5</td><td>0.3</td><td>List(1, 4, List(), List(5.1, 3.8, 1.5, 0.3))</td></tr><tr><td>5.4</td><td>3.4</td><td>1.7</td><td>0.2</td><td>List(1, 4, List(), List(5.4, 3.4, 1.7, 0.2))</td></tr><tr><td>5.1</td><td>3.7</td><td>1.5</td><td>0.4</td><td>List(1, 4, List(), List(5.1, 3.7, 1.5, 0.4))</td></tr><tr><td>4.6</td><td>3.6</td><td>1.0</td><td>0.2</td><td>List(1, 4, List(), List(4.6, 3.6, 1.0, 0.2))</td></tr><tr><td>5.1</td><td>3.3</td><td>1.7</td><td>0.5</td><td>List(1, 4, List(), List(5.1, 3.3, 1.7, 0.5))</td></tr><tr><td>4.8</td><td>3.4</td><td>1.9</td><td>0.2</td><td>List(1, 4, List(), List(4.8, 3.4, 1.9, 0.2))</td></tr><tr><td>5.0</td><td>3.0</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.0, 1.6, 0.2))</td></tr><tr><td>5.0</td><td>3.4</td><td>1.6</td><td>0.4</td><td>List(1, 4, List(), List(5.0, 3.4, 1.6, 0.4))</td></tr><tr><td>5.2</td><td>3.5</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.2, 3.5, 1.5, 0.2))</td></tr><tr><td>5.2</td><td>3.4</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.2, 3.4, 1.4, 0.2))</td></tr><tr><td>4.7</td><td>3.2</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(4.7, 3.2, 1.6, 0.2))</td></tr><tr><td>4.8</td><td>3.1</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(4.8, 3.1, 1.6, 0.2))</td></tr><tr><td>5.4</td><td>3.4</td><td>1.5</td><td>0.4</td><td>List(1, 4, List(), List(5.4, 3.4, 1.5, 0.4))</td></tr><tr><td>5.2</td><td>4.1</td><td>1.5</td><td>0.1</td><td>List(1, 4, List(), List(5.2, 4.1, 1.5, 0.1))</td></tr><tr><td>5.5</td><td>4.2</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.5, 4.2, 1.4, 0.2))</td></tr><tr><td>4.9</td><td>3.1</td><td>1.5</td><td>0.1</td><td>List(1, 4, List(), List(4.9, 3.1, 1.5, 0.1))</td></tr><tr><td>5.0</td><td>3.2</td><td>1.2</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.2, 1.2, 0.2))</td></tr><tr><td>5.5</td><td>3.5</td><td>1.3</td><td>0.2</td><td>List(1, 4, List(), List(5.5, 3.5, 1.3, 0.2))</td></tr><tr><td>4.9</td><td>3.1</td><td>1.5</td><td>0.1</td><td>List(1, 4, List(), List(4.9, 3.1, 1.5, 0.1))</td></tr><tr><td>4.4</td><td>3.0</td><td>1.3</td><td>0.2</td><td>List(1, 4, List(), List(4.4, 3.0, 1.3, 0.2))</td></tr><tr><td>5.1</td><td>3.4</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.1, 3.4, 1.5, 0.2))</td></tr><tr><td>5.0</td><td>3.5</td><td>1.3</td><td>0.3</td><td>List(1, 4, List(), List(5.0, 3.5, 1.3, 0.3))</td></tr><tr><td>4.5</td><td>2.3</td><td>1.3</td><td>0.3</td><td>List(1, 4, List(), List(4.5, 2.3, 1.3, 0.3))</td></tr><tr><td>4.4</td><td>3.2</td><td>1.3</td><td>0.2</td><td>List(1, 4, List(), List(4.4, 3.2, 1.3, 0.2))</td></tr><tr><td>5.0</td><td>3.5</td><td>1.6</td><td>0.6</td><td>List(1, 4, List(), List(5.0, 3.5, 1.6, 0.6))</td></tr><tr><td>5.1</td><td>3.8</td><td>1.9</td><td>0.4</td><td>List(1, 4, List(), List(5.1, 3.8, 1.9, 0.4))</td></tr><tr><td>4.8</td><td>3.0</td><td>1.4</td><td>0.3</td><td>List(1, 4, List(), List(4.8, 3.0, 1.4, 0.3))</td></tr><tr><td>5.1</td><td>3.8</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(5.1, 3.8, 1.6, 0.2))</td></tr><tr><td>4.6</td><td>3.2</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(4.6, 3.2, 1.4, 0.2))</td></tr><tr><td>5.3</td><td>3.7</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.3, 3.7, 1.5, 0.2))</td></tr><tr><td>5.0</td><td>3.3</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.3, 1.4, 0.2))</td></tr><tr><td>7.0</td><td>3.2</td><td>4.7</td><td>1.4</td><td>List(1, 4, List(), List(7.0, 3.2, 4.7, 1.4))</td></tr><tr><td>6.4</td><td>3.2</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(6.4, 3.2, 4.5, 1.5))</td></tr><tr><td>6.9</td><td>3.1</td><td>4.9</td><td>1.5</td><td>List(1, 4, List(), List(6.9, 3.1, 4.9, 1.5))</td></tr><tr><td>5.5</td><td>2.3</td><td>4.0</td><td>1.3</td><td>List(1, 4, List(), List(5.5, 2.3, 4.0, 1.3))</td></tr><tr><td>6.5</td><td>2.8</td><td>4.6</td><td>1.5</td><td>List(1, 4, List(), List(6.5, 2.8, 4.6, 1.5))</td></tr><tr><td>5.7</td><td>2.8</td><td>4.5</td><td>1.3</td><td>List(1, 4, List(), List(5.7, 2.8, 4.5, 1.3))</td></tr><tr><td>6.3</td><td>3.3</td><td>4.7</td><td>1.6</td><td>List(1, 4, List(), List(6.3, 3.3, 4.7, 1.6))</td></tr><tr><td>4.9</td><td>2.4</td><td>3.3</td><td>1.0</td><td>List(1, 4, List(), List(4.9, 2.4, 3.3, 1.0))</td></tr><tr><td>6.6</td><td>2.9</td><td>4.6</td><td>1.3</td><td>List(1, 4, List(), List(6.6, 2.9, 4.6, 1.3))</td></tr><tr><td>5.2</td><td>2.7</td><td>3.9</td><td>1.4</td><td>List(1, 4, List(), List(5.2, 2.7, 3.9, 1.4))</td></tr><tr><td>5.0</td><td>2.0</td><td>3.5</td><td>1.0</td><td>List(1, 4, List(), List(5.0, 2.0, 3.5, 1.0))</td></tr><tr><td>5.9</td><td>3.0</td><td>4.2</td><td>1.5</td><td>List(1, 4, List(), List(5.9, 3.0, 4.2, 1.5))</td></tr><tr><td>6.0</td><td>2.2</td><td>4.0</td><td>1.0</td><td>List(1, 4, List(), List(6.0, 2.2, 4.0, 1.0))</td></tr><tr><td>6.1</td><td>2.9</td><td>4.7</td><td>1.4</td><td>List(1, 4, List(), List(6.1, 2.9, 4.7, 1.4))</td></tr><tr><td>5.6</td><td>2.9</td><td>3.6</td><td>1.3</td><td>List(1, 4, List(), List(5.6, 2.9, 3.6, 1.3))</td></tr><tr><td>6.7</td><td>3.1</td><td>4.4</td><td>1.4</td><td>List(1, 4, List(), List(6.7, 3.1, 4.4, 1.4))</td></tr><tr><td>5.6</td><td>3.0</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(5.6, 3.0, 4.5, 1.5))</td></tr><tr><td>5.8</td><td>2.7</td><td>4.1</td><td>1.0</td><td>List(1, 4, List(), List(5.8, 2.7, 4.1, 1.0))</td></tr><tr><td>6.2</td><td>2.2</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(6.2, 2.2, 4.5, 1.5))</td></tr><tr><td>5.6</td><td>2.5</td><td>3.9</td><td>1.1</td><td>List(1, 4, List(), List(5.6, 2.5, 3.9, 1.1))</td></tr><tr><td>5.9</td><td>3.2</td><td>4.8</td><td>1.8</td><td>List(1, 4, List(), List(5.9, 3.2, 4.8, 1.8))</td></tr><tr><td>6.1</td><td>2.8</td><td>4.0</td><td>1.3</td><td>List(1, 4, List(), List(6.1, 2.8, 4.0, 1.3))</td></tr><tr><td>6.3</td><td>2.5</td><td>4.9</td><td>1.5</td><td>List(1, 4, List(), List(6.3, 2.5, 4.9, 1.5))</td></tr><tr><td>6.1</td><td>2.8</td><td>4.7</td><td>1.2</td><td>List(1, 4, List(), List(6.1, 2.8, 4.7, 1.2))</td></tr><tr><td>6.4</td><td>2.9</td><td>4.3</td><td>1.3</td><td>List(1, 4, List(), List(6.4, 2.9, 4.3, 1.3))</td></tr><tr><td>6.6</td><td>3.0</td><td>4.4</td><td>1.4</td><td>List(1, 4, List(), List(6.6, 3.0, 4.4, 1.4))</td></tr><tr><td>6.8</td><td>2.8</td><td>4.8</td><td>1.4</td><td>List(1, 4, List(), List(6.8, 2.8, 4.8, 1.4))</td></tr><tr><td>6.7</td><td>3.0</td><td>5.0</td><td>1.7</td><td>List(1, 4, List(), List(6.7, 3.0, 5.0, 1.7))</td></tr><tr><td>6.0</td><td>2.9</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(6.0, 2.9, 4.5, 1.5))</td></tr><tr><td>5.7</td><td>2.6</td><td>3.5</td><td>1.0</td><td>List(1, 4, List(), List(5.7, 2.6, 3.5, 1.0))</td></tr><tr><td>5.5</td><td>2.4</td><td>3.8</td><td>1.1</td><td>List(1, 4, List(), List(5.5, 2.4, 3.8, 1.1))</td></tr><tr><td>5.5</td><td>2.4</td><td>3.7</td><td>1.0</td><td>List(1, 4, List(), List(5.5, 2.4, 3.7, 1.0))</td></tr><tr><td>5.8</td><td>2.7</td><td>3.9</td><td>1.2</td><td>List(1, 4, List(), List(5.8, 2.7, 3.9, 1.2))</td></tr><tr><td>6.0</td><td>2.7</td><td>5.1</td><td>1.6</td><td>List(1, 4, List(), List(6.0, 2.7, 5.1, 1.6))</td></tr><tr><td>5.4</td><td>3.0</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(5.4, 3.0, 4.5, 1.5))</td></tr><tr><td>6.0</td><td>3.4</td><td>4.5</td><td>1.6</td><td>List(1, 4, List(), List(6.0, 3.4, 4.5, 1.6))</td></tr><tr><td>6.7</td><td>3.1</td><td>4.7</td><td>1.5</td><td>List(1, 4, List(), List(6.7, 3.1, 4.7, 1.5))</td></tr><tr><td>6.3</td><td>2.3</td><td>4.4</td><td>1.3</td><td>List(1, 4, List(), List(6.3, 2.3, 4.4, 1.3))</td></tr><tr><td>5.6</td><td>3.0</td><td>4.1</td><td>1.3</td><td>List(1, 4, List(), List(5.6, 3.0, 4.1, 1.3))</td></tr><tr><td>5.5</td><td>2.5</td><td>4.0</td><td>1.3</td><td>List(1, 4, List(), List(5.5, 2.5, 4.0, 1.3))</td></tr><tr><td>5.5</td><td>2.6</td><td>4.4</td><td>1.2</td><td>List(1, 4, List(), List(5.5, 2.6, 4.4, 1.2))</td></tr><tr><td>6.1</td><td>3.0</td><td>4.6</td><td>1.4</td><td>List(1, 4, List(), List(6.1, 3.0, 4.6, 1.4))</td></tr><tr><td>5.8</td><td>2.6</td><td>4.0</td><td>1.2</td><td>List(1, 4, List(), List(5.8, 2.6, 4.0, 1.2))</td></tr><tr><td>5.0</td><td>2.3</td><td>3.3</td><td>1.0</td><td>List(1, 4, List(), List(5.0, 2.3, 3.3, 1.0))</td></tr><tr><td>5.6</td><td>2.7</td><td>4.2</td><td>1.3</td><td>List(1, 4, List(), List(5.6, 2.7, 4.2, 1.3))</td></tr><tr><td>5.7</td><td>3.0</td><td>4.2</td><td>1.2</td><td>List(1, 4, List(), List(5.7, 3.0, 4.2, 1.2))</td></tr><tr><td>5.7</td><td>2.9</td><td>4.2</td><td>1.3</td><td>List(1, 4, List(), List(5.7, 2.9, 4.2, 1.3))</td></tr><tr><td>6.2</td><td>2.9</td><td>4.3</td><td>1.3</td><td>List(1, 4, List(), List(6.2, 2.9, 4.3, 1.3))</td></tr><tr><td>5.1</td><td>2.5</td><td>3.0</td><td>1.1</td><td>List(1, 4, List(), List(5.1, 2.5, 3.0, 1.1))</td></tr><tr><td>5.7</td><td>2.8</td><td>4.1</td><td>1.3</td><td>List(1, 4, List(), List(5.7, 2.8, 4.1, 1.3))</td></tr><tr><td>6.3</td><td>3.3</td><td>6.0</td><td>2.5</td><td>List(1, 4, List(), List(6.3, 3.3, 6.0, 2.5))</td></tr><tr><td>5.8</td><td>2.7</td><td>5.1</td><td>1.9</td><td>List(1, 4, List(), List(5.8, 2.7, 5.1, 1.9))</td></tr><tr><td>7.1</td><td>3.0</td><td>5.9</td><td>2.1</td><td>List(1, 4, List(), List(7.1, 3.0, 5.9, 2.1))</td></tr><tr><td>6.3</td><td>2.9</td><td>5.6</td><td>1.8</td><td>List(1, 4, List(), List(6.3, 2.9, 5.6, 1.8))</td></tr><tr><td>6.5</td><td>3.0</td><td>5.8</td><td>2.2</td><td>List(1, 4, List(), List(6.5, 3.0, 5.8, 2.2))</td></tr><tr><td>7.6</td><td>3.0</td><td>6.6</td><td>2.1</td><td>List(1, 4, List(), List(7.6, 3.0, 6.6, 2.1))</td></tr><tr><td>4.9</td><td>2.5</td><td>4.5</td><td>1.7</td><td>List(1, 4, List(), List(4.9, 2.5, 4.5, 1.7))</td></tr><tr><td>7.3</td><td>2.9</td><td>6.3</td><td>1.8</td><td>List(1, 4, List(), List(7.3, 2.9, 6.3, 1.8))</td></tr><tr><td>6.7</td><td>2.5</td><td>5.8</td><td>1.8</td><td>List(1, 4, List(), List(6.7, 2.5, 5.8, 1.8))</td></tr><tr><td>7.2</td><td>3.6</td><td>6.1</td><td>2.5</td><td>List(1, 4, List(), List(7.2, 3.6, 6.1, 2.5))</td></tr><tr><td>6.5</td><td>3.2</td><td>5.1</td><td>2.0</td><td>List(1, 4, List(), List(6.5, 3.2, 5.1, 2.0))</td></tr><tr><td>6.4</td><td>2.7</td><td>5.3</td><td>1.9</td><td>List(1, 4, List(), List(6.4, 2.7, 5.3, 1.9))</td></tr><tr><td>6.8</td><td>3.0</td><td>5.5</td><td>2.1</td><td>List(1, 4, List(), List(6.8, 3.0, 5.5, 2.1))</td></tr><tr><td>5.7</td><td>2.5</td><td>5.0</td><td>2.0</td><td>List(1, 4, List(), List(5.7, 2.5, 5.0, 2.0))</td></tr><tr><td>5.8</td><td>2.8</td><td>5.1</td><td>2.4</td><td>List(1, 4, List(), List(5.8, 2.8, 5.1, 2.4))</td></tr><tr><td>6.4</td><td>3.2</td><td>5.3</td><td>2.3</td><td>List(1, 4, List(), List(6.4, 3.2, 5.3, 2.3))</td></tr><tr><td>6.5</td><td>3.0</td><td>5.5</td><td>1.8</td><td>List(1, 4, List(), List(6.5, 3.0, 5.5, 1.8))</td></tr><tr><td>7.7</td><td>3.8</td><td>6.7</td><td>2.2</td><td>List(1, 4, List(), List(7.7, 3.8, 6.7, 2.2))</td></tr><tr><td>7.7</td><td>2.6</td><td>6.9</td><td>2.3</td><td>List(1, 4, List(), List(7.7, 2.6, 6.9, 2.3))</td></tr><tr><td>6.0</td><td>2.2</td><td>5.0</td><td>1.5</td><td>List(1, 4, List(), List(6.0, 2.2, 5.0, 1.5))</td></tr><tr><td>6.9</td><td>3.2</td><td>5.7</td><td>2.3</td><td>List(1, 4, List(), List(6.9, 3.2, 5.7, 2.3))</td></tr><tr><td>5.6</td><td>2.8</td><td>4.9</td><td>2.0</td><td>List(1, 4, List(), List(5.6, 2.8, 4.9, 2.0))</td></tr><tr><td>7.7</td><td>2.8</td><td>6.7</td><td>2.0</td><td>List(1, 4, List(), List(7.7, 2.8, 6.7, 2.0))</td></tr><tr><td>6.3</td><td>2.7</td><td>4.9</td><td>1.8</td><td>List(1, 4, List(), List(6.3, 2.7, 4.9, 1.8))</td></tr><tr><td>6.7</td><td>3.3</td><td>5.7</td><td>2.1</td><td>List(1, 4, List(), List(6.7, 3.3, 5.7, 2.1))</td></tr><tr><td>7.2</td><td>3.2</td><td>6.0</td><td>1.8</td><td>List(1, 4, List(), List(7.2, 3.2, 6.0, 1.8))</td></tr><tr><td>6.2</td><td>2.8</td><td>4.8</td><td>1.8</td><td>List(1, 4, List(), List(6.2, 2.8, 4.8, 1.8))</td></tr><tr><td>6.1</td><td>3.0</td><td>4.9</td><td>1.8</td><td>List(1, 4, List(), List(6.1, 3.0, 4.9, 1.8))</td></tr><tr><td>6.4</td><td>2.8</td><td>5.6</td><td>2.1</td><td>List(1, 4, List(), List(6.4, 2.8, 5.6, 2.1))</td></tr><tr><td>7.2</td><td>3.0</td><td>5.8</td><td>1.6</td><td>List(1, 4, List(), List(7.2, 3.0, 5.8, 1.6))</td></tr><tr><td>7.4</td><td>2.8</td><td>6.1</td><td>1.9</td><td>List(1, 4, List(), List(7.4, 2.8, 6.1, 1.9))</td></tr><tr><td>7.9</td><td>3.8</td><td>6.4</td><td>2.0</td><td>List(1, 4, List(), List(7.9, 3.8, 6.4, 2.0))</td></tr><tr><td>6.4</td><td>2.8</td><td>5.6</td><td>2.2</td><td>List(1, 4, List(), List(6.4, 2.8, 5.6, 2.2))</td></tr><tr><td>6.3</td><td>2.8</td><td>5.1</td><td>1.5</td><td>List(1, 4, List(), List(6.3, 2.8, 5.1, 1.5))</td></tr><tr><td>6.1</td><td>2.6</td><td>5.6</td><td>1.4</td><td>List(1, 4, List(), List(6.1, 2.6, 5.6, 1.4))</td></tr><tr><td>7.7</td><td>3.0</td><td>6.1</td><td>2.3</td><td>List(1, 4, List(), List(7.7, 3.0, 6.1, 2.3))</td></tr><tr><td>6.3</td><td>3.4</td><td>5.6</td><td>2.4</td><td>List(1, 4, List(), List(6.3, 3.4, 5.6, 2.4))</td></tr><tr><td>6.4</td><td>3.1</td><td>5.5</td><td>1.8</td><td>List(1, 4, List(), List(6.4, 3.1, 5.5, 1.8))</td></tr><tr><td>6.0</td><td>3.0</td><td>4.8</td><td>1.8</td><td>List(1, 4, List(), List(6.0, 3.0, 4.8, 1.8))</td></tr><tr><td>6.9</td><td>3.1</td><td>5.4</td><td>2.1</td><td>List(1, 4, List(), List(6.9, 3.1, 5.4, 2.1))</td></tr><tr><td>6.7</td><td>3.1</td><td>5.6</td><td>2.4</td><td>List(1, 4, List(), List(6.7, 3.1, 5.6, 2.4))</td></tr><tr><td>6.9</td><td>3.1</td><td>5.1</td><td>2.3</td><td>List(1, 4, List(), List(6.9, 3.1, 5.1, 2.3))</td></tr><tr><td>5.8</td><td>2.7</td><td>5.1</td><td>1.9</td><td>List(1, 4, List(), List(5.8, 2.7, 5.1, 1.9))</td></tr><tr><td>6.8</td><td>3.2</td><td>5.9</td><td>2.3</td><td>List(1, 4, List(), List(6.8, 3.2, 5.9, 2.3))</td></tr><tr><td>6.7</td><td>3.3</td><td>5.7</td><td>2.5</td><td>List(1, 4, List(), List(6.7, 3.3, 5.7, 2.5))</td></tr><tr><td>6.7</td><td>3.0</td><td>5.2</td><td>2.3</td><td>List(1, 4, List(), List(6.7, 3.0, 5.2, 2.3))</td></tr><tr><td>6.3</td><td>2.5</td><td>5.0</td><td>1.9</td><td>List(1, 4, List(), List(6.3, 2.5, 5.0, 1.9))</td></tr><tr><td>6.5</td><td>3.0</td><td>5.2</td><td>2.0</td><td>List(1, 4, List(), List(6.5, 3.0, 5.2, 2.0))</td></tr><tr><td>6.2</td><td>3.4</td><td>5.4</td><td>2.3</td><td>List(1, 4, List(), List(6.2, 3.4, 5.4, 2.3))</td></tr><tr><td>5.9</td><td>3.0</td><td>5.1</td><td>1.8</td><td>List(1, 4, List(), List(5.9, 3.0, 5.1, 1.8))</td></tr></tbody></table></div>"]}}],"execution_count":123},{"cell_type":"code","source":["from pyspark.ml.feature import PCA"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":124},{"cell_type":"code","source":["pca = PCA(k=3, inputCol='vec_data', outputCol='pca_vec_data')"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":125},{"cell_type":"code","source":["pca = pca.fit(ve_data)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":126},{"cell_type":"code","source":["pca_data = pca.transform(ve_data)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":127},{"cell_type":"code","source":["display(pca_data)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>PL</th><th>PW</th><th>SL</th><th>SW</th><th>vec_data</th><th>pca_vec_data</th></tr></thead><tbody><tr><td>5.1</td><td>3.5</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.1, 3.5, 1.4, 0.2))</td><td>List(1, 3, List(), List(-2.827135972679023, -5.641331045573361, 0.664276931510384))</td></tr><tr><td>4.9</td><td>3.0</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(4.9, 3.0, 1.4, 0.2))</td><td>List(1, 3, List(), List(-2.7959524821488406, -5.145166883252949, 0.8462865195139235))</td></tr><tr><td>4.7</td><td>3.2</td><td>1.3</td><td>0.2</td><td>List(1, 4, List(), List(4.7, 3.2, 1.3, 0.2))</td><td>List(1, 3, List(), List(-2.6215235581650553, -5.177378121203946, 0.6180558535094693))</td></tr><tr><td>4.6</td><td>3.1</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(4.6, 3.1, 1.5, 0.2))</td><td>List(1, 3, List(), List(-2.764905900474237, -5.00359941505698, 0.6050931192231175))</td></tr><tr><td>5.0</td><td>3.6</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.6, 1.4, 0.2))</td><td>List(1, 3, List(), List(-2.7827501159516568, -5.648648294377426, 0.546535394733807))</td></tr><tr><td>5.4</td><td>3.9</td><td>1.7</td><td>0.4</td><td>List(1, 4, List(), List(5.4, 3.9, 1.7, 0.4))</td><td>List(1, 3, List(), List(-3.2314457367733738, -6.062506444034105, 0.46843947549201426))</td></tr><tr><td>4.6</td><td>3.4</td><td>1.4</td><td>0.3</td><td>List(1, 4, List(), List(4.6, 3.4, 1.4, 0.3))</td><td>List(1, 3, List(), List(-2.690452415602341, -5.23261921978429, 0.3785140093177235))</td></tr><tr><td>5.0</td><td>3.4</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.4, 1.5, 0.2))</td><td>List(1, 3, List(), List(-2.884861104459153, -5.485129079769262, 0.6585666047727341))</td></tr><tr><td>4.4</td><td>2.9</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(4.4, 2.9, 1.4, 0.2))</td><td>List(1, 3, List(), List(-2.6233845324473375, -4.74392570447738, 0.6154296883939168))</td></tr><tr><td>4.9</td><td>3.1</td><td>1.5</td><td>0.1</td><td>List(1, 4, List(), List(4.9, 3.1, 1.5, 0.1))</td><td>List(1, 3, List(), List(-2.8374984110638506, -5.208032027056235, 0.8342983942440643))</td></tr><tr><td>5.4</td><td>3.7</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.4, 3.7, 1.5, 0.2))</td><td>List(1, 3, List(), List(-3.004816308444068, -5.9666587444815455, 0.7120400903223496))</td></tr><tr><td>4.8</td><td>3.4</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(4.8, 3.4, 1.6, 0.2))</td><td>List(1, 3, List(), List(-2.8982003795119153, -5.336244362769229, 0.5351147412585058))</td></tr><tr><td>4.8</td><td>3.0</td><td>1.4</td><td>0.1</td><td>List(1, 4, List(), List(4.8, 3.0, 1.4, 0.1))</td><td>List(1, 3, List(), List(-2.7239091217858737, -5.086983541937867, 0.8430928826038135))</td></tr><tr><td>4.3</td><td>3.0</td><td>1.1</td><td>0.1</td><td>List(1, 4, List(), List(4.3, 3.0, 1.1, 0.1))</td><td>List(1, 3, List(), List(-2.2861426515079906, -4.811443821323547, 0.5743514653360942))</td></tr><tr><td>5.8</td><td>4.0</td><td>1.2</td><td>0.2</td><td>List(1, 4, List(), List(5.8, 4.0, 1.2, 0.2))</td><td>List(1, 3, List(), List(-2.8677998808418246, -6.500918630222424, 0.7872707985180654))</td></tr><tr><td>5.7</td><td>4.4</td><td>1.5</td><td>0.4</td><td>List(1, 4, List(), List(5.7, 4.4, 1.5, 0.4))</td><td>List(1, 3, List(), List(-3.127473773983596, -6.659478075368834, 0.35903443056863926))</td></tr><tr><td>5.4</td><td>3.9</td><td>1.3</td><td>0.4</td><td>List(1, 4, List(), List(5.4, 3.9, 1.3, 0.4))</td><td>List(1, 3, List(), List(-2.8888168946571624, -6.132813405405569, 0.4974491056868149))</td></tr><tr><td>5.1</td><td>3.5</td><td>1.4</td><td>0.3</td><td>List(1, 4, List(), List(5.1, 3.5, 1.4, 0.3))</td><td>List(1, 3, List(), List(-2.863020365303844, -5.633860398559857, 0.6093708404377298))</td></tr><tr><td>5.7</td><td>3.8</td><td>1.7</td><td>0.3</td><td>List(1, 4, List(), List(5.7, 3.8, 1.7, 0.3))</td><td>List(1, 3, List(), List(-3.3122651363522086, -6.193967818900711, 0.7572865593067735))</td></tr><tr><td>5.1</td><td>3.8</td><td>1.5</td><td>0.3</td><td>List(1, 4, List(), List(5.1, 3.8, 1.5, 0.3))</td><td>List(1, 3, List(), List(-2.923996908865233, -5.835197369614939, 0.42319300650758995))</td></tr><tr><td>5.4</td><td>3.4</td><td>1.7</td><td>0.2</td><td>List(1, 4, List(), List(5.4, 3.4, 1.7, 0.2))</td><td>List(1, 3, List(), List(-3.200811396469838, -5.712591552397866, 0.8764607016063897))</td></tr><tr><td>5.1</td><td>3.7</td><td>1.5</td><td>0.4</td><td>List(1, 4, List(), List(5.1, 3.7, 1.5, 0.4))</td><td>List(1, 3, List(), List(-2.968108190479276, -5.754755485468787, 0.42792872422874906))</td></tr><tr><td>4.6</td><td>3.6</td><td>1.0</td><td>0.2</td><td>List(1, 4, List(), List(4.6, 3.6, 1.0, 0.2))</td><td>List(1, 3, List(), List(-2.2954854028828655, -5.456339302434554, 0.34314611299755127))</td></tr><tr><td>5.1</td><td>3.3</td><td>1.7</td><td>0.5</td><td>List(1, 4, List(), List(5.1, 3.3, 1.7, 0.5))</td><td>List(1, 3, List(), List(-3.208214560119089, -5.420246409238956, 0.5970850532339482))</td></tr><tr><td>4.8</td><td>3.4</td><td>1.9</td><td>0.2</td><td>List(1, 4, List(), List(4.8, 3.4, 1.9, 0.2))</td><td>List(1, 3, List(), List(-3.155172011099074, -5.283514141740631, 0.5133575186124055))</td></tr><tr><td>5.0</td><td>3.0</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.0, 1.6, 0.2))</td><td>List(1, 3, List(), List(-3.003425870945091, -5.175667390895801, 0.889881432399287))</td></tr><tr><td>5.0</td><td>3.4</td><td>1.6</td><td>0.4</td><td>List(1, 4, List(), List(5.0, 3.4, 1.6, 0.4))</td><td>List(1, 3, List(), List(-3.0422871002378487, -5.452611045399391, 0.5415020150787253))</td></tr><tr><td>5.2</td><td>3.5</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.2, 3.5, 1.5, 0.2))</td><td>List(1, 3, List(), List(-2.948952150946221, -5.6894082935590795, 0.7151242519444483))</td></tr><tr><td>5.2</td><td>3.4</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.2, 3.4, 1.4, 0.2))</td><td>List(1, 3, List(), List(-2.8715218294063893, -5.634013796769295, 0.782018468286962))</td></tr><tr><td>4.7</td><td>3.2</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(4.7, 3.2, 1.6, 0.2))</td><td>List(1, 3, List(), List(-2.8784951897522135, -5.124647900175347, 0.5962986308633687))</td></tr><tr><td>4.8</td><td>3.1</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(4.8, 3.1, 1.6, 0.2))</td><td>List(1, 3, List(), List(-2.9228810464795796, -5.117330651371282, 0.7140401676399455))</td></tr><tr><td>5.4</td><td>3.4</td><td>1.5</td><td>0.4</td><td>List(1, 4, List(), List(5.4, 3.4, 1.5, 0.4))</td><td>List(1, 3, List(), List(-3.101265760661376, -5.7328037390565925, 0.7811533345584816))</td></tr><tr><td>5.2</td><td>4.1</td><td>1.5</td><td>0.1</td><td>List(1, 4, List(), List(5.2, 4.1, 1.5, 0.1))</td><td>List(1, 3, List(), List(-2.863706424386071, -6.134706363368475, 0.4121794902542226))</td></tr><tr><td>5.5</td><td>4.2</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.5, 4.2, 1.4, 0.2))</td><td>List(1, 3, List(), List(-2.914183620707053, -6.414745658816239, 0.47918318188474673))</td></tr><tr><td>4.9</td><td>3.1</td><td>1.5</td><td>0.1</td><td>List(1, 4, List(), List(4.9, 3.1, 1.5, 0.1))</td><td>List(1, 3, List(), List(-2.8374984110638506, -5.208032027056235, 0.8342983942440643))</td></tr><tr><td>5.0</td><td>3.2</td><td>1.2</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.2, 1.2, 0.2))</td><td>List(1, 3, List(), List(-2.6443432508504374, -5.391916826532562, 0.799607445006461))</td></tr><tr><td>5.5</td><td>3.5</td><td>1.3</td><td>0.2</td><td>List(1, 4, List(), List(5.5, 3.5, 1.3, 0.2))</td><td>List(1, 3, List(), List(-2.88611463310255, -5.921523739230563, 0.9039282509901403))</td></tr><tr><td>4.9</td><td>3.1</td><td>1.5</td><td>0.1</td><td>List(1, 4, List(), List(4.9, 3.1, 1.5, 0.1))</td><td>List(1, 3, List(), List(-2.8374984110638506, -5.208032027056235, 0.8342983942440643))</td></tr><tr><td>4.4</td><td>3.0</td><td>1.3</td><td>0.2</td><td>List(1, 4, List(), List(4.4, 3.0, 1.3, 0.2))</td><td>List(1, 3, List(), List(-2.529500432929063, -4.834473681952896, 0.5630402871488038))</td></tr><tr><td>5.1</td><td>3.4</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.1, 3.4, 1.5, 0.2))</td><td>List(1, 3, List(), List(-2.9210200721972974, -5.5507830680978465, 0.7166663327554978))</td></tr><tr><td>5.0</td><td>3.5</td><td>1.3</td><td>0.3</td><td>List(1, 4, List(), List(5.0, 3.5, 1.3, 0.3))</td><td>List(1, 3, List(), List(-2.741204187036647, -5.585783150574141, 0.5585235200036665))</td></tr><tr><td>4.5</td><td>2.3</td><td>1.3</td><td>0.3</td><td>List(1, 4, List(), List(4.5, 2.3, 1.3, 0.3))</td><td>List(1, 3, List(), List(-2.6591320162165797, -4.3818583633394335, 0.9837265856156068))</td></tr><tr><td>4.4</td><td>3.2</td><td>1.3</td><td>0.2</td><td>List(1, 4, List(), List(4.4, 3.2, 1.3, 0.2))</td><td>List(1, 3, List(), List(-2.51304665495062, -4.980416156218194, 0.4437566695611772))</td></tr><tr><td>5.0</td><td>3.5</td><td>1.6</td><td>0.6</td><td>List(1, 4, List(), List(5.0, 3.5, 1.6, 0.6))</td><td>List(1, 3, List(), List(-3.1058289964982704, -5.510640988505035, 0.37204802413960314))</td></tr><tr><td>5.1</td><td>3.8</td><td>1.9</td><td>0.4</td><td>List(1, 4, List(), List(5.1, 3.8, 1.9, 0.4))</td><td>List(1, 3, List(), List(-3.3025101436062654, -5.757419761229972, 0.3392772852401351))</td></tr><tr><td>4.8</td><td>3.0</td><td>1.4</td><td>0.3</td><td>List(1, 4, List(), List(4.8, 3.0, 1.4, 0.3))</td><td>List(1, 3, List(), List(-2.7956779070355173, -5.072042247910861, 0.7332807004585051))</td></tr><tr><td>5.1</td><td>3.8</td><td>1.6</td><td>0.2</td><td>List(1, 4, List(), List(5.1, 3.8, 1.6, 0.2))</td><td>List(1, 3, List(), List(-2.973769726769464, -5.825091276285576, 0.4708466900315438))</td></tr><tr><td>4.6</td><td>3.2</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(4.6, 3.2, 1.4, 0.2))</td><td>List(1, 3, List(), List(-2.6710218009559625, -5.094147392532495, 0.5527037179780043))</td></tr><tr><td>5.3</td><td>3.7</td><td>1.5</td><td>0.2</td><td>List(1, 4, List(), List(5.3, 3.7, 1.5, 0.2))</td><td>List(1, 3, List(), List(-2.968657340705923, -5.901004756152961, 0.6539403623395854))</td></tr><tr><td>5.0</td><td>3.3</td><td>1.4</td><td>0.2</td><td>List(1, 4, List(), List(5.0, 3.3, 1.4, 0.2))</td><td>List(1, 3, List(), List(-2.8074307829193215, -5.42973458297948, 0.7254608211152473))</td></tr><tr><td>7.0</td><td>3.2</td><td>4.7</td><td>1.4</td><td>List(1, 4, List(), List(7.0, 3.2, 4.7, 1.4))</td><td>List(1, 3, List(), List(-6.796137685628042, -6.000162916941898, 1.048894647585385))</td></tr><tr><td>6.4</td><td>3.2</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(6.4, 3.2, 4.5, 1.5))</td><td>List(1, 3, List(), List(-6.443753850765889, -5.633921820642623, 0.6598950037135474))</td></tr><tr><td>6.9</td><td>3.1</td><td>4.9</td><td>1.5</td><td>List(1, 4, List(), List(6.9, 3.1, 4.9, 1.5))</td><td>List(1, 3, List(), List(-6.975404420562047, -5.818913563781431, 0.9810258222263797))</td></tr><tr><td>5.5</td><td>2.3</td><td>4.0</td><td>1.3</td><td>List(1, 4, List(), List(5.5, 2.3, 4.0, 1.3))</td><td>List(1, 3, List(), List(-5.69231030413067, -4.489119787232864, 0.8198479509018006))</td></tr><tr><td>6.5</td><td>2.8</td><td>4.6</td><td>1.5</td><td>List(1, 4, List(), List(6.5, 2.8, 4.6, 1.5))</td><td>List(1, 3, List(), List(-6.598477584989972, -5.390114120097746, 0.9493095593228644))</td></tr><tr><td>5.7</td><td>2.8</td><td>4.5</td><td>1.3</td><td>List(1, 4, List(), List(5.7, 2.8, 4.5, 1.3))</td><td>List(1, 3, List(), List(-6.151779847306116, -4.897400247838947, 0.601576325154761))</td></tr><tr><td>6.3</td><td>3.3</td><td>4.7</td><td>1.6</td><td>List(1, 4, List(), List(6.3, 3.3, 4.7, 1.6))</td><td>List(1, 3, List(), List(-6.60656680772145, -5.598614941747455, 0.4727425607669158))</td></tr><tr><td>4.9</td><td>2.4</td><td>3.3</td><td>1.0</td><td>List(1, 4, List(), List(4.9, 2.4, 3.3, 1.0))</td><td>List(1, 3, List(), List(-4.759875957134745, -4.31361621783458, 0.6270929002702674))</td></tr><tr><td>6.6</td><td>2.9</td><td>4.6</td><td>1.3</td><td>List(1, 4, List(), List(6.6, 2.9, 4.6, 1.3))</td><td>List(1, 3, List(), List(-6.554640878489251, -5.543680639585984, 1.0575796606571228))</td></tr><tr><td>5.2</td><td>2.7</td><td>3.9</td><td>1.4</td><td>List(1, 4, List(), List(5.2, 2.7, 3.9, 1.4))</td><td>List(1, 3, List(), List(-5.501153027055118, -4.594148864107071, 0.359327848254301))</td></tr><tr><td>5.0</td><td>2.0</td><td>3.5</td><td>1.0</td><td>List(1, 4, List(), List(5.0, 2.0, 3.5, 1.0))</td><td>List(1, 3, List(), List(-5.000256901887881, -4.052231776946836, 0.9092550483308841))</td></tr><tr><td>5.9</td><td>3.0</td><td>4.2</td><td>1.5</td><td>List(1, 4, List(), List(5.9, 3.0, 4.2, 1.5))</td><td>List(1, 3, List(), List(-6.0224411584664495, -5.212439625763004, 0.5104372040334546))</td></tr><tr><td>6.0</td><td>2.2</td><td>4.0</td><td>1.0</td><td>List(1, 4, List(), List(6.0, 2.2, 4.0, 1.0))</td><td>List(1, 3, List(), List(-5.7736788539361505, -4.766830432783642, 1.334706672827396))</td></tr><tr><td>6.1</td><td>2.9</td><td>4.7</td><td>1.4</td><td>List(1, 4, List(), List(6.1, 2.9, 4.7, 1.4))</td><td>List(1, 3, List(), List(-6.4953876429524025, -5.1903633105866955, 0.7049225221219495))</td></tr><tr><td>5.6</td><td>2.9</td><td>3.6</td><td>1.3</td><td>List(1, 4, List(), List(5.6, 2.9, 3.6, 1.3))</td><td>List(1, 3, List(), List(-5.336479095817276, -5.062908159728805, 0.5491064563164847))</td></tr><tr><td>6.7</td><td>3.1</td><td>4.4</td><td>1.4</td><td>List(1, 4, List(), List(6.7, 3.1, 4.4, 1.4))</td><td>List(1, 3, List(), List(-6.438916039815671, -5.7829599358520944, 0.9559944950770065))</td></tr><tr><td>5.6</td><td>3.0</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(5.6, 3.0, 4.5, 1.5))</td><td>List(1, 3, List(), List(-6.170935886839173, -4.962747439748655, 0.3143807974390622))</td></tr><tr><td>5.8</td><td>2.7</td><td>4.1</td><td>1.0</td><td>List(1, 4, List(), List(5.8, 2.7, 4.1, 1.0))</td><td>List(1, 3, List(), List(-5.7458836840428065, -4.982801901446854, 0.9130457653441013))</td></tr><tr><td>6.2</td><td>2.2</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(6.2, 2.2, 4.5, 1.5))</td><td>List(1, 3, List(), List(-6.453704805181813, -4.772901472658967, 1.140113635686153))</td></tr><tr><td>5.6</td><td>2.5</td><td>3.9</td><td>1.1</td><td>List(1, 4, List(), List(5.6, 2.5, 3.9, 1.1))</td><td>List(1, 3, List(), List(-5.554589498111676, -4.733234284196616, 0.8757286509909461))</td></tr><tr><td>5.9</td><td>3.2</td><td>4.8</td><td>1.8</td><td>List(1, 4, List(), List(5.9, 3.2, 4.8, 1.8))</td><td>List(1, 3, List(), List(-6.627583821536787, -5.230509716930598, 0.18292086793566398))</td></tr><tr><td>6.1</td><td>2.8</td><td>4.0</td><td>1.3</td><td>List(1, 4, List(), List(6.1, 2.8, 4.0, 1.3))</td><td>List(1, 3, List(), List(-5.8681296656134325, -5.24789990286761, 0.8702372748293175))</td></tr><tr><td>6.3</td><td>2.5</td><td>4.9</td><td>1.5</td><td>List(1, 4, List(), List(6.3, 2.5, 4.9, 1.5))</td><td>List(1, 3, List(), List(-6.8078119480685055, -4.9871622110140335, 0.9902783070926762))</td></tr><tr><td>6.1</td><td>2.8</td><td>4.7</td><td>1.2</td><td>List(1, 4, List(), List(6.1, 2.8, 4.7, 1.2))</td><td>List(1, 3, List(), List(-6.431845746691981, -5.132333367481053, 0.8743765130610712))</td></tr><tr><td>6.4</td><td>2.9</td><td>4.3</td><td>1.3</td><td>List(1, 4, List(), List(6.4, 2.9, 4.3, 1.3))</td><td>List(1, 3, List(), List(-6.225351311425803, -5.465102883957414, 0.963137427337696))</td></tr><tr><td>6.6</td><td>3.0</td><td>4.4</td><td>1.4</td><td>List(1, 4, List(), List(6.6, 3.0, 4.4, 1.4))</td><td>List(1, 3, List(), List(-6.410983961066747, -5.644334710390861, 0.9575365758880561))</td></tr><tr><td>6.8</td><td>2.8</td><td>4.8</td><td>1.4</td><td>List(1, 4, List(), List(6.8, 2.8, 4.8, 1.4))</td><td>List(1, 3, List(), List(-6.842384516637691, -5.559393251411268, 1.16401001924641))</td></tr><tr><td>6.7</td><td>3.0</td><td>5.0</td><td>1.7</td><td>List(1, 4, List(), List(6.7, 3.0, 5.0, 1.7))</td><td>List(1, 3, List(), List(-7.068739369853673, -5.582116315621742, 0.8074035853606567))</td></tr><tr><td>6.0</td><td>2.9</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(6.0, 2.9, 4.5, 1.5))</td><td>List(1, 3, List(), List(-6.323798646780973, -5.152392155930341, 0.6064215181639314))</td></tr><tr><td>5.7</td><td>2.6</td><td>3.5</td><td>1.0</td><td>List(1, 4, List(), List(5.7, 2.6, 3.5, 1.0))</td><td>List(1, 3, List(), List(-5.204008342119566, -4.9496371180428165, 0.9581022914473514))</td></tr><tr><td>5.5</td><td>2.4</td><td>3.8</td><td>1.1</td><td>List(1, 4, List(), List(5.5, 2.4, 3.8, 1.1))</td><td>List(1, 3, List(), List(-5.4410002088336995, -4.612185799078251, 0.884523139350696))</td></tr><tr><td>5.5</td><td>2.4</td><td>3.7</td><td>1.0</td><td>List(1, 4, List(), List(5.5, 2.4, 3.7, 1.0))</td><td>List(1, 3, List(), List(-5.319458605679825, -4.637233186434619, 0.9466816379720502))</td></tr><tr><td>5.8</td><td>2.7</td><td>3.9</td><td>1.2</td><td>List(1, 4, List(), List(5.8, 2.7, 3.9, 1.2))</td><td>List(1, 3, List(), List(-5.6463380482343455, -5.00301408810558, 0.8177383982961928))</td></tr><tr><td>6.0</td><td>2.7</td><td>5.1</td><td>1.6</td><td>List(1, 4, List(), List(6.0, 2.7, 5.1, 1.6))</td><td>List(1, 3, List(), List(-6.890080080558554, -4.893518592594345, 0.6272845993867024))</td></tr><tr><td>5.4</td><td>3.0</td><td>4.5</td><td>1.5</td><td>List(1, 4, List(), List(5.4, 3.0, 4.5, 1.5))</td><td>List(1, 3, List(), List(-6.098617951362883, -4.831439463091488, 0.19818134147353472))</td></tr><tr><td>6.0</td><td>3.4</td><td>4.5</td><td>1.6</td><td>List(1, 4, List(), List(6.0, 3.4, 4.5, 1.6))</td><td>List(1, 3, List(), List(-6.318548594459687, -5.509777694580084, 0.25330638312221065))</td></tr><tr><td>6.7</td><td>3.1</td><td>4.7</td><td>1.5</td><td>List(1, 4, List(), List(6.7, 3.1, 4.7, 1.5))</td><td>List(1, 3, List(), List(-6.731772064027651, -5.722759067809994, 0.8793311813582521))</td></tr><tr><td>6.3</td><td>2.3</td><td>4.4</td><td>1.3</td><td>List(1, 4, List(), List(6.3, 2.3, 4.4, 1.3))</td><td>List(1, 3, List(), List(-6.324210888152042, -4.94404473249007, 1.2556361445691118))</td></tr><tr><td>5.6</td><td>3.0</td><td>4.1</td><td>1.3</td><td>List(1, 4, List(), List(5.6, 3.0, 4.1, 1.3))</td><td>List(1, 3, List(), List(-5.7565382594733165, -5.047995695147124, 0.4532026097791708))</td></tr><tr><td>5.5</td><td>2.5</td><td>4.0</td><td>1.3</td><td>List(1, 4, List(), List(5.5, 2.5, 4.0, 1.3))</td><td>List(1, 3, List(), List(-5.675856526152227, -4.635062261498161, 0.7005643333141736))</td></tr><tr><td>5.5</td><td>2.6</td><td>4.4</td><td>1.2</td><td>List(1, 4, List(), List(5.5, 2.6, 4.4, 1.2))</td><td>List(1, 3, List(), List(-5.974374086654396, -4.64519718427285, 0.6668189853982143))</td></tr><tr><td>6.1</td><td>3.0</td><td>4.6</td><td>1.4</td><td>List(1, 4, List(), List(6.1, 3.0, 4.6, 1.4))</td><td>List(1, 3, List(), List(-6.401503543434127, -5.280911288062211, 0.6525331208768361))</td></tr><tr><td>5.8</td><td>2.6</td><td>4.0</td><td>1.2</td><td>List(1, 4, List(), List(5.8, 2.6, 4.0, 1.2))</td><td>List(1, 3, List(), List(-5.740222147752619, -4.912466110630065, 0.8701277995413066))</td></tr><tr><td>5.0</td><td>2.3</td><td>3.3</td><td>1.0</td><td>List(1, 4, List(), List(5.0, 2.3, 3.3, 1.0))</td><td>List(1, 3, List(), List(-4.804261813862111, -4.3062989690305145, 0.7448344370468445))</td></tr><tr><td>5.6</td><td>2.7</td><td>4.2</td><td>1.3</td><td>List(1, 4, List(), List(5.6, 2.7, 4.2, 1.3))</td><td>List(1, 3, List(), List(-5.866876136970035, -4.811505243406311, 0.6248756286119104))</td></tr><tr><td>5.7</td><td>3.0</td><td>4.2</td><td>1.2</td><td>List(1, 4, List(), List(5.7, 3.0, 4.2, 1.2))</td><td>List(1, 3, List(), List(-5.842470045115695, -5.103543590146345, 0.5589560212858893))</td></tr><tr><td>5.7</td><td>2.9</td><td>4.2</td><td>1.3</td><td>List(1, 4, List(), List(5.7, 2.9, 4.2, 1.3))</td><td>List(1, 3, List(), List(-5.886581326729737, -5.023101706000194, 0.5636917390070479))</td></tr><tr><td>6.2</td><td>2.9</td><td>4.3</td><td>1.3</td><td>List(1, 4, List(), List(6.2, 2.9, 4.3, 1.3))</td><td>List(1, 3, List(), List(-6.153033375949514, -5.333794907300247, 0.8469379713721681))</td></tr><tr><td>5.1</td><td>2.5</td><td>3.0</td><td>1.1</td><td>List(1, 4, List(), List(5.1, 2.5, 3.0, 1.1))</td><td>List(1, 3, List(), List(-4.602879764659477, -4.563155005639491, 0.6505016790154277))</td></tr><tr><td>5.7</td><td>2.8</td><td>4.1</td><td>1.3</td><td>List(1, 4, List(), List(5.7, 2.8, 4.1, 1.3))</td><td>List(1, 3, List(), List(-5.809151005189905, -4.96770720921041, 0.6305859553495616))</td></tr><tr><td>6.3</td><td>3.3</td><td>6.0</td><td>2.5</td><td>List(1, 4, List(), List(6.3, 3.3, 6.0, 2.5))</td><td>List(1, 3, List(), List(-8.043070078222529, -5.3028814941686715, -0.11569355702007433))</td></tr><tr><td>5.8</td><td>2.7</td><td>5.1</td><td>1.9</td><td>List(1, 4, List(), List(5.8, 2.7, 5.1, 1.9))</td><td>List(1, 3, List(), List(-6.925415322956729, -4.739798674896669, 0.3463668702032119))</td></tr><tr><td>7.1</td><td>3.0</td><td>5.9</td><td>2.1</td><td>List(1, 4, List(), List(7.1, 3.0, 5.9, 2.1))</td><td>List(1, 3, List(), List(-8.127827706067015, -5.656659017796272, 0.7549064650627941))</td></tr><tr><td>6.3</td><td>2.9</td><td>5.6</td><td>1.8</td><td>List(1, 4, List(), List(6.3, 2.9, 5.6, 1.8))</td><td>List(1, 3, List(), List(-7.482158043689453, -5.133598036104059, 0.536225945858559))</td></tr><tr><td>6.5</td><td>3.0</td><td>5.8</td><td>2.2</td><td>List(1, 4, List(), List(6.5, 3.0, 5.8, 2.2))</td><td>List(1, 3, List(), List(-7.861101081733913, -5.272841181154133, 0.3586544136422565))</td></tr><tr><td>7.6</td><td>3.0</td><td>6.6</td><td>2.1</td><td>List(1, 4, List(), List(7.6, 3.0, 6.6, 2.1))</td><td>List(1, 3, List(), List(-8.908223018461108, -5.861891777039131, 0.9946382521357133))</td></tr><tr><td>4.9</td><td>2.5</td><td>4.5</td><td>1.7</td><td>List(1, 4, List(), List(4.9, 2.5, 4.5, 1.7))</td><td>List(1, 3, List(), List(-6.030726342867909, -4.1233720417583175, 0.09607956338347323))</td></tr><tr><td>7.3</td><td>2.9</td><td>6.3</td><td>1.8</td><td>List(1, 4, List(), List(7.3, 2.9, 6.3, 1.8))</td><td>List(1, 3, List(), List(-8.443348194774273, -5.667100736989836, 1.0664563728452978))</td></tr><tr><td>6.7</td><td>2.5</td><td>5.8</td><td>1.8</td><td>List(1, 4, List(), List(6.7, 2.5, 5.8, 1.8))</td><td>List(1, 3, List(), List(-7.831015891657024, -5.069175560202067, 0.9926872778674682))</td></tr><tr><td>7.2</td><td>3.6</td><td>6.1</td><td>2.5</td><td>List(1, 4, List(), List(7.2, 3.6, 6.1, 2.5))</td><td>List(1, 3, List(), List(-8.429477331427222, -6.095104360181007, 0.22102616089466087))</td></tr><tr><td>6.5</td><td>3.2</td><td>5.1</td><td>2.0</td><td>List(1, 4, List(), List(6.5, 3.2, 5.1, 2.0))</td><td>List(1, 3, List(), List(-7.1732780448024585, -5.556762131846497, 0.39994983104083914))</td></tr><tr><td>6.4</td><td>2.7</td><td>5.3</td><td>1.9</td><td>List(1, 4, List(), List(6.4, 2.7, 5.3, 1.9))</td><td>List(1, 3, List(), List(-7.313683550443704, -5.09856912418244, 0.6804604230023958))</td></tr><tr><td>6.8</td><td>3.0</td><td>5.5</td><td>2.1</td><td>List(1, 4, List(), List(6.8, 3.0, 5.5, 2.1))</td><td>List(1, 3, List(), List(-7.676721960736368, -5.530004014181985, 0.6096169113093031))</td></tr><tr><td>5.7</td><td>2.5</td><td>5.0</td><td>2.0</td><td>List(1, 4, List(), List(5.7, 2.5, 5.0, 2.0))</td><td>List(1, 3, List(), List(-6.855937315292796, -4.5383083056321505, 0.35989707628412093))</td></tr><tr><td>5.8</td><td>2.8</td><td>5.1</td><td>2.4</td><td>List(1, 4, List(), List(5.8, 2.8, 5.1, 2.4))</td><td>List(1, 3, List(), List(-7.096610397091616, -4.775416676961803, 0.012194606046128431))</td></tr><tr><td>6.4</td><td>3.2</td><td>5.3</td><td>2.3</td><td>List(1, 4, List(), List(6.4, 3.2, 5.3, 2.3))</td><td>List(1, 3, List(), List(-7.416086675996883, -5.433542721791674, 0.1626270147427129))</td></tr><tr><td>6.5</td><td>3.0</td><td>5.5</td><td>1.8</td><td>List(1, 4, List(), List(6.5, 3.0, 5.5, 1.8))</td><td>List(1, 3, List(), List(-7.460591879647469, -5.355453990236741, 0.600036000578974))</td></tr><tr><td>7.7</td><td>3.8</td><td>6.7</td><td>2.2</td><td>List(1, 4, List(), List(7.7, 3.8, 6.7, 2.2))</td><td>List(1, 3, List(), List(-9.000108477439356, -6.486268275072537, 0.513445011146616))</td></tr><tr><td>7.7</td><td>2.6</td><td>6.9</td><td>2.3</td><td>List(1, 4, List(), List(7.7, 2.6, 6.9, 2.3))</td><td>List(1, 3, List(), List(-9.30602995899294, -5.567989301781515, 1.159735810502322))</td></tr><tr><td>6.0</td><td>2.2</td><td>5.0</td><td>1.5</td><td>List(1, 4, List(), List(6.0, 2.2, 5.0, 1.5))</td><td>List(1, 3, List(), List(-6.809672922350787, -4.553709794287469, 0.9876521419771236))</td></tr><tr><td>6.9</td><td>3.2</td><td>5.7</td><td>2.3</td><td>List(1, 4, List(), List(6.9, 3.2, 5.7, 2.3))</td><td>List(1, 3, List(), List(-7.93951035680382, -5.69150570206313, 0.4241160244617319))</td></tr><tr><td>5.6</td><td>2.8</td><td>4.9</td><td>2.0</td><td>List(1, 4, List(), List(5.6, 2.8, 4.9, 2.0))</td><td>List(1, 3, List(), List(-6.709440470057935, -4.709144769044379, 0.13012432946861718))</td></tr><tr><td>7.7</td><td>2.8</td><td>6.7</td><td>2.0</td><td>List(1, 4, List(), List(7.7, 2.8, 6.7, 2.0))</td><td>List(1, 3, List(), List(-9.010608582081929, -5.771497197773053, 1.219675281230058))</td></tr><tr><td>6.3</td><td>2.7</td><td>4.9</td><td>1.8</td><td>List(1, 4, List(), List(6.3, 2.7, 4.9, 1.8))</td><td>List(1, 3, List(), List(-6.899011347964527, -5.110692744238823, 0.7062764162870865))</td></tr><tr><td>6.7</td><td>3.3</td><td>5.7</td><td>2.1</td><td>List(1, 4, List(), List(6.7, 3.3, 5.7, 2.1))</td><td>List(1, 3, List(), List(-7.7871967470886645, -5.648110256565616, 0.35808694184769907))</td></tr><tr><td>7.2</td><td>3.2</td><td>6.0</td><td>1.8</td><td>List(1, 4, List(), List(7.2, 3.2, 6.0, 1.8))</td><td>List(1, 3, List(), List(-8.125536928481305, -5.873090681087797, 0.8511884411271939))</td></tr><tr><td>6.2</td><td>2.8</td><td>4.8</td><td>1.8</td><td>List(1, 4, List(), List(6.2, 2.8, 4.8, 1.8))</td><td>List(1, 3, List(), List(-6.7689682807081075, -5.135586733385754, 0.5957872870592098))</td></tr><tr><td>6.1</td><td>3.0</td><td>4.9</td><td>1.8</td><td>List(1, 4, List(), List(6.1, 3.0, 4.9, 1.8))</td><td>List(1, 3, List(), List(-6.802012745520573, -5.198298478979602, 0.41115153394011905))</td></tr><tr><td>6.4</td><td>2.8</td><td>5.6</td><td>2.1</td><td>List(1, 4, List(), List(6.4, 2.8, 5.6, 2.1))</td><td>List(1, 3, List(), List(-7.634197078291283, -5.103868846259486, 0.48924920941717387))</td></tr><tr><td>7.2</td><td>3.0</td><td>5.8</td><td>1.6</td><td>List(1, 4, List(), List(7.2, 3.0, 5.8, 1.6))</td><td>List(1, 3, List(), List(-7.898907500151998, -5.777242981535236, 1.0947890559575293))</td></tr><tr><td>7.4</td><td>2.8</td><td>6.1</td><td>1.9</td><td>List(1, 4, List(), List(7.4, 2.8, 6.1, 1.9))</td><td>List(1, 3, List(), List(-8.352304023068355, -5.687466321858, 1.1437966336466214))</td></tr><tr><td>7.9</td><td>3.8</td><td>6.4</td><td>2.0</td><td>List(1, 4, List(), List(7.9, 3.8, 6.4, 2.0))</td><td>List(1, 3, List(), List(-8.743685996078845, -6.6852477667853085, 0.7612138719035526))</td></tr><tr><td>6.4</td><td>2.8</td><td>5.6</td><td>2.2</td><td>List(1, 4, List(), List(6.4, 2.8, 5.6, 2.2))</td><td>List(1, 3, List(), List(-7.670081470916105, -5.096398199245982, 0.43434311834451966))</td></tr><tr><td>6.3</td><td>2.8</td><td>5.1</td><td>1.5</td><td>List(1, 4, List(), List(6.3, 2.8, 5.1, 1.5))</td><td>List(1, 3, List(), List(-6.954445702158946, -5.1709224417262485, 0.7968480656138359))</td></tr><tr><td>6.1</td><td>2.6</td><td>5.6</td><td>1.4</td><td>List(1, 4, List(), List(6.1, 2.6, 5.6, 1.4))</td><td>List(1, 3, List(), List(-7.290983204681541, -4.813258936102956, 0.8185762805650882))</td></tr><tr><td>7.7</td><td>3.0</td><td>6.1</td><td>2.3</td><td>List(1, 4, List(), List(7.7, 3.0, 6.1, 2.3))</td><td>List(1, 3, List(), List(-8.587864718803633, -6.000488173055038, 0.9791878357166706))</td></tr><tr><td>6.3</td><td>3.4</td><td>5.6</td><td>2.4</td><td>List(1, 4, List(), List(6.3, 3.4, 5.6, 2.4))</td><td>List(1, 3, List(), List(-7.656329954492275, -5.4536303396862875, -0.09141964454643237))</td></tr><tr><td>6.4</td><td>3.1</td><td>5.5</td><td>1.8</td><td>List(1, 4, List(), List(6.4, 3.1, 5.5, 1.8))</td><td>List(1, 3, List(), List(-7.416206022920103, -5.362771239040807, 0.4822944638023965))</td></tr><tr><td>6.0</td><td>3.0</td><td>4.8</td><td>1.8</td><td>List(1, 4, List(), List(6.0, 3.0, 4.8, 1.8))</td><td>List(1, 3, List(), List(-6.6801965672533745, -5.150221230993884, 0.3603042135060548))</td></tr><tr><td>6.9</td><td>3.1</td><td>5.4</td><td>2.1</td><td>List(1, 4, List(), List(6.9, 3.1, 5.4, 2.1))</td><td>List(1, 3, List(), List(-7.61899682895624, -5.6862059799860845, 0.6153272380469534))</td></tr><tr><td>6.7</td><td>3.1</td><td>5.6</td><td>2.4</td><td>List(1, 4, List(), List(6.7, 3.1, 5.6, 2.4))</td><td>List(1, 3, List(), List(-7.825646492412519, -5.4973325816026755, 0.319904693766063))</td></tr><tr><td>6.9</td><td>3.1</td><td>5.1</td><td>2.3</td><td>List(1, 4, List(), List(6.9, 3.1, 5.1, 2.3))</td><td>List(1, 3, List(), List(-7.433793982618724, -5.723994906987676, 0.5272722785477462))</td></tr><tr><td>5.8</td><td>2.7</td><td>5.1</td><td>1.9</td><td>List(1, 4, List(), List(5.8, 2.7, 5.1, 1.9))</td><td>List(1, 3, List(), List(-6.925415322956729, -4.739798674896669, 0.3463668702032119))</td></tr><tr><td>6.8</td><td>3.2</td><td>5.9</td><td>2.3</td><td>List(1, 4, List(), List(6.8, 3.2, 5.9, 2.3))</td><td>List(1, 3, List(), List(-8.074665810123781, -5.590698233048814, 0.3515114813815674))</td></tr><tr><td>6.7</td><td>3.3</td><td>5.7</td><td>2.5</td><td>List(1, 4, List(), List(6.7, 3.3, 5.7, 2.5))</td><td>List(1, 3, List(), List(-7.930734317587951, -5.618227668511604, 0.13846257755708224))</td></tr><tr><td>6.7</td><td>3.0</td><td>5.2</td><td>2.3</td><td>List(1, 4, List(), List(6.7, 3.0, 5.2, 2.3))</td><td>List(1, 3, List(), List(-7.455360146660708, -5.5021389528549935, 0.4634622238273316))</td></tr><tr><td>6.3</td><td>2.5</td><td>5.0</td><td>1.9</td><td>List(1, 4, List(), List(6.3, 2.5, 5.0, 1.9))</td><td>List(1, 3, List(), List(-7.037006729096843, -4.939702882617156, 0.763401535253359))</td></tr><tr><td>6.5</td><td>3.0</td><td>5.2</td><td>2.0</td><td>List(1, 4, List(), List(6.5, 3.0, 5.2, 2.0))</td><td>List(1, 3, List(), List(-7.2753890333099545, -5.393242917238334, 0.5119810410797658))</td></tr><tr><td>6.2</td><td>3.4</td><td>5.4</td><td>2.3</td><td>List(1, 4, List(), List(6.2, 3.4, 5.4, 2.3))</td><td>List(1, 3, List(), List(-7.412972173071204, -5.430600479056938, -0.0801084663591416))</td></tr><tr><td>5.9</td><td>3.0</td><td>5.1</td><td>1.8</td><td>List(1, 4, List(), List(5.9, 3.0, 5.1, 1.8))</td><td>List(1, 3, List(), List(-6.901009231102387, -5.0318370216367025, 0.2804472628771908))</td></tr></tbody></table></div>"]}}],"execution_count":128},{"cell_type":"markdown","source":["### 8. Imputer for Handling Missing Data\n<hr>\n* Missing data needs to be handled before feeding them to learning algorithms.\n* This becomes even more challenging at large scale.\n* PySpark provides transformer for handling missing data as well"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import Imputer"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":130},{"cell_type":"code","source":["df = spark.createDataFrame([\n    (1.0, float(\"nan\")),\n    (2.0, float(\"nan\")),\n    (float(\"nan\"), 3.0),\n    (4.0, 4.0),\n    (5.0, 5.0)\n], [\"a\", \"b\"])\n\nimputer = Imputer(inputCols=[\"a\", \"b\"], outputCols=[\"out_a\", \"out_b\"], strategy=\"median\")\nmodel = imputer.fit(df)\n\nmodel.transform(df).show()"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\">+---+---+-----+-----+\n  a|  b|out_a|out_b|\n+---+---+-----+-----+\n1.0|NaN|  1.0|  4.0|\n2.0|NaN|  2.0|  4.0|\nNaN|3.0|  2.0|  3.0|\n4.0|4.0|  4.0|  4.0|\n5.0|5.0|  5.0|  5.0|\n+---+---+-----+-----+\n\n</div>"]}}],"execution_count":131},{"cell_type":"markdown","source":["### 8. Transforming data to higher derivatives\n<hr>\n* PolynomialExpansion is a transformer to convert data to higher degree\n* Linear models needs data in higher degree to check possibility of a non-linear seperator"],"metadata":{}},{"cell_type":"code","source":["from pyspark.ml.feature import PolynomialExpansion\nfrom pyspark.ml.linalg import Vectors\n\ndf = spark.createDataFrame([\n    (Vectors.dense([2.0, 1.0]),),\n    (Vectors.dense([0.0, 0.0]),),\n    (Vectors.dense([3.0, -1.0]),)\n], [\"features\"])\n\npolyExpansion = PolynomialExpansion(degree=3, inputCol=\"features\", outputCol=\"polyFeatures\")\npolyDF = polyExpansion.transform(df)\n\npolyDF.show(truncate=False)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\">+----------+------------------------------------------+\nfeatures  |polyFeatures                              |\n+----------+------------------------------------------+\n[2.0,1.0] |[2.0,4.0,8.0,1.0,2.0,4.0,1.0,2.0,1.0]     |\n[0.0,0.0] |[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]     |\n[3.0,-1.0]|[3.0,9.0,27.0,-1.0,-3.0,-9.0,1.0,3.0,-1.0]|\n+----------+------------------------------------------+\n\n</div>"]}}],"execution_count":133},{"cell_type":"markdown","source":["### 9. Custom Transformer\n<hr>\n* What if all the transformers provided till now is not sufficient ?\n* We can create a custom transformer."],"metadata":{}},{"cell_type":"code","source":["from pyspark import keyword_only\nfrom pyspark.ml import Transformer\nfrom pyspark.ml.param.shared import HasInputCol, HasOutputCol\nfrom pyspark.sql.functions import udf\nfrom pyspark.sql.types import IntegerType\n \nclass StringLengthTransformer(Transformer, HasInputCol, HasOutputCol):\n \n   @keyword_only\n   def __init__(self, inputCol=None, outputCol=None):\n       super(StringLengthTransformer, self).__init__()\n       kwargs = self._input_kwargs\n       self.setParams(**kwargs)\n \n   @keyword_only\n   def setParams(self, inputCol=None, outputCol=None):\n       kwargs = self._input_kwargs\n       return self._set(**kwargs)\n \n   def _transform(self, dataset):\n       reverse = udf(lambda sentence: len(sentence), IntegerType())\n       return dataset.withColumn(self.getOutputCol(), reverse(dataset[self.getInputCol()]))"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":135},{"cell_type":"code","source":["df = spark.createDataFrame([(\"foo bar\",),(\"hello world\",)]).toDF(\"sentence\")\nstrlength = StringLengthTransformer(inputCol=\"sentence\", outputCol=\"len\")"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":136},{"cell_type":"code","source":["display(strlength.transform(df))"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>sentence</th><th>len</th></tr></thead><tbody><tr><td>foo bar</td><td>7</td></tr><tr><td>hello world</td><td>11</td></tr></tbody></table></div>"]}}],"execution_count":137},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":138}],"metadata":{"name":"7. Data Preprocessing using PySpark","notebookId":1424747355796727},"nbformat":4,"nbformat_minor":0}