Setting up Windows machine for PySpark and Apache Sedona in Jupyter-Notebook. | Part-1.
Video demonstration on how to set the environment in Windows machine for the PySpark and Apache Sedona and using the same in Jupyter notebooks.
⏰ Timestamps:
00:00 Introduction
03:30 Issues resolved
04:28 Useful resources
04:40 Python packages
06:30 Environment variables in notebook
10:20 Hadoop home setup
13:40 Spark session
15:22 Data read/write
15:40 Display configurations with spark
16:35 Third party packages with Pyspark
17:35 Local pyspark installation and Jars
18:12 Manual downloading of Sedona Jars
19:50 Part 2 coming soon
📒 Folder access for notebook: https://drive.google.com/drive/folders/1ej...
📘 Resources links:
1. https://techcommunity.microsoft.com/t5/azu...
2. https://stackoverflow.com/questions/210761...
3. https://sedona.apache.org/1.5.0/setup/inst...
4. https://github.com/steveloughran/winutils
5. https://docs.oracle.com/javase/tutorial/se...
6. https://stackoverflow.com/questions/357624...
7. https://spark.apache.org/docs/latest/confi...
8. https://changhsinlee.com/install-pyspark-w...
9. https://medium.com/@sharifuli/running-spar...
10. https://towardsdatascience.com/how-to-use-...
11. https://medium.com/analytics-vidhya/instal...
12. https://stackoverflow.com/questions/461256...