Spark Setup on windows - ayushmathur94/Spark GitHub Wiki

Setup on asus :

spark version 3.0.3
hadoop 2.7 

Spark WebUI
http://laptop-5nmn0eq1:4040/jobs/

th process name: 11928@LAPTOP-5NMN0EQ1


go to spark home using cd %SPARK_HOME%
start Spark Master using C:\BigData\spark3>
   bin\spark-class2.cmd org.apache.spark.deploy.master.Master

successfully started service 'sparkMaster' on port 7077.
22/05/27 00:18:44 INFO Master: Starting Spark master at spark://192.168.1.10:7077
22/05/27 00:18:44 INFO Master: Running Spark version 3.0.3
22/05/27 00:18:44 WARN Utils: Service 'MasterUI' could not bind on port 8080. Attempting port 8081.
22/05/27 00:18:44 INFO Utils: Successfully started service 'MasterUI' on port 8081.
22/05/27 00:18:44 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://LAPTOP-5NMN0EQ1:8081
22/05/27 00:18:45 INFO Master: I have been elected leader! New state: ALIVE
localhost:8081   Master UI 

URL: spark://192.168.1.10:7077


start worker :
cd %SPARK_HOME%
start Spark Worker
using C:\BigData\spark3>
 bin\spark-class2.cmd org.apache.spark.deploy.worker.Worker -c 1 -m 4G spark://192.168.1.10:7077
 
 
 
 
 
 submit a spark job using -:
 1. cd %SPARK_HOME% 
 2. cd bin
 3. spark-submit --class "com.utils.LineCountDemo" --master local[2] "C:\JavaWorkSpace\Line_Count_Spark\target\Line_Count_Spark-0.0.1-SNAPSHOT.jar"