Hadoop Streaming Emulator: Work on PyDev Eclipse - mikec964/chelmbigstock GitHub Wiki
This Wiki explains how to create a PyDev project on Eclipse to use our emulator.
- Eclipse is already installed
- PyDev is already installed on Eclipse and configured properly.
- You have the emulator on your machine.
First, we create an Eclipse workspace for PyDev, then create a MapReduce project in it.
Hint:
If you already have a workspace for PyDev and want to reuse it, skip 'Create a workspace' and go to 'Create a project'.
- Start Eclipse and create a new workspace for your MapReduce project.
When you start Eclipse, the Workspace Launcher dialog box appears. Enter a name for your new workspace. - Configure the workspace
- From the top menu, select
<Window><Preferences>
. The Preferences dialog box appears. - From the list box on the left, select
<PyDev><Interpreters><Python Interpreter>
- Click the
New...
button. The Select interpreter dialog box appears. - Click the
Browse...
button. Select the python.exe you want to use. The Selection needed dialog box appears. Just clickOK
. - Click
OK
on the Preferences dialog box.
Hint:
A workspace can contain multiple projects. You can reuse the workspace for another project later so that you can skip this step. Alternatively, you can also create a new workspace every time you work on a new MapReduce project.
- Create a new project
- From the top menu, select
<File><New><Project...>
.
The New Project dialog box appears. - Enter
pydev
in the Wizards text box. SelectPyDev Project
from the list box. ClickNext >
.
The PyDev Project dialog box appears. - In the Project name text box, enter whatever name you like.
- Make sure the version number in the Grammer version dropdown list is correct.
- If you have set up multiple Python interpreter, make sure the Interpreter dropdown list is correct.
- Click
Finish
at the bottom. - If the Open Associated Perspective message box appears, just click
Yes
.
The dialog box disappears and the project name you entered appears in the PyDev Package Explorer pane on the left side. - Import the emulator source to the project
- Right click the project name in the PyDev Package Explorer.
The context menu appears. - From the context menu, select
<New><Link to Existing Source>
.
A dialog box appears. - Click
Browse...
From the dialog box which just appeared, select the emulator folder (something like your-source-path\chelmbigstock\emulator), and clickOK
thenFinish
. - Make sure the entry, emulator, appeared under the project name.
- Make Mapper and Reducer source files.
- To craete a mapper source file, right click the project name and select
<New><PyDev Module>
.
The Create a new Python module dialog box appears. - Enter the mapper name in the Name text box. Do not append the
.py
extention to the name. Leave the Package text box blank. - Click
OK
.
The dialob box disappears and another dialog box appears. - Select the template,
Module: Main
. ClickOK
.
The source file for the mapper appears in the editor pane. - If you are using python 3.x, add the line
#/usr/bin/env python3
at the top. If it is python 2.7, add#/usr/bin/env python
. - For the reducer, do the same steps above.
- Make or import the test data
TBD - Configure the debug enrionment
- Open
hdemu.py
under the emulator folder under the project. - TBD