Hadoop Streaming Emulator: Work on PyDev Eclipse - mikec964/chelmbigstock GitHub Wiki
This Wiki explains how to create a PyDev project on Eclipse to use our emulator.
- Eclipse is already installed
- PyDev is already installed on Eclipse and configured properly.
- You have the emulator on your machine.
First, we create an Eclipse workspace for PyDev, then create a MapReduce project in it.
Hint:
If you already have a workspace for PyDev and want to reuse it, skip 'Create a workspace' and go to 'Create a project'.
- Start Eclipse and create a new workspace for your MapReduce project.
When you start Eclipse, the Workspace Launcher dialog box appears. Enter a name for your new workspace. - Configure the workspace
- From the top menu, select
<Window><Preferences>. The Preferences dialog box appears. - From the list box on the left, select
<PyDev><Interpreters><Python Interpreter> - Click the
New...button. The Select interpreter dialog box appears. - Click the
Browse...button. Select the python.exe you want to use. The Selection needed dialog box appears. Just clickOK. - Click
OKon the Preferences dialog box.
Hint:
A workspace can contain multiple projects. You can reuse the workspace for another project later so that you can skip this step. Alternatively, you can also create a new workspace every time you work on a new MapReduce project.
- Create a new project
- From the top menu, select
<File><New><Project...>.
The New Project dialog box appears. - Enter
pydevin the Wizards text box. SelectPyDev Projectfrom the list box. ClickNext >.
The PyDev Project dialog box appears. - In the Project name text box, enter whatever name you like.
- Make sure the version number in the Grammer version dropdown list is correct.
- If you have set up multiple Python interpreter, make sure the Interpreter dropdown list is correct.
- Click
Finishat the bottom. - If the Open Associated Perspective message box appears, just click
Yes.
The dialog box disappears and the project name you entered appears in the PyDev Package Explorer pane on the left side. - Import the emulator source to the project
- Right click the project name in the PyDev Package Explorer.
The context menu appears. - From the context menu, select
<New><Link to Existing Source>.
A dialog box appears. - Click
Browse...From the dialog box which just appeared, select the emulator folder (something like your-source-path\chelmbigstock\emulator), and clickOKthenFinish. - Make sure the entry, emulator, appeared under the project name.
- Make Mapper and Reducer source files.
- To craete a mapper source file, right click the project name and select
<New><PyDev Module>.
The Create a new Python module dialog box appears. - Enter the mapper name in the Name text box. Do not append the
.pyextention to the name. Leave the Package text box blank. - Click
OK.
The dialob box disappears and another dialog box appears. - Select the template,
Module: Main. ClickOK.
The source file for the mapper appears in the editor pane. - If you are using python 3.x, add the line
#/usr/bin/env python3at the top. If it is python 2.7, add#/usr/bin/env python. - For the reducer, do the same steps above.
- Make or import the test data
TBD - Configure the debug enrionment
- Open
hdemu.pyunder the emulator folder under the project. - TBD