Hadoop Streaming Emulator: Work on PyDev Eclipse - mikec964/chelmbigstock GitHub Wiki

Under construction

This Wiki explains how to create a PyDev project on Eclipse to use our emulator.

Prerequisite

  • Eclipse is already installed
  • PyDev is already installed on Eclipse and configured properly.
  • You have the emulator on your machine.

Steps to create a PyDev project for the emulator

First, we create an Eclipse workspace for PyDev, then create a MapReduce project in it.
Hint:
If you already have a workspace for PyDev and want to reuse it, skip 'Create a workspace' and go to 'Create a project'.

Create a workspace

  1. Start Eclipse and create a new workspace for your MapReduce project.
    When you start Eclipse, the Workspace Launcher dialog box appears. Enter a name for your new workspace.
  2. Configure the workspace
  3. From the top menu, select <Window><Preferences>. The Preferences dialog box appears.
  4. From the list box on the left, select <PyDev><Interpreters><Python Interpreter>
  5. Click the New... button. The Select interpreter dialog box appears.
  6. Click the Browse... button. Select the python.exe you want to use. The Selection needed dialog box appears. Just click OK.
  7. Click OK on the Preferences dialog box.

Hint:
A workspace can contain multiple projects. You can reuse the workspace for another project later so that you can skip this step. Alternatively, you can also create a new workspace every time you work on a new MapReduce project.

Create a project

  1. Create a new project
  2. From the top menu, select <File><New><Project...>.
    The New Project dialog box appears.
  3. Enter pydev in the Wizards text box. Select PyDev Project from the list box. Click Next >.
    The PyDev Project dialog box appears.
  4. In the Project name text box, enter whatever name you like.
  5. Make sure the version number in the Grammer version dropdown list is correct.
  6. If you have set up multiple Python interpreter, make sure the Interpreter dropdown list is correct.
  7. Click Finish at the bottom.
  8. If the Open Associated Perspective message box appears, just click Yes.
    The dialog box disappears and the project name you entered appears in the PyDev Package Explorer pane on the left side.
  9. Import the emulator source to the project
  10. Right click the project name in the PyDev Package Explorer.
    The context menu appears.
  11. From the context menu, select <New><Link to Existing Source>.
    A dialog box appears.
  12. Click Browse... From the dialog box which just appeared, select the emulator folder (something like your-source-path\chelmbigstock\emulator), and click OK then Finish.
  13. Make sure the entry, emulator, appeared under the project name.
  14. Make Mapper and Reducer source files.
  15. To craete a mapper source file, right click the project name and select <New><PyDev Module>.
    The Create a new Python module dialog box appears.
  16. Enter the mapper name in the Name text box. Do not append the .py extention to the name. Leave the Package text box blank.
  17. Click OK.
    The dialob box disappears and another dialog box appears.
  18. Select the template, Module: Main. Click OK.
    The source file for the mapper appears in the editor pane.
  19. If you are using python 3.x, add the line #/usr/bin/env python3 at the top. If it is python 2.7, add #/usr/bin/env python.
  20. For the reducer, do the same steps above.
  21. Make or import the test data
    TBD
  22. Configure the debug enrionment
  23. Open hdemu.py under the emulator folder under the project.
  24. TBD
⚠️ **GitHub.com Fallback** ⚠️