example: mapreduce - modrpc/info GitHub Wiki
By building a simple MapReduce framework with ModPy, we can illustrate how ModPy resources can be used to build a distributed application. In this framework, we will slightly twist the Reduce phase so that reduction will happen through paralel prefix computation.
We will create two types of nodes: master node and worker nodes and data node. For simplicity, assume there is one master node and 10 worker nodes. The master node coordinates the overall workflow, worker nodes perform Map and Reduce functions, and the data node stores the data to operate on.
def map_reduce(nworkers):
# initialization phase
# map phase
for i in range(nworkers):
await modpy.
# reduce phase
# print result
There are three types of nodes: start node, finish node, internal nodes.
A digital system is typically described by finite state machines which is synchronized to a global clock.
Let's create a node which generate clock signals.
@modpy.event
def clock_tick_event()
return True
@modpy.timer
def clock_timer()
return yield from asyncio.sleep(1)
@modpy.proc
def clock_generator():
while True:
yield from modpy.wait_timer("clock_timer")
yield from modpy.publish("clock_tick_event", True)
counter = 0
@modpy.proc
def counter():
global counter
while True:
yield from modpy.wait_for("node0", "clock_tick_event")
counter = counter + 1