The river routing module computes the lateral exchange of discharge between grid cells through the river network. For the transport directions we use the global () Simulated Topological Network (STN-30 drainage direction map (Vörösmarty et al., 2000). STN-30 organizes the Earth’s land area into drainage basins and provides the river network topology under the assumption that each grid cell can drain into one of the eight next-neighbor cells.

The transport of water in the river channel is approximated by a cascade of linear reservoirs. River sections of length d are divided into n homogeneous segments of length L, each behaving like a linear reservoir:

Following the unit hydrograph method (Nash, 1957), the outflow Q_out(t) of a linear reservoir cascade for an instantaneous inflow *Q_in is given as:

where Γ(n) is the gamma function that replaces (n-1) to allow for non-integer values of n. K is the storage parameter, defined as the hydraulic retention time of a single linear reservoir segment of length L. It can be calculated as the average travel time of water through a single river segment:

where v is the average flow velocity.

The discharge routing in LPJmL is calculated at a time step of Δt=3h. We assume a globally constant flow velocity v of 1 m s^-1 and a segment length L of 10 km to calculate the parameters n and K for each route between grid cell mid points. At simulation begin, for each route the unit hydrograph for a rectangular input impulse of length Δt is calculated. Because the first equation assumes and instantaneous input impulse, we numerically determine the response to a rectangular input impulse by adding up the responses of a series of 100 consecutive instantaneous input impulses. From the obtained unit hydrograph, the sum of outflow during each subsequent time step Δt is recorded until 99% of the total input impulse has been released (maximum 24 time steps). During simulation the thus determined response function is then used to calculate the convolution integral for the flow packages routed through the network.

The parallelization of the code is done with the message-passing paradigm. A portable implementation running on different platforms and architectures is provided by the message-passing interface (MPI). A MPI program consists of a number of tasks communicating via send and receive operations. While send and receive operations are performed by the tasks individually the collective operations defined in MPI are performed by all tasks simultaneously. The most important operations used in our parallel code are MPI_AlltoAll and MPI_Alltoallv performing a complete exchange of data.
MPI_Alltoall is a collective operation in which all tasks send the same amount of data to each other and receive the same amount of data from each other. MPI_Alltoallv adds flexibility to MPI_AlltoAll by allowing the caller to specify data to send and receive via a displacement end element count.The operation MPI_Gather is used to collect data from all task to one task in order to write out the results.

The water transport equation couples adjacent cells through the in- and outflows of water. Because of only next-neighbor connections a domain decomposition would be at a first glance the appropriate method for running the model in parallel. Dividing the full global two-dimensional grid in evenly sized subdomains would generate a poor performance due to the massive load imbalance mainly caused by the different fraction of ocean cells. Similar to other vegetation models the grid cells containing land area are therefore stored in a continuous array, but in that case the topology of the grid is lost and the indices of the neighbor cells have to be stored in a connection list. River-routing networks have a tree-like topology because the outflow is going to exactly one cell.

In the parallel implementation of LPJmL introduced here, the grid cells are evenly distributed among all tasks. The connection lists required for the river routing have to be distributed to the different tasks also. Because some grid cells need input from grid cells located at different task, a communication process must be established.
In a distributed network the connection list contains entries to remote grid cells, and the outflow of the cell has to be transferred to the remote cell stored in a different task. The algorithm for setting up the communication structure works in the following way:

1. For each task a sorted list of cell indices has to be created to which a connection exists. Sorting is necessary in order to delete duplicate entries. For a tree-like topology deleting duplicate entries is not necessary.
2. It must be determined how many outflow data have to be sent to remote cells of all tasks. This defines the length of the output buffers.
3. This information is distributed to all tasks via a MPI_Alltoall collective operation.
4. The indices of the cells which have to send the outflow to specific tasks must be distributed from all tasks to all tasks via a MPI_Alltoallv call. The length of the packets has been determined by step 2 and 3.
5. The index list linking the input buffer to the connection lists has to be built.

https://github.com/PIK-LPJmL/LPJmL/blob/master/src/lpj/initdrain.c
https://github.com/PIK-LPJmL/LPJmL/blob/master/src/lpj/drain.c
https://github.com/PIK-LPJmL/LPJmL/blob/master/src/lpj/transfer_function.c

River routing network is defined in /p/projects/biodiversity/input_VERSION2/drainagestn.bin. The file in CLM format contains the index the outflow is routed and the length of the river (m) stored as 32bit integers.
Lake fraction (%) is defined in /p/projects/biodiversity/input_VERSION2/glwd_lakes_and_rivers.bin in byte format.

• W. von Bloh, S. Rost, D. Gerten, and W. Lucht, 2010. Efficient parallelization of a dynamic global vegetation model with river routing. Environmental Modelling & Software 25, 685-690.
• D. Gerten, S. Rost, W. von Bloh, and W. Lucht, 2008. Causes of change in 20th century global river discharge. Geophys. Res. Lett. 35, L20405.
• S. Rost, D. Gerten, A. Bondeau, W. Lucht, J. Rohwer, and S. Schaphoff,

2008. Agricultural green and blue water consumption and its influence on the global water system. Water Resources Research 44: W09405.