LargestSimulations - adda-team/adda GitHub Wiki
The aim of this page is to give a feeling of how large light scattering simulations can be done using ADDA. Hopefully, it will also present a variety of different applications, so that each example will tell an interesting short story by itself. Complexity of the light scattering simulation with ADDA depends on the many (connected) factors: scatterer size parameter, required discretization or accuracy, number of the dipoles in a computational grid, convergence speed of the iterative solver, and need for repeated calculations (e.g. orientation averaging). Therefore, this page consists of several categories (nominations) with several examples in each of them.
Currently there are only a very small number of examples. Users are encouraged to submit new examples based on their own work/experience. This can be done either by a comment to this wiki page or contacting the maintainer of this page directly. Before proceeding to the description of particular examples, we present a summary table.
| Parameter | X1 | X2 | N1 | N2 | T1 | T2 |
|---|---|---|---|---|---|---|
| Shape | sphere | aggregate | slab | red blood cell | sphere | aggregate |
| Volume-equivalent size parameter | 320 | 176 | 16 | 36 | 130 | 82 |
| Refractive index (average) | 1.05 | 1.5+10−4i | 1.31+0.1i | 1.03 | 1.2 | 1.5+10−4i |
| Dipoles per wavelength | 10 | 10 | 257 | 93 | 12 | 10 |
| Largest 1D grid size | 1024 | 958 | 8192 | 1408 | 512 | 444 |
| Number of dipoles in a grid, 106 | 1073 | 879 | 1073 | 1035 | 134 | 87 |
| Number of occupied dipoles, 106 | 562 | 92 | 1073 | 619 | 70 | 9.2 |
Relative residual norm (-eps) |
10−4 | 10−3 | 10−5 | 10−5 | 10−5 | 10−2 |
| Iterative solver used | Bi-CGStab | QMR | QMR | QMR | Bi-CGStab | QMR |
| Number of iterations (average) | 334 | 9200 | 23 | 10 | 29200 | 1100 |
| Number of runs of iterative solver | 1 | 2 | 2 | 1 | 1 | 116 |
| Number of processor cores | 512 | 1008 | 4096 | 560 | 64 | 240 |
| Processor time, THz×hours | 8.8 | 41 | 0.2 | 1.1 | 90 | 35 |
| Total memory used, GiB | 698 | 662 | 1093 | 772 | 71 | 65 |
| ADDA version | 0.79a2 | 1.3b4 | 1.2 | 0.79 | 0.75b | 1.3b4 |
| Supercomputer | MareNostrum | Sisu | Sisu | LISA | LISA | Sisu |
The values in bold are the largest (the hardest to reach) ones for a particular parameter among all examples
This was a benchmark simulation, performed to test the performance and extreme capabilities of ADDA on MareNostrum. It is published in Yurkin M.A. and Hoekstra A.G. The discrete-dipole-approximation code ADDA: capabilities and known limitations, J. Quant. Spectrosc. Radiat. Transfer 112, 2234–2247 (2011). ADDA was executed with a command line:
adda_mpi -m 1.05 0 -grid 1024 -size 640 -ntheta 1800 -iter bicgstab -eps 4
The aggregates consisisted of 1 million spheres with size parameter 1.76 and refractive index 1.5 + 0.0001i, packed with volume fraction 20%. Outer shape of the aggregate is spherical with circumscribing size parameter 301. ADDA was executed with a command line:
adda_mpi -m 1.5 0.0001 -shape read n1000000_dpl10.geom -dpl 10.0003 -ntheta 1800 -eps 3
This is the maximum size we could reach, but it was only for a single aggregate in fixed orientation. Thus, it was not included in the paper in contrast to the results for 105 spheres, where we could peform orientation averaging.
This was a part of a scalability study of ADDA on Sisu, performed by Antti Penttila. The overall parallel efficiency for the largest number of cores (4096) was estimated to be 63%, but it was 77% and 83% for 2048 and 512 cores respectively. ADDA was executed with a command line:
adda_mpi -size 100 -m 1.31 0.1 -grid 4096 32 8192 -shape box 0.0078125 2
The size was chosen relatively small (corresponding to very large number of dipoles per wavelength) to keep the simulations fast. However, at least 10 times larger sizes seem feasible with such hardware. In particular, such simulations can be used to study optical properties of effectively infinite inhomogeneous (particulate) slabs.
This simulation was a part of comparison of two methods for simulation of light scattering, published in Gilev K.V., Eremina E., Yurkin M.A., and Maltsev V.P. Comparison of the discrete dipole approximation and the discrete source method for simulation of light scattering by red blood cells, Opt. Express 18, 5681–5690 (2010).
Since both considered methods involve certain numerical errors, there was need for some reference, so that accuracy of the methods can be independently evaluated. Described simulation for very fine discretization was used as such a reference. ADDA was executed with a command line:
adda_mpi -lambda 0.4936 -m 1.03 0.0 -shape axisymmetric fung_100points.txt -size 7.5 -grid 1408 -scat_grid_inp scat_params_eryth.dat -yz
Moreover, convergence of DDA results with refining discretization was studied to estimate the accuracy of the reference itself.
This simulation was performed in frame of a benchmark study: Yurkin M.A., Maltsev V.P., and Hoekstra A.G. The discrete dipole approximation for simulation of light scattering by particles much larger than the wavelength, J. Quant. Spectrosc. Radiat. Transfer 106, 546–557 (2007). ADDA was executed 25 times in a row with a command line:
adda_mpi -m 1.2 0 -size 260 -grid 512 -ntheta 1440 -iter bicgstab -chpoint 17h [-chp_load]
where the last option was used for every incantation except the first one. Use of checkpoint system was necessary, since at that time no single job could use more than 1200 processor-hours on LISA.
The aggregates consisisted of 100 000 spheres with size parameter 1.76 and refractive index 1.5 + 0.0001i, packed with volume fraction 20%. Outer shape of the aggregate is spherical with circumscribing size parameter 140. The orientation of the aggregate was averaged, by performing 58 independent ADDA simulations and using 512 scattering planes for each orientation. ADDA was executed with a command line:
adda_mpi -m 1.5 0.0001 -shape read 1h_10dpl.geom -dpl 10.0 -ntheta 1800 -eps 2 -orient avg avg-all.dat
The results were compared with other methods in Penttilä A., Markkanen J., Väisänen T., Räbinä J., Yurkin M.A., and Muinonen K. How much is enough? The convergence of finite sample scattering properties to those of infinite media, J. Quant. Spectrosc. Radiat. Transfer 262, 107524 (2021). One of the graphs is shown on another wiki page. Even larger aggregate has been simulated, but only in a fixed orientation.