Mercurial > repos > public > sbplib_julia

annotate distributedTest.jl @ 148:95a3ba70bccb parallel_test

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added some clarifying comments on the state of distributedTest.jl
author Vidar Stiernström <vidar.stiernstrom@it.uu.se>
date Tue, 26 Feb 2019 10:02:20 +0100
parents aa18b7bf4926
children 11b6646918d4
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
148
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
1 #NOTE: The followig code "works" in that the resulting error is small. Much more
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
2 # work needs to be done in order to figure out how julias distributed parallelism works
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
3 # especially w.r.t data movement.
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
4
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
5 #TODO: everywhere using here or just everywhere include?
146
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
6 @everywhere using DistributedArrays
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
7
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
8 # TODO: Currently uses integer division to calculate the local grid size.
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
9 # Should we make sure this is handled in some way if mod(sz./nworkers()) != 0
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
10 # or keep assertions?
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
11 @everywhere function create_partitioned_grid(size::NTuple{Dim, Int}, limit_lower::NTuple{Dim, T}, limit_upper::NTuple{Dim, T}, nworkers_per_dim::NTuple{Dim, Int}) where Dim where T
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
12 @assert mod.(size, nworkers_per_dim) == (0,0)
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
13 @assert prod(nworkers_per_dim) == nworkers()
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
14 # Translate the current worker id to a cartesian index, based on nworkers_per_dim
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
15 ci = CartesianIndices(nworkers_per_dim);
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
16 id = Tuple(ci[myid()-1])
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
17
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
18 # Compute the size of each partitioned grid
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
19 size_partition = size./nworkers_per_dim
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
20 size_partition = map(x->Int(x),size_partition) # TODO: Cant this be done in an easier way?...
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
21 # Compute domain size for each partition
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
22 domain_size = limit_upper.-limit_lower
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
23 domain_partition_size = domain_size./nworkers_per_dim
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
24
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
25 # Compute the lower and upper limit for each grid partition, then construct the grid
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
26 ll_partition = limit_lower .+ domain_partition_size.*(id.-1)
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
27 lu_partition = limit_lower .+ domain_partition_size.*id
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
28 grid = sbp.Grid.EquidistantGrid(size_partition, ll_partition, lu_partition)
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
29 return grid
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
30 end
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
31
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
32 # Create grid
148
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
33 #TODO: Should these be declared globally?
146
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
34 gridsize = (10000, 10000);
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
35 limit_lower = (0., 0.)
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
36 limit_upper = (2pi, 3pi/2)
148
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
37 nworkers_per_dim = (Int(nworkers()/2),Int(nworkers()/2))
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
38
146
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
39 # TODO: Currently only works with same number of processes in each direction and for
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
40 # an even number of processes
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
41 grids_partitioned = [@spawnat p create_partitioned_grid(gridsize, limit_lower , limit_upper, nworkers_per_dim) for p in workers()]
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
42
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
43 # Create Laplace operator
148
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
44 # TODO: If we dont have fetch here, then the error is large. Does this indicate that we need to move data, or simply that
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
45 # the future is not yet computed once this statement is reached?
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
46 Laplace_partitioned = [@spawnat p sbp.Laplace(fetch(grids_partitioned[p-1]), 1.0, sbp.readOperator("d2_4th.txt","h_4th.txt")) for p in workers()]
146
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
47
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
48 # Create initial value grid function v and solution grid function u
148
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
49 #TODO: Should init be declared globally?
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
50 init(x,y) = sin(x) + sin(y)
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
51 v = dzeros(gridsize) # Distribured arrays
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
52 u = dzeros(gridsize) # Distribured arrays
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
53 fetch([@spawnat p v[:L] = sbp.Grid.evalOn(fetch(grids_partitioned[p-1]), init) for p in workers()]) #TODO: Don't want to fetch here
146
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
54
147
aa18b7bf4926 Changed to initialize u and v as distributed arrays
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 146
diff changeset
55 # Apply Laplace
148
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
56 fetch([@spawnat p sbp.apply_tiled!(fetch(Laplace_partitioned[p-1]),u[:L], v[:L]) for p in workers()]) #TODO: Don't want to fetch here
146
21b188f38358 Started branch for testing out parallel computing
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents:
diff changeset
57
148
95a3ba70bccb Added some clarifying comments on the state of distributedTest.jl
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 147
diff changeset
58 #TODO: Here we need to make sure that the data is ready.
147
aa18b7bf4926 Changed to initialize u and v as distributed arrays
Vidar Stiernström <vidar.stiernstrom@it.uu.se>
parents: 146
diff changeset
59 @show maximum(abs.(u + v))