Skip to content

Commit d27e7c8

Browse files
DilumAluthgeDilumAluthge
committed
Docs: Move SGE docs from README to separate file
1 parent 14fbe35 commit d27e7c8

File tree

2 files changed

+70
-64
lines changed

2 files changed

+70
-64
lines changed

‎README.md‎

Lines changed: 0 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -39,70 +39,6 @@ You can also write your own custom cluster manager; see the instructions in the
3939

4040
For Slurm, please see the [SlurmClusterManager.jl](https://github.com/JuliaParallel/SlurmClusterManager.jl) package.
4141

42-
### SGE - a simple interactive example
43-
44-
```julia
45-
julia> using ClusterManagers
46-
47-
julia> ClusterManagers.addprocs_sge(5; qsub_flags=`-q queue_name`)
48-
job id is 961, waiting for job to start .
49-
5-element Array{Any,1}:
50-
2
51-
3
52-
4
53-
5
54-
6
55-
56-
julia> @parallel for i=1:5
57-
run(`hostname`)
58-
end
59-
60-
julia> From worker 2: compute-6
61-
From worker 4: compute-6
62-
From worker 5: compute-6
63-
From worker 6: compute-6
64-
From worker 3: compute-6
65-
```
66-
67-
Some clusters require the user to specify a list of required resources.
68-
For example, it may be necessary to specify how much memory will be needed by the job - see this [issue](https://github.com/JuliaLang/julia/issues/10390).
69-
The keyword `qsub_flags` can be used to specify these and other options.
70-
Additionally the keyword `wd` can be used to specify the working directory (which defaults to `ENV["HOME"]`).
71-
72-
```julia
73-
julia> using Distributed, ClusterManagers
74-
75-
julia> addprocs_sge(5; qsub_flags=`-q queue_name -l h_vmem=4G,tmem=4G`, wd=mktempdir())
76-
Job 5672349 in queue.
77-
Running.
78-
5-element Array{Int64,1}:
79-
2
80-
3
81-
4
82-
5
83-
6
84-
85-
julia> pmap(x->run(`hostname`),workers());
86-
87-
julia> From worker 26: lum-7-2.local
88-
From worker 23: pace-6-10.local
89-
From worker 22: chong-207-10.local
90-
From worker 24: pace-6-11.local
91-
From worker 25: cheech-207-16.local
92-
93-
julia> rmprocs(workers())
94-
Task (done)
95-
```
96-
97-
### SGE via qrsh
98-
99-
`SGEManager` uses SGE's `qsub` command to launch workers, which communicate the
100-
TCP/IP host:port info back to the master via the filesystem. On filesystems
101-
that are tuned to make heavy use of caching to increase throughput, launching
102-
Julia workers can frequently timeout waiting for the standard output files to appear.
103-
In this case, it's better to use the `QRSHManager`, which uses SGE's `qrsh`
104-
command to bypass the filesystem and captures STDOUT directly.
105-
10642
### Using `LocalAffinityManager` (for pinning local workers to specific cores)
10743

10844
- Linux only feature.

‎docs/sge.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Sun Grid Engine (SGE)
2+
3+
> [!WARNING]
4+
> The SGE functionality is not currently being maintained.
5+
>
6+
> We are currently seeking a new maintainer for the SGE functionality. If you are an active user of SGE and are interested in being a maintainer, please open a GitHub issue - say that you are interested in being a maintainer for the SGE functionality.
7+
8+
## SGE via `qsub`: Use `ClusterManagers.addprocs_sge` (or `ClusterManagers.SGEManager`)
9+
10+
```julia
11+
julia> using ClusterManagers
12+
13+
julia> ClusterManagers.addprocs_sge(5; qsub_flags=`-q queue_name`)
14+
job id is 961, waiting for job to start .
15+
5-element Array{Any,1}:
16+
2
17+
3
18+
4
19+
5
20+
6
21+
22+
julia> @parallel for i=1:5
23+
run(`hostname`)
24+
end
25+
26+
julia> From worker 2: compute-6
27+
From worker 4: compute-6
28+
From worker 5: compute-6
29+
From worker 6: compute-6
30+
From worker 3: compute-6
31+
```
32+
33+
Some clusters require the user to specify a list of required resources.
34+
For example, it may be necessary to specify how much memory will be needed by the job - see this [issue](https://github.com/JuliaLang/julia/issues/10390).
35+
The keyword `qsub_flags` can be used to specify these and other options.
36+
Additionally the keyword `wd` can be used to specify the working directory (which defaults to `ENV["HOME"]`).
37+
38+
```julia
39+
julia> using Distributed, ClusterManagers
40+
41+
julia> addprocs_sge(5; qsub_flags=`-q queue_name -l h_vmem=4G,tmem=4G`, wd=mktempdir())
42+
Job 5672349 in queue.
43+
Running.
44+
5-element Array{Int64,1}:
45+
2
46+
3
47+
4
48+
5
49+
6
50+
51+
julia> pmap(x->run(`hostname`),workers());
52+
53+
julia> From worker 26: lum-7-2.local
54+
From worker 23: pace-6-10.local
55+
From worker 22: chong-207-10.local
56+
From worker 24: pace-6-11.local
57+
From worker 25: cheech-207-16.local
58+
59+
julia> rmprocs(workers())
60+
Task (done)
61+
```
62+
63+
## SGE via `qrsh`: Use `ClusterManagers.addprocs_qrsh` (or `ClusterManagers.QRSHManager`)
64+
65+
`SGEManager` uses SGE's `qsub` command to launch workers, which communicate the
66+
TCP/IP host:port info back to the master via the filesystem. On filesystems
67+
that are tuned to make heavy use of caching to increase throughput, launching
68+
Julia workers can frequently timeout waiting for the standard output files to appear.
69+
In this case, it's better to use the `QRSHManager`, which uses SGE's `qrsh`
70+
command to bypass the filesystem and captures STDOUT directly.

0 commit comments

Comments
 (0)