Skip to content

Commit 29e5972

Browse files
DilumAluthgeDilumAluthge
committed
Docs: Move SGE docs from README to separate file
1 parent 446f970 commit 29e5972

File tree

2 files changed

+70
-64
lines changed

2 files changed

+70
-64
lines changed

‎README.md‎

Lines changed: 0 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -68,70 +68,6 @@ for i in workers()
6868
end
6969
```
7070

71-
### SGE - a simple interactive example
72-
73-
```julia
74-
julia> using ClusterManagers
75-
76-
julia> ClusterManagers.addprocs_sge(5; qsub_flags=`-q queue_name`)
77-
job id is 961, waiting for job to start .
78-
5-element Array{Any,1}:
79-
2
80-
3
81-
4
82-
5
83-
6
84-
85-
julia> @parallel for i=1:5
86-
run(`hostname`)
87-
end
88-
89-
julia> From worker 2: compute-6
90-
From worker 4: compute-6
91-
From worker 5: compute-6
92-
From worker 6: compute-6
93-
From worker 3: compute-6
94-
```
95-
96-
Some clusters require the user to specify a list of required resources.
97-
For example, it may be necessary to specify how much memory will be needed by the job - see this [issue](https://github.com/JuliaLang/julia/issues/10390).
98-
The keyword `qsub_flags` can be used to specify these and other options.
99-
Additionally the keyword `wd` can be used to specify the working directory (which defaults to `ENV["HOME"]`).
100-
101-
```julia
102-
julia> using Distributed, ClusterManagers
103-
104-
julia> addprocs_sge(5; qsub_flags=`-q queue_name -l h_vmem=4G,tmem=4G`, wd=mktempdir())
105-
Job 5672349 in queue.
106-
Running.
107-
5-element Array{Int64,1}:
108-
2
109-
3
110-
4
111-
5
112-
6
113-
114-
julia> pmap(x->run(`hostname`),workers());
115-
116-
julia> From worker 26: lum-7-2.local
117-
From worker 23: pace-6-10.local
118-
From worker 22: chong-207-10.local
119-
From worker 24: pace-6-11.local
120-
From worker 25: cheech-207-16.local
121-
122-
julia> rmprocs(workers())
123-
Task (done)
124-
```
125-
126-
### SGE via qrsh
127-
128-
`SGEManager` uses SGE's `qsub` command to launch workers, which communicate the
129-
TCP/IP host:port info back to the master via the filesystem. On filesystems
130-
that are tuned to make heavy use of caching to increase throughput, launching
131-
Julia workers can frequently timeout waiting for the standard output files to appear.
132-
In this case, it's better to use the `QRSHManager`, which uses SGE's `qrsh`
133-
command to bypass the filesystem and captures STDOUT directly.
134-
13571
### Using `LocalAffinityManager` (for pinning local workers to specific cores)
13672

13773
- Linux only feature.

‎docs/sge.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Sun Grid Engine (SGE)
2+
3+
> [!WARNING]
4+
> The SGE functionality is not currently being maintained.
5+
>
6+
> We are currently seeking a new maintainer for the SGE functionality. If you are an active user of SGE and are interested in being a maintainer, please open a GitHub issue - say that you are interested in being a maintainer for the SGE functionality.
7+
8+
## SGE via `qsub`: Use `ClusterManagers.addprocs_sge` (or `ClusterManagers.SGEManager`)
9+
10+
```julia
11+
julia> using ClusterManagers
12+
13+
julia> ClusterManagers.addprocs_sge(5; qsub_flags=`-q queue_name`)
14+
job id is 961, waiting for job to start .
15+
5-element Array{Any,1}:
16+
2
17+
3
18+
4
19+
5
20+
6
21+
22+
julia> @parallel for i=1:5
23+
run(`hostname`)
24+
end
25+
26+
julia> From worker 2: compute-6
27+
From worker 4: compute-6
28+
From worker 5: compute-6
29+
From worker 6: compute-6
30+
From worker 3: compute-6
31+
```
32+
33+
Some clusters require the user to specify a list of required resources.
34+
For example, it may be necessary to specify how much memory will be needed by the job - see this [issue](https://github.com/JuliaLang/julia/issues/10390).
35+
The keyword `qsub_flags` can be used to specify these and other options.
36+
Additionally the keyword `wd` can be used to specify the working directory (which defaults to `ENV["HOME"]`).
37+
38+
```julia
39+
julia> using Distributed, ClusterManagers
40+
41+
julia> addprocs_sge(5; qsub_flags=`-q queue_name -l h_vmem=4G,tmem=4G`, wd=mktempdir())
42+
Job 5672349 in queue.
43+
Running.
44+
5-element Array{Int64,1}:
45+
2
46+
3
47+
4
48+
5
49+
6
50+
51+
julia> pmap(x->run(`hostname`),workers());
52+
53+
julia> From worker 26: lum-7-2.local
54+
From worker 23: pace-6-10.local
55+
From worker 22: chong-207-10.local
56+
From worker 24: pace-6-11.local
57+
From worker 25: cheech-207-16.local
58+
59+
julia> rmprocs(workers())
60+
Task (done)
61+
```
62+
63+
## SGE via `qrsh`: Use `ClusterManagers.addprocs_qrsh` (or `ClusterManagers.QRSHManager`)
64+
65+
`SGEManager` uses SGE's `qsub` command to launch workers, which communicate the
66+
TCP/IP host:port info back to the master via the filesystem. On filesystems
67+
that are tuned to make heavy use of caching to increase throughput, launching
68+
Julia workers can frequently timeout waiting for the standard output files to appear.
69+
In this case, it's better to use the `QRSHManager`, which uses SGE's `qrsh`
70+
command to bypass the filesystem and captures STDOUT directly.

0 commit comments

Comments
 (0)