This is an old revision of the document!
ArcGIS
Using ArcGIS on the cluster to execute GIS tasks is based on using the embedded Python shell for ArcGIS Server 10.1 and you have paid to use the Esri software at UD. A single seat includes all Esri products that UD licenses for all architectures. See UD GIS Esri licensing for complete details. Each user must install the ArcGIS shell in their own account before running an interactive or batch job on a cluster.
Details by cluster
Install ArcGIS shell
Each user must install the embedded Python shell for ArcGIS Server 10.1 in their own account on the cluster. It only needs to be done one time and must be done on a compute node. It requires approximately 2GB of storage, so make sure you choose a directory location with sufficient storage available such as your workgroup storage ($WORKDIR/$USER
) for the installation.
The basics steps are
- Define your
workgroup
- Connect to a compute node using
qlogin
- Run the install script
/opt/shared/arcgis/bin/install-arcgis-10.1 <install_dir>
exit
the compute node after the installation is complete
This information is based on the installation procedure documented for ArcGIS on a Linux cluster.
Update ArcGIS shell License
Each year the Esri license is updated for ArcGIS products, so you will need to update your ArcGIS shell installation with the updated license file too. You can do this by using the following command (all on one line) by replacing <your_install_directory>
with the name of the directory location you provided when you installed your ArcGIS shell such as your workgroup storage ($WORKDIR/$USER
).
<your_install_directory>/arcgis/server/tools/authorizeSoftware /opt/shared/arcgis/data/10.1/ArcGISforServerAdvancedEnterprise_server.prvc
Batch job
In order to use the ArcGIS shell it needs to be installed in your account. For account traine
in workgroup it_css
, the ArcGIS shell was installed in $WORKDIR/traine/arcgis-10.1-shell
.
A python script is used to execute the GIS tasks like the arc-test.py
below. This example refers to the full path of the input file as /lustre/work/it_css/ArcGIS
and the output file as /lustre/work/it_css/traine/ArcGIS
.
- arc-test.py
# This example code produces a 100 Meter buffer of a point located in the center of Newark, DE import arcpy arcpy.Buffer_analysis("z:\\lustre\\work\\it_css\\ArcGIS\\nwkcntr.shp","z:\\lustre\\work\\it_css\\traine\\ArcGIS\\nwkcntr_Buffer","100 Meters","FULL","ROUND","NONE","#")
You will also need a queue submission file to run your GIS tasks which will be based on the serial.qs
template in /opt/templates/gridengine
. The example below is a modified version called arc-submit.qs
for the account traine
in workgroup it_css
using the installed ArcGIS shell in /archive/it_css/traine/arcgis-10.1-shell
.
- arc-submit.qs
# Setup the environment; add vpkg_require commands after this # line: # Now append all of your shell commands necessary to run your program # after this line: $WORKDIR/traine/arcgis-10.1-shell arc-test.py
Create a directory <ArcGIS_projects>
to store your .py
and .qs
file. This is also the directory you most likely will want to reference for your input and output files. For account traine
in workgroup it_css
we will use $WORKDIR/traine/ArcGIS
. Now we can submit our batch run by using
[traine@mills ArcGIS]$ workgroup -g it_css [(it_css:traine)@mills ArcGIS]$ qsub arc-submit.qs
Interactive job
All interactive jobs should be run on a compute node by setting your workgroup
and using qlogin
. For account traine
in workgroup it_css
, the ArcGIS shell was installed in $WORKDIR/traine/arcgis-10.1-shell
. The same python script, arc-test.py, used for the batch job can be used for an interactive job.
Although there is no GUI with the ArcGIS shell, if it detects that your SSH connection is configured to enable X11-forwarding you will get an error unless you run an X-Windows server (e.g. Xming) or unset DISPLAY
before running the ArcGIS shell.
Running X-Windows and DISPLAY set properly
The example below shows an ArcGIS interactive job with an X-Windows server running.
[traine@mills ArcGIS]$ workgroup -g it_css [(it_css:traine)@mills ArcGIS]$ qlogin Your job 357451 ("QLOGIN") has been submitted waiting for interactive job to be scheduled ... Your interactive job 357451 has been successfully scheduled. Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n016 ... [traine@n016 ArcGIS]$ $WORKDIR/traine/arcgis-10.1-shell Adding package `x11/RHEL6.1` to your environment Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> execfile("arc-test.py") >>> quit() [traine@n016 ArcGIS]$
Troubleshooting
Not running X-Windows
ImportError: DLL load failed: DLL init failed
The example below shows the error you get when you try to run the ArcGIS shell during an interactive job without an X-Windows server running. In this case, import arcpy
cannot load successfully. You must run an X-windows server like Xming with the DISPLAY set properly or unset DISPLAY
before you run the ArcGIS shell as explained in the next section.
[traine@mills ~]$ workgroup -g it_css [(it_css:traine)@mills ~]$ qlogin Your job 1422937 ("QLOGIN") has been submitted waiting for interactive job to be scheduled ... Your interactive job 1422937 has been successfully scheduled. Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n016 ... [traine@n016 ~]$ $WORKDIR/traine/arcgis-10.1-shell Adding package `x11/RHEL6.1` to your environment Application tried to create a window, but no driver could be loaded. Make sure that your X server is running and that $DISPLAY is set correctly. Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import arcpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\__init__.py", line 21, in <module> from arcpy.geoprocessing import gp File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\__init__.py", line 14, in <module> from _base import * File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 14, in <module> import arcgisscripting ImportError: DLL load failed: DLL init failed >>> quit() [traine@n016 ~]$
unset DISPLAY
The example below shows the LIBGL
error you get when you unset DISPLAY
before running the ArcGIS shell during an interactive job without an X-Windows server running. This error will likely appear in the .o file for a batch job. In both cases, this error can be ignored because the ArcGIS shell does not use a GUI and import arcpy
does load successfully.
[traine@mills ~]$ workgroup -g it_css [(it_css:traine)@mills ~]$ qlogin Your job 1423025 ("QLOGIN") has been submitted waiting for interactive job to be scheduled ... Your interactive job 1423025 has been successfully scheduled. Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n015 ... Last login: Tue Nov 24 16:21:11 2015 from mills.mills.hpc.udel.edu [traine@n015 ~]$ unset DISPLAY [traine@n015 ~]$ $WORKDIR/traine/arcgis-10.1-shell Adding package `x11/RHEL6.1` to your environment ERROR: Unable to start Xvfb on port 600. Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import arcpy libGL error: failed to load driver: swrast libGL error: Try again with LIBGL_DEBUG=verbose for more details. >>> quit() XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":601" after 9 requests (7 known processed) with 0 events remaining. [traine@n015 ~]$
Error: license expired
Although it is not obvious, the error shown below is what you get when you try to import arcpy
if the license has expired for your ArcGIS shell installation. You will need to update your ArcGIS shell license to fix this problem.
[traine@mills traine]$ workgroup -g it_css [(it_css:traine)@mills traine]$ qlogin Your job 386764 ("QLOGIN") has been submitted waiting for interactive job to be scheduled ... Your interactive job 386764 has been successfully scheduled. Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n016 ... Last login: Wed Oct 2 16:31:09 2013 from mills.mills.hpc.udel.edu [traine@n016 traine]$ $WORKDIR/traine/arcgis-10.1-shell Adding package `x11/RHEL6.1` to your environment Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import arcpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\__init__.py", line 21, in <module> from arcpy.geoprocessing import gp File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\__init__.py", line 14, in <module> from _base import * File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 578, in <module> env = GPEnvironments(gp) File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 575, in GPEnvironments return GPEnvironment(geoprocessor) File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 531, in __init__ self._refresh() File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 533, in _refresh envset = (set(env for env in self._gp.listEnvironments())) RuntimeError: NotInitialized >>> quit() [traine@n016 traine]$
Error: existing output file
If the output file specified already exists, then an error smiliar to the one below is generated. The ArcGIS shell will not overwrite an existing output file. You must remove it before the run or specify a different file.
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "arc-test.py", line 3, in <module> arcpy.Buffer_analysis("z:\\lustre\\scratch\\it_css\\ArcGISex\\nwkcntr.shp","z:\\lustre\\work\\it_css\\traine\\ArcGIS\\nwkcntr_Buffer","100 Meters","FULL","ROUND","NONE","#") File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\analysis.py", line 687, in Buffer raise e arcgisscripting.ExecuteError: Failed to execute. Parameters are not valid. ERROR 000725: Output Feature Class: Dataset z:\lustre\work\it_css\traine\ArcGIS\nwkcntr_Buffer.shp already exists. Failed to execute (Buffer).