hpc documentation

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== ArcGIS ======

Using ArcGIS on the cluster to execute GIS tasks is based on using the embedded Python shell for ArcGIS Server 10.1 and you have paid to use the Esri software at UD. A single seat includes all Esri products that UD licenses for all architectures. See [[http://sites.udel.edu/gis/software/esri-licensing/|UD GIS Esri licensing]] for complete details. Each user must install the ArcGIS shell in their own account before running an interactive or batch job on a cluster.

===== Install ArcGIS shell =====

Each user must install the embedded Python shell for ArcGIS Server 10.1 in their own account on the cluster.  It only needs to be done one time and must be done on a compute node. It requires approximately 2GB of storage, so make sure you choose a directory location with sufficient storage available such as your workgroup storage (''$WORKDIR/$USER'') for the installation. 

<note important>Please be patient.  The installation may take up to 5 minutes to complete. </note> 

The basics steps are

  * Define your ''workgroup''
  * Connect to a compute node using ''qlogin''
  * Run the install script ''/opt/shared/arcgis/bin/install-arcgis-10.1 <//install_dir//>''
  * ''exit'' the compute node after the installation is complete

<note important>
Installing the ArcGIS shell only needs to be done once for your account and it will only work for your account! However, you will need to [[:software:arcgis:arcgis#update-arcgis-shell-license|update the license]] annually when the [[:software:arcgis:arcgis#errorlicense-expired|license expires]]. 
</note>

This information is based on the [[:software:arcgis:procedural_install|installation procedure]] documented for ArcGIS on a Linux cluster.

===== Update ArcGIS shell License =====

Each year the Esri license is updated for ArcGIS products, so you will need to update your ArcGIS shell installation with the updated license file too. You can do this by using the following command (all on one line) by replacing ''<//your_install_directory//>'' with the name of the directory location you provided when you installed your ArcGIS shell such as your workgroup storage (''$WORKDIR/$USER'').

<code>
<your_install_directory>/arcgis/server/tools/authorizeSoftware /opt/shared/arcgis/data/10.1/ArcGISforServerAdvancedEnterprise_server.prvc
</code>



===== Troubleshooting =====

==== Not running X-Windows ====

=== ImportError: DLL load failed: DLL init failed  ===

The example below shows the error you get when you try to run the ArcGIS shell during an interactive job without an X-Windows server running. In this case, ''import arcpy'' cannot load successfully. You must run an [[arcgis#running-x-windows-and-display-set-properly|X-windows server like Xming with the DISPLAY set properly]] or ''unset DISPLAY'' before you run the ArcGIS shell as explained in the next section. 

<code bash>
[traine@mills ~]$ workgroup -g it_css
[(it_css:traine)@mills ~]$ qlogin
Your job 1422937 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 1422937 has been successfully scheduled.
Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n016 ...
[traine@n016 ~]$ $WORKDIR/traine/arcgis-10.1-shell
Adding package `x11/RHEL6.1` to your environment
Application tried to create a window, but no driver could be loaded.
Make sure that your X server is running and that $DISPLAY is set correctly.
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import arcpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\__init__.py", line 21, in <module>
    from arcpy.geoprocessing import gp
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\__init__.py", line 14, in <module>
    from _base import *
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 14, in <module>
    import arcgisscripting
ImportError: DLL load failed: DLL init failed
>>> quit()
[traine@n016 ~]$
</code>

=== unset DISPLAY ===

The example below shows the ''LIBGL'' error you get when you ''unset DISPLAY'' before running the ArcGIS shell during an interactive job without an X-Windows server running.  This error will likely appear in the .o file for a batch job. In both cases, this error can be ignored because the ArcGIS shell does not use a GUI and ''import arcpy'' does load successfully.

<code bash>
[traine@mills ~]$ workgroup -g it_css
[(it_css:traine)@mills ~]$ qlogin
Your job 1423025 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 1423025 has been successfully scheduled.
Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n015 ...
Last login: Tue Nov 24 16:21:11 2015 from mills.mills.hpc.udel.edu
[traine@n015 ~]$ unset DISPLAY
[traine@n015 ~]$ $WORKDIR/traine/arcgis-10.1-shell
Adding package `x11/RHEL6.1` to your environment
ERROR: Unable to start Xvfb on port 600.
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import arcpy
libGL error: failed to load driver: swrast
libGL error: Try again with LIBGL_DEBUG=verbose for more details.
>>> quit()
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":601"
      after 9 requests (7 known processed) with 0 events remaining.
[traine@n015 ~]$

</code>
==== Error: license expired ====

Although it is not obvious, the error shown below is what you get when you try to ''import arcpy'' if the license has expired for your ArcGIS shell installation. You will need to [[:software:arcgis:arcgis#update-arcgis-shell-license | update your ArcGIS shell license]] to fix this problem.

<code>
[traine@mills traine]$ workgroup -g it_css
[(it_css:traine)@mills traine]$ qlogin
Your job 386764 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 386764 has been successfully scheduled.
Establishing /opt/shared/OpenGridScheduler/local/qlogin_ssh session to host n016 ...
Last login: Wed Oct  2 16:31:09 2013 from mills.mills.hpc.udel.edu
[traine@n016 traine]$ $WORKDIR/traine/arcgis-10.1-shell
Adding package `x11/RHEL6.1` to your environment
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import arcpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\__init__.py", line 21, in <module>
    from arcpy.geoprocessing import gp
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\__init__.py", line 14, in <module>
    from _base import *
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 578, in <module>
    env = GPEnvironments(gp)
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 575, in GPEnvironments
    return GPEnvironment(geoprocessor)
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 531, in __init__
    self._refresh()
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\geoprocessing\_base.py", line 533, in _refresh
    envset = (set(env for env in self._gp.listEnvironments()))
RuntimeError: NotInitialized
>>> quit()
[traine@n016 traine]$
</code>
==== Error: existing output file ====

If the output file specified already exists, then an error smiliar to the one below is generated. The ArcGIS shell will not overwrite an existing output file. You must remove it before the run or specify a different file.

<code>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "arc-test.py", line 3, in <module>
    arcpy.Buffer_analysis("z:\\lustre\\scratch\\it_css\\ArcGISex\\nwkcntr.shp","z:\\lustre\\work\\it_css\\traine\\ArcGIS\\nwkcntr_Buffer","100 Meters","FULL","ROUND","NONE","#")
  File "C:\Program Files\ArcGIS\Server\arcpy\arcpy\analysis.py", line 687, in Buffer
    raise e
arcgisscripting.ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000725: Output Feature Class: Dataset z:\lustre\work\it_css\traine\ArcGIS\nwkcntr_Buffer.shp already exists.
Failed to execute (Buffer).
</code>

=====Details by cluster=====
  * [[software:arcgis:mills| Mills]]
  * [[software:arcgis:farber | Farber]]