Publishing AWDB data to ArcGIS Online with ArcGIS Pro API - PSU-CSAR/awdb-retrieve GitHub Wiki

We are using the Python API provided with ArcGIS Pro for this task because it is based on Python 3 (the future direction of ESRI Python) and because ESRI provided an example using this version of Python. Our deployment is based on this example.

Configuring the Python environment

ArcGIS Pro provides the capability to manage and customize multiple Python environments. However, you cannot customize the default environment. Click the 'clone' button next to the default environment to create a clone. This allows you to specify the name for the environment. In our deployment the environment name is arcgispro-py3-awdb. Cloning an environment can take a while so be patient while ArcGIS Pro copies the files.

UPDATE: After upgrading to ArcGIS Pro 3.1, the suds-jurko package was not available to be added. I copied the suds and suds_jurko-0.6-py3.9.egg-info folders into the C:\Users\lbross\AppData\Local\ESRI\conda\envs\arcgispro-py3-awdb\Lib\site-packages folder and this seems to have worked, although suds is not showing up in the ArcGIS Pro interface as installed package.

After cloning the environment, you need to add the package for suds-jurko 0.6 to use the SOAP protocols that some AWDB services require. Note that these tasks will need to be repeated when ArcGIS Pro is upgraded as each version of ArcGIS Pro has a unique Python environment rendering the old one unusable.

There are two files in the C:\Program Files\ArcGIS\Pro\bin\Python\Scripts directory that need to be customized to point at the awdb Python environment. They are propy.bat and proenv.txt. To avoid confusion and to leave the installed files in their default state, I created copies named propy_awdb.bat and proenv_awdb.txt respectively. Propy_awdb.bat is the script used when instantiating this process from a scheduled task. See this posting from ESRI on the utility of propy.bat.

If you need to add custom packages, you can also add them by adding a line to C:\Users\lbross\AppData\Local\ESRI\conda\envs\arcgispro-py3-awdb\Lib\site-packages\ArcGISPro.pth. I did not have to do this.

Running the script

To test the script locally using Windows Powershell:

  1. Navigate to C:\Program Files\ArcGIS\Pro\bin\Python\Scripts
  2. .\propy_awdb.bat "D:\projects\gisdata\awdb-retrieve\get_AWDB_stations_3.py" where the second argument is the full path to the script .

To verify that the script has run/is running successfully, check the logs at D:\projects\gisdata\awdb-retrieve\log. The logs that end in _3 are for the ArcGIS Online script. Those without that suffix are the original scripts that update the web services on basins.

Scheduling the script

Use Windows task scheduler with these settings:

  1. "Properties" > "Actions" > "Edit"
    • Enter ONLY the directory (no quotes) in the 'Start in' text box. Example: C:\Program Files\ArcGIS\Pro\bin\Python\Scripts
    • Enter ONLY the executable name (no quotes) preceded with "." in the 'Program/Script' text box. Example: .\propy_awdb.bat
    • Enter the full path to the script (including quotes) in the 'Add Arguments' text box. Example: "D:\projects\gisdata\awdb-retrieve\get_AWDB_stations_3.py"
  2. "Properties" > "General"
    • "Run with highest privileges"
    • "Run whether user is logged on or not"
    • Check that "Configure for" is set to your machine's OS version. Basins example: Windows Server 2012 R2

Current schedule

  • AWDB Retrieve runs at 3:00am every Monday. This script updates the ArcGIS services hosted on basins
  • AWDB Retrieve 3 runs at 10:00pm every Tuesday. This script updates feature services hosted on ArcGIS Online

ArcGIS Pro Licensing

The Python script will fail if it ArcGIS Pro does not have a valid session on the machine where the script is running. The error message is: RuntimeError: Not signed into Portal. To avoid this error, ArcGIS Pro needs to be authorized to work offline. Note that when this option is enabled, the user login associated with that license can only be used on that one machine. More information about this error.

When running the script through Powershell, usually when testing, Powershell needs to run under the same named user as ArcGIS Pro. Running Powershell as an Administrator will NOT work. See the PSU ArcGIS Online administrator for any licensing issues.

Supporting files and folders

ArcGIS Pro requires that everything be in a project. The project that hosts the files used by this script is located at D:\projects\gisdata\awdb-retrieve\projects\awdb. This project contains a map for each data layer that is published as a feature service.

There is a file geodatabase located at D:\projects\gisdata\awdb-retrieve\projects\awdb\awdb.gdb that contains a feature class for each map that is published as a feature service. These feature classes are not archived and are updated each time the script runs. Note that a feature class cannot be published to ArcGIS online if it is hosted in an SDE database. The attribute table does not get published. Also it is not recommended to use a shapefile for the data as shapefiles have limitations on the length of column names.