Connection Profile 3.5 - gflewis/sndml3 GitHub Wiki
The Connection Profile is a properties file that looks like this:
app.instance=dev00000
app.username=datapump.agent
app.password=******
app.agent=main
reader.instance=dev00000
reader.username=datapump.reader
reader.password=******
database.url=jdbc:mysql://localhost/sndm
database.schema=sndm
database.username=admin
database.password=******
The connection profile is specified with the -p (or --profile) command line option.
java -jar sndm-3.n.n-mysql.jar -p profile -y yamlfile
Beginning with release 3.5, the loader uses two separate connections for communicating with the ServiceNow instance. If you have installed and are using the DataPump App, then it is recommended that you use two separate user accounts.
-
reader— These credentials are used to read data records from the instance. This account should be grantedsnc_read_onlyrole, plus additional roles as required to access the records being exported. This account also requiressoap_query. -
app— These credentials are used to connect to the DataPump App, but are not used to read data. This account should be grantedx_108443_sndml.daemonrole. This account is unnecessary if you are using the command line (YAML) loader.
If you are NOT using the DataPump App, then you only need to specify a reader account. However, you must insure that account has read access to the following ServiceNow tables in order to create new SQL tables in the database:
sys_db_objectsys_dictionarysys_glide_object
Database username, password and schema are required unless you are using SQLite. For SQLite you only need to provide a URL. The value of datamart.schema will vary based on the DBMS.
- For MySQL,
datamart.schemashould be the name of the database - For Oracle,
datamart.schemashould be the name of the user - For Microsoft SQL Server,
datamart.schemashould bedbo - For SQLite,
datamart.schemashould bemainor left blank
The loader will interpolate environment variables using the notation ${varname}. For example, if the environment variable SNDML_CONFIG has been defined, then you can do something like this:
loader.metrics_folder=${SNDML_CONFIG}/metrics
If the entire property value is enclosed in backticks, then it will be passed to Runtime.exec() for evaluation. For example, if you have the LastPass CLI installed in /usr/local/bin, then you may be able to do something like this:
reader.password=`/usr/local/bin/lpass show --password my.servicenow.reader.password`
app.password=`/usr/local/bin/lpass show --password my.servicenow.datapump.password`
database.password=`/usr/local/bin/lpass show --password my.database.password`
Alternate names are provided for some properties in order to maintain backwards compatibility with earlier versions of the code. The list of property default values and alternate names is maintained in the file property_names.xml.
| Property Name | Alternate Name(s) | Notes / Description |
|---|---|---|
agent.backlog |
server.backlog or daemon.backlog | Maximum number of jobs that can be enqueued waiting for a thread. Applicable for --daemon or --server. Default is 60. |
agent.workers |
server.threads or daemon.threads | Maximum number of jobs that can be processed concurrently. Additional jobs will wait until a thread becomes available. Applicable for --daemon or --server. Default is 4. |
agent.shutdown_seconds |
server.shutdown_seconds or daemon.shutdown_seconds | Seconds that the --daemon or --server will wait before aborting worker threads. Default is 30. |
app.agent |
daemon.agent | Name of the Agent as configured in the DataPump app. Required if using the DataPump app. |
app.instance |
servicenow.instance | Required if using the DataPump app. Name or URL of the instance where the app is installed. |
app.password |
servicenow.password | Required if app.instance is specified. |
app.scope |
daemon.scope | Optional. Default is x_108443_sndml. |
app.username |
servicenow.username | Required if app.instance is specified. |
daemon.continue |
Controls whether or not the daemon will continue running after encountering an IO error in communication with ServiceNow or the database. If “true” then it writes an error to the log and goes back to sleep. If “false” (default) then it will immediately abort. | |
daemon.interval |
Recommended if using --daemon. Interval between scans in seconds. Default is 300. |
|
database.autocreate |
datamart.autocreate | Causes tables in the SQL database to be created if they do not already exist. Defaults is "true". |
database.dialect |
datamart.dialect | Specifies a section in the templates file. If not specified, the loader will infer the correct section using the drivers element of the template and the JDBC prefix of the datamart URL. |
database.password |
datamart.password | Required unless database is SQLite. |
database.schema |
datamart.schema | Recommended. See notes above. |
database.templates |
datamart.templates | The name of an XML file to use for generating SQL statements. If not specified, the loader will use sqltemplates.xml which is embedded in the JAR. For information refer to Templates. |
database.timezone |
This is normally not specified. The default value is "GMT". If set to "local" then the time zone of the Java VM will be used. | |
database.url |
datamart.url | Required. JDBC URL for the target database. Refer to the documentation for your selected JDBC driver. |
database.username |
datamart.username | Required unless database is SQLite. User name for the target database. |
database.warn_on_truncate |
loader.warn_on_truncate | Causes a warning to be written to the log whenever a field is truncated when written to the database. Default is "true". |
loader.metrics_folder |
The name of a directory where metrics files are located. If not specified, metric files are assumed to reside in the current working directory. For information refer to Metrics. | |
loader.pidfile |
daemon.pidfile or server.pidfile | If specified, then the process ID will be written to this file. This is used in conjunction with --server or --daemon. If you want to shut down the server then read the PID number from this file and send a kill signal to the PID. |
loader.rescan_delay_millisec |
Number of milliseconds to pause before each rescan. Applicable for --scan or --daemon. |
|
reader.domain |
servicenow.domain | For use in domain separated instances. This can be a single sys_id or a comma separated list of domains. Refer to Domain Separation. |
reader.instance |
servicenow.instance | Required. Name or URL of the ServiceNow instance from which data records will be retrieved. You can specify an instance name (e.g. dev12345) or a URL (e.g. https://dev12345.service-now.com) |
reader.pagesize |
servicenow.pagesize | Number of records to be processed at a time if not overridden at the table level. Default is 200. |
reader.password |
servicenow.password | Required. |
reader.username |
servicenow.username | Required. |
reader.verify_session |
servicenow.verify_session | Causes the loader, at startup, to verify that it can connect to ServiceNow by reading the sys_user record of the connecting user. Default is true. |
reader.verify_timezone |
servicenow.verify_timezone | Causes an exception to be thrown if the ServiceNow user time zone is not "GMT". If false then a warning is displayed. Default is false. |
server.heartbeat_seconds |
Applicable if using --server. Interval in seconds that the HTTP server will send a heartbeat message to the scoped app confirming that it is still listening and that the app session is still active. Default is 300. Set to 0 to disable the heartbeat. |
|
server.port |
Required if using --server. |