MongoDB - anodot/daria GitHub Wiki

Supported versions: MongoDB 3.0+

Source config

• Connection string - database connection string e.g. mongodb://mongo:27017
• Username
• Password
• Authentication Source - for delegated authentication, specify an alternate database name. Leave blank for normal authentication
• Database
• Collection
• Offset type - OBJECTID, STRING or DATE. Default - OBJECTID
• Initial offset - Number of days ago or specific date (format YYYY-MM-DD HH:mm:ss) when to start pulling data from (default 3 days ago).
• Offset field - default _id
• Batch size - how many records to send to further pipeline stages. Default - 1000 records
• Max batch wait time (seconds) - how much time to wait until the batch reaches its size. Default - 5 seconds

Example:

> agent source create
Choose source (mongo, kafka, influx): mongo
Enter unique name for this source config: mongo_test
Connection string: mongodb://mongo:27017
Username []:
Password []:
Authentication Source []:
Database: test
Collection: transactions
Is collection capped [False]:
Initial offset: 0
Offset type (OBJECTID, STRING, DATE) [OBJECTID]:
Offset field [_id]:
Batch size [1000]:
Max batch wait time (seconds) [5]:
Source config created

Source file config

Property	Type	Description
type	String	Specify source type. Value - influx
name	String	Unique source name - also the config filename
config	Object	Source configuration

All properties are required

config object properties:

Property	Type	Required	Description
configBean.mongoConfig.connectionString	String	yes	database connection string e.g. mongodb://mongo:27017
configBean.mongoConfig.username	String	no	username
configBean.mongoConfig.password	String	no	password
configBean.mongoConfig.authSource	String	no	for delegated authentication, specify alternate database name. Leave blank for normal authentication
configBean.mongoConfig.database	String	yes	database
configBean.mongoConfig.collection	String	yes	Collection
configBean.offsetType	String	no	OBJECTID, STRING or DATE. Default - OBJECTID
configBean.initialOffset	String	no	Amount of days ago or specific date (format YYYY-MM-DD HH:mm:ss) when to start pulling data from (default 3 days ago)
configBean.offsetField	String	no	default _id
configBean.batchSize	String	no	how many records to send to further pipeline stages. Default - 1000 records
configBean.maxBatchWaitTime	String	no	how many time to wait until batch will reach it's size. Default - 5 seconds

Example

[
  {
    "type": "mongo",
    "name": "test_mongo_1",
    "config": {
      "configBean.mongoConfig.connectionString": "mongodb://mongo:27017",
      "configBean.mongoConfig.database": "test",
      "configBean.mongoConfig.collection": "adtech",
      "configBean.mongoConfig.username": "root",
      "configBean.mongoConfig.password": "root",
      "configBean.mongoConfig.authSource": "admin"
    }
  }
]

Pipeline config

Pipeline config