These settings control how Threat Manager ingests data from iManage Work databases. All settings are configured by editing docker-compose_v10.yaml.

In this section:

(On-premises) INC_INGESTION_SCHEDULE

INC_INGESTION_SCHEDULE controls the frequency at which Threat Manager ingests new data from the iManage Work database.

• Default: 3,600 seconds (one hour)

• Minimum: 900 seconds (15 minutes)

• Effect of reducing the interval: Alerts fire closer to the time the anomalous behavior occurred.

Task: Update INC_INGESTION_SCHEDULE

1. Open docker-compose_v10.yaml with a suitable editing program.

2. Under analytics-taskservice, uncomment the following line by deleting the #:

   # - INC_INGESTION_SCHEDULE=240

3. Modify the value (in seconds) to the interval that you require. For example, to set a 30-minute interval (1,800 seconds):

   - INC_INGESTION_SCHEDULE=1800

4. Save docker-compose_v10.yaml.

5. Restart the Docker stack to apply the changes. Refer to Stopping and restarting the Docker stack to apply new settings.

(On-premises) DATAIMPORTER_WORKETL_EXTRACTOR_PAGESIZE 

Threat Manager’s ingestion service uses SQL Server paging. If your environment uses a Microsoft SQL compatibility level lower than 110, you need to disable SQL paging within iManage Threat Manager by adjusting this value.

Task: Update DATAIMPORTER_WORKETL_EXTRACTOR_PAGESIZE 

1. Open docker-compose_v10.yaml with a suitable editing program.

2. Under analytics-dataimporter, within the environment: subsection, add the following:

   - DATAIMPORTER_WORKETL_EXTRACTOR_PAGESIZE=-1

3. Save docker-compose_v10.yaml.

4. Restart the Docker stack to apply the changes. Refer to Stopping and restarting the Docker stack to apply new settings.

(On-premises) DATAIMPORTER_TASKQUEUE_CONCURRENCY - Ingestion threads

DATAIMPORTER_TASKQUEUE_CONCURRENCY controls the number of CPU cores available for ingestion. Increasing this value can improve performance for large databases, multiple iManage Work libraries, or particularly active environments.

• Minimum: 1 (or 1-1). A range can be specified, for example 1-2.

• Each additional thread requires more JVM memory.

Task: Update DATAIMPORTER_TASKQUEUE_CONCURRENCY

1. Open docker-compose_v10.yaml with a suitable editing program.

2. Under analytics-dataimporter, within the environment: subsection, modify the value in the following parameter:

   DATAIMPORTER_TASKQUEUE_CONCURRENCY=1

   1 (or 1-1) is the minimum value. A range can be specified, for example 1-2.

3. Adjust the JVM memory allocation to match. For the following parameter:

   - JAVA_OPTS=-Xmx3g -XX:-UseGCOverheadLimit

   Adjust the 3g (3GB) to the required amount of RAM. For example, if using two threads, double this to 6g.

4. Save docker-compose_v10.yaml.

5. Restart the Docker stack to apply the changes. Refer to Stopping and restarting the Docker stack to apply new settings.

(On-premises) AUTO_RETRY_ settings

The AUTO_RETRY_ settings control how Threat Manager handles permanently failed ingestion tasks: the number of retries and the interval between them.

• Default behavior if not configured: A retry occurs automatically every four hours, up to a maximum of 720 retries (three months).

Task: Update AUTO_RETRY_ settings

1. Open docker-compose_v10.yaml.

2. Under analytics-taskservice, add the following parameters as required.

   • To set the number of retries for a permanently failed ingestion task, add:

     - AUTO_RETRY_COUNT=

     Specify the value as required.

   • To set the interval between retries, add:

     - AUTO_RETRY_TIME_PERIOD=

     Specify the value, in hours.

3. Save docker-compose_v10.yaml.

4. Restart the Docker stack to apply the changes. Refer to Stopping and restarting the Docker stack to apply new settings.

(On-premises) Adjust ingestion resolution in event of batching errors

If your ingestion logs show errors of the type “Number of results in batch has exceeded the configured threshold. Batch resolution will be dynamically reduced”, adjust the ingestion resolution period to reduce these errors and improve performance.

• Default: 3 days

• Recommendation: Reduce this period to less than the average time it takes the docHistory table to accumulate 50,000 (the batch size limit).

Task: Adjust ingestion resolution

1. Open docker-compose_v10.yaml.

2. Under analytics-dataimporter , within the environment: subsection, update the following:

   1. Set the time period unit:

      DATAIMPORTER_INGESTIONTASK_SUBTASKRESOLUTION_UNIT=

      Specify the value, for example DAYS or HOURS.

   2. Set the number of time period units for splitting an ingestion job into smaller tasks:

      DATAIMPORTER_INGESTIONTASK_SUBTASKRESOLUTION_DURATION=

      Specify the value, for example 1.

3. Save docker-compose_v10.yaml.

4. Restart the Docker stack to apply the changes. Refer to Stopping and restarting the Docker stack to apply new settings.

(On-premises) Adjust ingestion batch size in event of out of memory errors

In some environments, ingestion may continue running indefinitely and produce “out of memory” errors in the ingestion logs. Reducing the ingestion batch size reduces ingestion load and resolves this issue.

Task: Adjust ingestion batch size

1. Open docker-compose_v10.yaml.

2. Under analytics-dataimporter, within the environment: subsection, add:

   - DATAIMPORTER_WORKETL_EXTRACTOR_BATCHSIZE=25000

   - DATAIMPORTER_WORKETL_EXTRACTOR_BATCHRESOLUTION_DURATION=1

3. Save docker-compose_v10.yaml.

4. Restart the Docker stack to apply the changes. Refer to Stopping and restarting the Docker stack to apply new settings.

5. From Configuration > Settings > Status, manually retry any failed ingestion tasks.

If the error persists, repeat steps 1–5, reducing the DATAIMPORTER_WORKETL_EXTRACTOR_BATCHSIZE= value by 5,000, down to a minimum of 10,000. iManage recommends not reducing below 10,000 unless required, as lower values can increase overall ingestion time.