Dataflow Pool Size

Dataflow pool size controls how may instances of each service dataflow can be running at a time. You can increase the pool size to improve performance to a point, but increased pool size may result in reduced performance if the server does not have the processor or memory resources available to handle several instances of each service dataflow running concurrently. If processor and memory resources are being used to their limit, you may find that reducing dataflow pool size, which limits the number of concurrent instances of each service dataflow, may provide more acceptable performance overall.

When finding the right pool size for your system, keep in mind that the dataflow pool size limits the number of instances of each service dataflow, not the total number of concurrent service dataflows. For example, with the default setting of 8, each service dataflow is allowed to have eight instances running at a time. If there are two service dataflows each utilizing the maximum of 8 concurrent instances, there would be 16 total instances of service dataflows running concurrently on your system.

Note: Dataflow pool size affects the performance of services only, not jobs.

To configure the dataflow pool size:

  1. Enable the JMX Console performance monitor. For more information, see Monitoring Performance with the JMX Console.
  2. Send multiple concurrent requests to the service through web services or the API.
  3. In the JMX Console, note the time shown in ServiceRuntimeManager.borrow.DataflowName. This shows the amount of time that a service request is waiting for space in the pool before running. The time is shown in milliseconds.
  4. Open this file:

    SpectrumLocation\server\app\conf\dataflowpool-pool-sizes.properties

  5. Increase or decrease the pool size based on what you think will result in improved performance. The default pool size is eight.
    • If the wait time shown in ServiceRuntimeManager.borrow.DataflowName was high, consider increasing the dataflow pool size to reduce the amount of time that service requests are waiting run. Or, you may want a high wait time in order to throttle requests and not overload server resources.
    • If the wait time shown in ServiceRuntimeManager.borrow.DataflowName was not high and you are experiencing slow performance, it may indicate that your server does not have the processor or memory resources to handle many concurrent service dataflows. Consider reducing the dataflow pool size to throttle requests.
  6. Save and close the file.
  7. Send multiple concurrent requests again, and observe the wait time shown in ServiceRuntimeManager.borrow.DataflowName in the JMX Console.
  8. Experiment with additional pool sizes until you find the optimal setting.