Managing Malformed Input Records

A malformed record is one that Spectrum™ Technology Platform cannot parse. When Spectrum™ Technology Platform encounters a malformed record, it can do one or more of the following:

  • Terminate the job
  • Continue processing
  • Continue processing until a certain number of bad records are encountered
  • Continue processing but write bad records to a log file (via an optional sink stage)
Note: Malformed records functionality is limited to sources configured to read from files local to the server and that do not have sorting configured. When a source is configured with either a remote file or with sort fields and the source encounters a malformed record, the job will terminate regardless of the configuration for malformed records.

To manage malformed records,

  1. Open the job in Enterprise Designer.
  2. Add a malformed records sink in your dataflow.
    1. Create your job by defining your input file and source stage and adding services and subflows to your dataflow.
    2. Do one of the following:
      • Connect a sink stage to the optional output port on the source stage in your dataflow. The optional port is the clear output port just beneath the black output port on your source stage. If you mouse over this port, you will see a tool tip that says, "error_port." Malformed records are written to this sink.
      • Connect nothing to the optional output port on the source stage in your dataflow, which causes Spectrum™ Technology Platform to ignore malformed records.

    The completed dataflow should look something like this:

    When you run your job, the Execution History will contain a column that shows the number of malformed records that were encountered during the job.

  3. By default Spectrum™ Technology Platform will terminate a job when it encounters a malformed record. This default behavior can be changed in Management Console. Regardless of your system's defualt behavior, you can override the default behavior for a job by following these steps:
    1. Open the job in Enterprise Designer.
    2. Within an open job, go to Edit > Job Options.
    3. Select either Do not terminate the job on a malformed record or select Terminate the job after encountering this many malformed records and enter the number of malformed records you will allow a job to encounter before terminating.