Overriding Job File Locations
When you run a job at the command line using job executor or the Administration Utility, you can override the input file specified in the dataflow's source stage (such as Read from File), as well as the output file specified in the dataflow's sink stage (such as Write to File).
To do this in job executor, specify the following at the end of the job executor command line command:
StageName=Protocol:FileName
In the Administration Utility, use the --l argument in the job execute command:
--l StageName=Protocol:FileName
Where:
- StageName
-
The stage label shown under the stage's icon in the dataflow in Enterprise Designer. For example, if the stage is labeled "Read from File" you would specify
Read from File
for the stage name.To specify a stage within an embedded dataflow or a subflow, preface the stage name with the name of the embedded dataflow or subflow, followed by a period then the stage name:
EmbeddedOrSubflowName.StageName
For example, to specify a stage named Write to File in a subflow named Subflow1, you would specify:
Subflow1.Write to File
To specify a stage in an embedded dataflow that is within another embedded dataflow, add the parent dataflow, separating each with a period. For example, if Embedded Dataflow 2 is inside Embedded Dataflow 1, and you want to specify the Write to File stage in Embedded Dataflow 2, you would specify this:
Embedded Dataflow 1.Embedded Dataflow 2.Write to File
- Protocol
-
A communication protocol. One of the following:
- file
- Use the file protocol if the file is on the same machine
as the Spectrum™ Technology Platform server. For example, on Windows specify:
"file:C:/myfile.txt"
On Unix or Linux specify:"file:/testfiles/myfile.txt"
- esclient
- Use the esclient protocol if the file is on the computer where you are executing
the job if it is a different computer from the one running the Spectrum™ Technology Platform server. Use the following format:
esclient:ComputerName/path to file
For example,esclient:mycomputer/testfiles/myfile.txt
Note: If you are executing the job on the server itself, you can use either the file or esclient protocol, but are likely to have better performance using the file protocol.If the host name of the Spectrum™ Technology Platform server cannot be resolved, you may get the error "Error occurred accessing file". To resolve this issue, open this file on the server: SpectrumLocation/server/app/conf/spectrum-container.properties. Set the spectrum.runtime.hostname property to the IP address of the server. - esfile
- Use the esfile protocol if the file is on a file server. The file server must be
defined in Management Console as a resource. Use the following format:
esfile://file server/path to file
For example,esfile://myserver/testfiles/myfile.txt
Where myserver is an FTP file server resource defined in Management Console. - webhdfs
- Use the webhdfs protocol if the file is on a Hadoop Distributed File Server. The
HDFS server must be defined in Management Console as a resource. Use the following
format:
webhdfs://file server/path to file
For example,webhdfs://myserver/testfiles/myfile.txt
Where myserver is an HDFS file server resource defined in Management Console.
- FileName
-
The full path to the file you want to use as input or output.
Note: You must use forward slashes (/) in file paths, not backslashes.
To specify multiple overrides, separate each override with a comma.
Example File Override
The following job executor command would use the file C:/myfile_input.txt as the input file for the Read from File stage and would use the file C:/myfile_output.txt as the output file for the Write to File stage.
java -jar jobexecutor.jar -j Job1 -u Bob1234 -p "" "Read from File"="file:C:/myfile_input.txt" "Write to File"="file:C:/myfile_output.txt"