Builder for an IGRAInstance that behaves efficiently in Spark.
Builder for an IGRAInstance that behaves efficiently in Spark.
The IGRAInstance will be serializable from the driver to the executors.
All tasks in an executor will share a single IGRAInstance.
For example:
val downloadManager = new DownloadManagerBuilder("/pb/downloads").addDownloader(new S3Downloader(spark.sparkContext.hadoopConfiguration)).build()
val graInstance = GRAInstanceBuilder()
.withDownloadManager(downloadManager)
.addDataset("s3://my.s3.bucket/pb/routing/data/usa")
.build()
Builder for an IGRAInstance that behaves efficiently in Spark. The IGRAInstance will be serializable from the driver to the executors. All tasks in an executor will share a single IGRAInstance.
For example:
val downloadManager = new DownloadManagerBuilder("/pb/downloads").addDownloader(new S3Downloader(spark.sparkContext.hadoopConfiguration)).build() val graInstance = GRAInstanceBuilder() .withDownloadManager(downloadManager) .addDataset("s3://my.s3.bucket/pb/routing/data/usa") .build()