MemSQL Pipelines

Pipeline Definitions

Click PipelineDefinition to view the list of pipelines that are configured in the system. By default, the list of pipelines is displayed for the last 24 hours and is sorted in order of latest batch executed.

Pipeline Definition

You can search for pipelines by database name, status of the pipeline, and by status of the last batch. The following information is displayed about a pipeline.

AttributeDescription
Pipeline NameThe name of the pipeline.
Database NameThe name of the database on which the pipeline is executing.
StateThe current status of the pipeline, whether running, or stopped.
Last Batch Rows WrittenThe number of rows written while running the most recent batch of the pipeline.
Last Batch StateThe status of the last batch, whether succeeded, or failed.
Last Batch TimeThe time at which the last batch was run in date:time format.

Viewing JSON script of a pipeline

To view the JSON script of pipeline, click the name of the pipeline. The following image shows the JSON script for the pipeline named call_center_pipeline.

JSON Script

Click the Copy button to copy the JSON script to the clipboard.

Pipeline Runs

Click PipelineRuns to view details about all the runs of a pipeline. By default, the pipelines are sorted by highest number of runs.

By default, the pipeline runs are grouped by pipeline. Click the Group by drop-down menu to select a different grouping - by Batch Status, by Cluster, or by Database Name.

By default, the information is displayed for the Last 24 Hours. You can select another timeline from the Timeline drop-down menu.

Pipeline Runs 1

Click the Download button to download information about the pipelines as a .xlsx file.

You can view the following information for Pipelines.

AttributeDescription
Group NameThe name of the pipeline group.
Total RunsThe total number of pipeline runs executed.
Failed RunsThe number of pipeline runs that failed execution.
Total RowsThe number of rows in the selected pipeline.
Average Batch TimeThe average time taken by the batches of a run.
Total batch TimeThe total time taken by all the batches of a run.
Avg Extractor Wait TimeThe average waiting time by a pipeline to extract data into the MemSQL database.
Total Extractor Wait TimeThe total time taken by a pipeline to extract data into the MemSQL database.

Click the name of a pipeline to see the latest 10 runs of that pipeline. Click Rows, to change the number of runs you want to view.

Pipeline Runs 2

AttributeDescription
IDThe ID of a run within the selected pipeline.
Pipeline NameThe name of the pipeline to which the run belongs.
DatabaseThe database the run belongs to.
Batch StateThe state of the pipeline run, which can either be Failed or Queued or Succeeded.
#PartitionsThe number of partitions to create in the pipeline run.
#Partitions FinishedThe number of partitions successfully created.
Rows/secThe number of rows read from the source file per second.
#RowsThe total number of rows read from the source file.
Start TimeThe time at which the pipeline run started.
Batch TimeThe time taken for a particular batch.
Extractor Wait TimeThe time taken by a pipeline run to extract data into the MemSQL database.

Click the Download button for a pipeline to download the information about a particular pipeline and its runs. Information of the earliest 10 runs is downloaded as a .xlsx file.

Pipeline Files

To view files read by pipeline, do the following.

  1. Click PipelineFiles.
    You can view details about the files included in a pipeline.

  2. In the left pane, select a pipeline to view all the files that are read by the pipeline.

note

You can filter by database name, or by the run status of a pipeline.

Pipeline Files

The following information is displayed about the files.

MetricsDescription
File NameThe name of the source file from which the pipeline reads data.
StateDenotes the status of the data read process, whether succeeded, or failed.
BytesThe size of the source file.
Time TakenThe time taken to read the file.
Pipeline NameThe name of the pipeline to which the file belongs.
Database nameThe name of the database where the file is located.
Wait TimeThe time before the reading starts.
# of rowsThe number of rows read from the source file.
Processed timeThe time at which the file was processed in date: time format.