02-Add a data pipeline

Let's understand the different components in this stage -

Pipeline name: Collect feedback from mongodb

Pipeline name - The name of the pipeline. In our scenario "Collect feedbacks from MongoDB"

Pipeline type - Batch / Stream.

Pipeline trigger -

  • Manual (UI) - Pipeline will run only when a user triggers it manually.

  • Scheduled - Pipeline will run on a scheduled basis. The default schedule is every hour. By clicking the "Set schedule" button we can set our preferred time for the pipeline to run.

Scheduling options (Basic / Advanced) -

Basic - Allows choosing a repeat option: minute, hour, day of the week, or day of the monthly basis.

Advanced - Allows choosing manually schedule, based on Cron expression. Cron expression is a string in the form of "* * * * *".

Stars location representation (from left to right) -

  1. Minute

  2. Hour

  3. Day (month)

  4. Month

  5. Day (week)

For example: the expression "15 14 1 * *" represents “At 14:15 on day-of-month 1.”

Useful links: