Ingesting data from a authenticated REST API using Job Secrets

• Overview
• Prerequisites

Overview

In this tutorial you are going to learn how to use Secrets in a data job.

Scenario

You like to read news daily and are a huge Taylor Swift fan. Let's combine these passions into a single data job, which searches for Taylor Swift news and stores them in a database.

For source of our data, we are going to use the free, key protected API of newsapi.org.

Who is this article for?

Users who wants to learn how to use Secrets in a data job.

Estimated Time Commitment

If you have all the prerequisites in place, the completion of this tutorial should take 10 to 15 minutes.

Prerequisites

Since Job Secrets are stored securely, you'll need a pre-configured installation of the VDK Control Service and Hashicorp Vault:

1. A VDK Control Service installation Install VDK Control Service with custom SDK and a local VDK SDK installation configured to use it
2. A Configured VDK Control Service/Hashicorp vault integration Configuring Hashicorp Vault Instance for storing Secrets

Storing Secrets

In the first part of tutorial, you are going to create a data job and store a Secret for it.

Create Data Job

Create a data job, by executing the following command:

vdk create -n taylor-swift-news -t my-team

This will create a taylor-swift-news directory with some sample data jobs file inside. Delete the files so that only the empty directory remains.

Obtain an API Key

Go to newsapi.org and click the "Get API Key" button. Fill in the form and copy the API Key.

Store the API key in a Job Secret

You can use the "vdk secrets" command to store and retrieve secrets via the command line. If you are using the vdk cli on a private/secure console, you can directly set a secret via the following command

vdk create -n taylor-swift-news -t my-team --set "apiKey" "<your API Key goes here>"

Alternatively you can pass just the key for you secret to the command and then you'll get prompted to enter it and it won't be kept in your console's history.

vdk create -n taylor-swift-news -t my-team --set "apiKey"

Using secrets in a data job

How to use this template?

[ Edit: Start with a goal - what will people learn (what problem are you solving? How will people benefit from the completion of your tutorial? Short and as straightforward as possible) ]

[ If applicable: Who is your audience? ]

[ Edit: how much time will it take? to complete your tutorial]

Prerequisites

[ If applicable: guidelines or prerequisites (what do they need to know prior to starting the tutorial?) ]

[ Edit: First step ]

[ Edit: When asking users to carry out a step, explain the reasoning behind the action and the expected result. ]

[ Edit: Part 1 ]

[ Edit: Part one instructions. ]

[ Edit: Part 2 ]

[ Edit: Part two instructions. ]

Tip: [ If applicable: place your tip here or anywhere else ]

[ Edit: Second Step ]

[ Edit: Explain the reasoning and expected result. ]

[ Edit if applicable: Part 1 ]

[ Edit if applicable: instructions ]

Wrap-up

[ Edit: Congratulations on completing this tutorial! (Edit this part as you feel, with your free-form congrats sentence) ]

Conclusion

[ Edit: Summarize important lessons you'd like users to take away from the tutorial. ]

What's Next?

[ Edit: Help users along their journey by providing the next steps - these might be recommended reading, supplementary resources, or even another tutorial. ]

Checklist before publishing

• I have read the Versatile Data Kit Tutorial guidelines.
• Is the goal clear and can be found within the first 30 seconds of opening your tutorial?
• Are the steps clear and actionable for accomplishing the task your reader needed help with?
• Is it possible to complete your tutorial within 15-30 minutes?
• Do you have at least 1-2 images/visuals/media in your tutorial?
• Did you spell-check and proofread? Grammarly
• Is it easy to read? Hemingway. Quillbot is good for fluency and paraphrasing.
• Did you check and fix the accessibility issues? WAVE
• Is there a Call to Action / Next Steps?
• Test with a user if the tutorial is understandable - if applicable and if possible