Connect to APIs

How to connect, consume, transform and analyze data from external APIs

"Strech" provides the ability to connect and pull data from external API endpoints, for example

  • Your organization's API gateway

  • 3rd party APIs (For example: Twitter / Facebook / Instagram / Salesforce / SAP / etc...)

How to fill up the connection information?

At the section of defining a data source, please choose "Custom API"

Data integrity - Data chunks/objects will be sent toward the destination API in stream mode, meaning that if in the middle of the process the destination API gets shutdown, your destination might not contain 100% of the data. "Strech" will try to resend the failed object 3 times before canceling the failed request.

Using that mechanism we do our best to maintain both performance and data integrity.

(1) Endpoint URL. What is the endpoint URL? for example https://myhome.com/api/v1/getItems

(2) HTTP method.

There are two types of possible requests - GET and POST. Usually, "GET" is for quick and general REST retrieve data request, while "POST" is richer - It's not enough to reach a URL like "GET", you also need to send something in the "Request body" (payload). You should choose between the two based on the structure of the API you are trying to reach and the way the endpoints configured.

(3) Headers (JSON). Here is the place to put additional headers to the request, such as "authentication token", "Bearer Authentication", etc...

For OAuth types of authentication like the following for example:

"Authorization": "OAuth oauth_consumer_key='qpL2nl8os5DsLbbMMcfuCijx5',oauth_signature_method='HMAC-SHA1',oauth_timestamp='1616944848',oauth_nonce='0lL0ToUX9Du',oauth_version='1.0',oauth_signature='3qODFlxUmpSHmjyOGvdFYwfNnH0%3D'"

Should be written in this section like this:

{

"Authorization": "OAuth oauth_consumer_key=\"qpL2nl8os5DsLbbMMcfuCijx5\",oauth_signature_method=\"HMAC-SHA1\",oauth_timestamp=\"1616944848\",oauth_nonce=\"0lL0ToUX9Du\",oauth_version=\"1.0\",oauth_signature=\"3qODFlxUmpSHmjyOGvdFYwfNnH0%3D\""

}

(4) Params type.

There are two ways to attach your payload to your request.

  1. Query = exmaple: https://myhome.com/api/v1/getItems?category_id=2 (Usually for GET requests)

  2. Body = The payload will be encapsulated within the request (POST requests)

Same as section 2 - Depend on the way the API is configured to accept payload.

For both types, you should right the params the same way.

for example Query params like ?category_id=2, will be written like this: {"category_id":2}

(5) Params (JSON). The payload section. If you chose "Query" at the params type, the JSON format will be automatically cast into the "Query params" structure.

(6) Field name which holds your data.

At the response message, coming back from the API, at which field in the payload we should capture data from? for example, the response for a certain request is:

{
"statuses": [
{
"created_at": "Sun Feb 25 18:11:01 +0000 2018",
"id": 967824267948773377,
"id_str": "967824267948773377",
"text": "From pilot to astronaut, Robert H. Lawrence was the first African-American to be selected as an astronaut by any na… https://t.co/FjPEWnh804",
"truncated": true
},
{
"created_at": "Sun Feb 25 18:11:01 +0000 2018",
"id": 96782123123123773377,
"id_str": "967824267948773377",
"text": "From pilot to astronaut, Robert H. Lawrence was the first African-American to be selected as an astronaut by any na… https://t.co/FjPEWnh804",
"truncated": true
},
{
"created_at": "Sun Feb 25 18:11:01 +0000 2018",
"id": 96782543534534573377,
"id_str": "967824267948773377",
"text": "From pilot to astronaut, Robert H. Lawrence was the first African-American to be selected as an astronaut by any na… https://t.co/FjPEWnh804",
"truncated": true
},
]
}

At the example above, the proper input to the field should be statuses

Twitter API tutorial here.

Use "Custom API" with caution. POST requests towards API endpoint can be destructive.