How can your application integrate with external services?
25 May
Lately, we have been working on applications interacting with a large series of external services.
We've integrated with Paypal, Salesforce, Microsoft Dynamics, Fotolia, Winmentor. The list could continue, but that is not in the scope of this article.
I have recently noticed that many developers and product owners are a bit scared by the idea of interacting with external services in their application. The risks are many:
External services should definitely not be used without a real purpose. They can bring so much complexity to the project and business dependency from a third party, that I strongly recommend thinking twice or even three times before picking such a technical partner. I even recommend getting in touch with the third party, to get a sense of their medium to long term plans on:
On numerous times, we have provided support to third parties in implementing the API that best fited our needs, or in implementing the Version 2 of their API according to customer needs. In order to optimise the communication between the two systems, focusing on the following aspects is recommended:
Get the information in the right format
In an API request more than half the time needed is taken up by the DNS lookup, connection and data transfer, more than the server will take to deliver the response. This is why I strongly recommend getting all the data in the right format all at once.
Below is a good example. You need to get the latest list of orders created in your system and you have two ways of pulling data to your application:
Method 1
We have two methods:
getRecentOrders(lastDownload: timestamp)
Returns a list in the format: array(orderId1, orderId2, etc.). Ex. array(1122, 1123, etc.)
getOrderInformation(orderId: int)
Returns a list of order details: object(orderId, products, customerName, etc.)
Method 2
We have just one method:
getRecentOrders(lastDownload: timestamp)
Returns a list in the format: array(orderId1 => object(orderId, products, customerName, etc.), orderId2 => object(orderId, products, customerName, etc.), etc.).
The advantage is obvious, if you need to retrieve 20 orders on a per hour cron, you will get all the information you need in just one call. If you used Method 1 it would take much more time to retrieve all the information. In addition, the external service will spend more CPU power to make separate queries for each system.
Also make sure the character encoding is right. Especially when you integrate a multi-language application, problems can occur due to special characters on each system. Make sure the caracter encoding matches between the API and your system.
Make sure the connection and transfer are done in a secure manner
You don't want unauthorised users having access to your data. Depending on the type of information you transfer, it might be sufficient to just use a API key authentication, or you might force SSL protocol for the tranfer, fixed IP filtering or additional encryption algorithms.
Never trust that data comes in the format required, always validate information retrieved from the API.
Don't forget about performance
If you need to retrieve large chunks of data consider streaming the response or taking data into smaller chunks. Also, if connection to the API might impact the user experience, consider creating a cache layer on your application (you can even use Drupal's built-in Cache layer) to keep local versions of the data and deliver data to the end user in a timely manner.
You can use cron jobs to update cache data depending on the statistical update intervals.
Failover mechanism
It's very important that you never truly rely on any system, regardless of their SLA. We have seen big third-parties having their own down-times. Most often, maintenance time-frame are scheduled so that your customers are affected as little as possible. But you still need to make sure your application never fails, even if one or more of your external systems are down or working at limited capacities.
One good method is to define a handshake mechanism, maybe an uptime system check on the API, to make sure the service is functional before runing sensitive operations.
You should also define a queue mechanism when sending data to the API. In case the system is not available, the requests remain in the queue and your application will try at the next cron job to re-send the data.
A local cache storage can be built on top of your API integration. This could mean that when, for example, you receive the currency exchange rate from your API, in case the API is down, you can decide to use the currency rate that was retrived one hour ago (of course in the scenario that your app is not focused on financial services and you need currency rates at granularity of seconds...).
Flexibility
Very often, the API provider will change the response format, authentication method or simply deploy a new version of the API. You don't want to revise your whole application in case something is changed on the API. I recommend starting your API implementation with a well written PHP class, that manages to separate the communication layer from the logical layer. You must implement good unit tests as well, to make sure the integration is proper prior to implementing it in the final app.
Define a good error logging mechanism. In case you are using Drupal, the watchdog layer might be sufficient for this job.
We've integrated with Paypal, Salesforce, Microsoft Dynamics, Fotolia, Winmentor. The list could continue, but that is not in the scope of this article.
I have recently noticed that many developers and product owners are a bit scared by the idea of interacting with external services in their application. The risks are many:
- Service uptime, failover mechanism,
- Owning your data,
- API updates breaking your application,
- Performance aspects,
- Data transfer, character encoding.
External services should definitely not be used without a real purpose. They can bring so much complexity to the project and business dependency from a third party, that I strongly recommend thinking twice or even three times before picking such a technical partner. I even recommend getting in touch with the third party, to get a sense of their medium to long term plans on:
- API structure,
- Monetization plan,
- Partner support,
- SLA.
On numerous times, we have provided support to third parties in implementing the API that best fited our needs, or in implementing the Version 2 of their API according to customer needs. In order to optimise the communication between the two systems, focusing on the following aspects is recommended:
Get the information in the right format
In an API request more than half the time needed is taken up by the DNS lookup, connection and data transfer, more than the server will take to deliver the response. This is why I strongly recommend getting all the data in the right format all at once.
Below is a good example. You need to get the latest list of orders created in your system and you have two ways of pulling data to your application:
Method 1
We have two methods:
getRecentOrders(lastDownload: timestamp)
Returns a list in the format: array(orderId1, orderId2, etc.). Ex. array(1122, 1123, etc.)
getOrderInformation(orderId: int)
Returns a list of order details: object(orderId, products, customerName, etc.)
Method 2
We have just one method:
getRecentOrders(lastDownload: timestamp)
Returns a list in the format: array(orderId1 => object(orderId, products, customerName, etc.), orderId2 => object(orderId, products, customerName, etc.), etc.).
The advantage is obvious, if you need to retrieve 20 orders on a per hour cron, you will get all the information you need in just one call. If you used Method 1 it would take much more time to retrieve all the information. In addition, the external service will spend more CPU power to make separate queries for each system.
Also make sure the character encoding is right. Especially when you integrate a multi-language application, problems can occur due to special characters on each system. Make sure the caracter encoding matches between the API and your system.
Make sure the connection and transfer are done in a secure manner
You don't want unauthorised users having access to your data. Depending on the type of information you transfer, it might be sufficient to just use a API key authentication, or you might force SSL protocol for the tranfer, fixed IP filtering or additional encryption algorithms.
Never trust that data comes in the format required, always validate information retrieved from the API.
Don't forget about performance
If you need to retrieve large chunks of data consider streaming the response or taking data into smaller chunks. Also, if connection to the API might impact the user experience, consider creating a cache layer on your application (you can even use Drupal's built-in Cache layer) to keep local versions of the data and deliver data to the end user in a timely manner.
You can use cron jobs to update cache data depending on the statistical update intervals.
Failover mechanism
It's very important that you never truly rely on any system, regardless of their SLA. We have seen big third-parties having their own down-times. Most often, maintenance time-frame are scheduled so that your customers are affected as little as possible. But you still need to make sure your application never fails, even if one or more of your external systems are down or working at limited capacities.
One good method is to define a handshake mechanism, maybe an uptime system check on the API, to make sure the service is functional before runing sensitive operations.
You should also define a queue mechanism when sending data to the API. In case the system is not available, the requests remain in the queue and your application will try at the next cron job to re-send the data.
A local cache storage can be built on top of your API integration. This could mean that when, for example, you receive the currency exchange rate from your API, in case the API is down, you can decide to use the currency rate that was retrived one hour ago (of course in the scenario that your app is not focused on financial services and you need currency rates at granularity of seconds...).
Flexibility
Very often, the API provider will change the response format, authentication method or simply deploy a new version of the API. You don't want to revise your whole application in case something is changed on the API. I recommend starting your API implementation with a well written PHP class, that manages to separate the communication layer from the logical layer. You must implement good unit tests as well, to make sure the integration is proper prior to implementing it in the final app.
Define a good error logging mechanism. In case you are using Drupal, the watchdog layer might be sufficient for this job.
Share on:
