Providers of products, services, or data in the primary production sector are under pressure from their users and partners. Everyone wants to exchange useful pieces of information (data) rather than forcing farmers and customers to read information from one application and retype it into another. They not only expect data to flow, but applications to collaborate, providing benefits of automation, efficiency, and the whole being greater than the sum of the parts.
In this cloud computing and internet-connected world, we do that through web service interfaces called APIs. Application Programming Interfaces have been around for a long time (you may recall people using APIs to automate Excel and Word).
Today’s APIs are implemented through web servers, although unlike web sites, web service APIs are designed for other computers to consume, so there are no pretty user interfaces, just code. Web servers can scale to handle high loads and use secure protocols to protect data, so web service APIs are a logical step to make data available securely and to handle unpredictable demand from other systems.
Grasping the benefits
There are benefits to be gained over time by improving how your product, service, or data system integrates:
- Increasing connectivity with the other systems that your customers or suppliers use, enabling them to make better informed decisions, improve product quality or productivity;
- Leveraging the investment you have already made in capturing and cleansing data, turning that “hygiene” activity into something that adds value to your supply chain;
- Reducing duplication of data entry with its attendant increase in errors and latency; and
- Opening the door to two-way flow of information with your suppliers, collaboration partners, and others in the supply chain.
Challenges for existing systems
Connecting legacy applications through web services can be challenging if those applications were designed as standalone tools, client-server platforms, or even delivered through “green screen” terminal interfaces. These applications won’t have been designed for connection to the web.
- Modern web applications are “stateless”, freeing the server from maintaining the state of a transaction or workflow between calls from the client. This allows web applications to scale to thousands or millions of users. In web applications, the web browser itself may hold state as it provides the user experience, but this may not be possible for other systems connecting to our API. Careful design of these APIs is needed to ensure they don’t just reflect the flow of the original legacy application.
- Legacy applications may not support the technical features that would allow us to connect them directly to a web server. Often only file input and output or “scraping” data from an ANSI terminal screen are available, so that intermediate technologies are needed to collate and make data available.
- One of the largest challenges we’ve found is that engineers who maintain legacy applications may have not been exposed to web technologies. They are great developers and grasp the concepts, but there is a significant learning curve to the new platforms, and these developers are already committed to maintaining and enhancing the core system.
Some approaches to consider
We’ve assisted our customers to connect their existing agricultural applications to the world of REST web services, and it is worth sharing some of what we have learned.
Can we access the database? Sometimes we’ve been fortunate to have access to the underlying SQL database used by the application, which makes it feasible to expose data through an API. Care needs to be taken though:
- The security model of the application might not be fully represented through the database, so you may need to reproduce this or take other steps to avoid exposing data to which users should not have access.
- Inserting or editing data may not be possible, or may require significant effort to enforce the business rules for that data. If you’re lucky, stored procedures or other server-side mechanisms might be in place to help with this.
- The legacy database might not be able to provide the up-time or performance needed to handle hits from external systems. We worked with one database where the overall system performance was already marginal and adding more load would have brought the server to its knees.
Can an intermediate database help? With less accessible data storage, and systems where adding significant load to the existing database would be a problem, we have sometimes been able to use file export or database sync to take a copy of the data. We then use that database to provide data through the API. There are some additional advantages to this approach:
- We can re-map the data to a simpler data model, or one which better reflects how people will use the data;
- We can implement ways to secure the data that align with external access rules;
However, we also need to remember that this method will introduce latency – transactions that take place in the legacy system won’t immediately be available through the API (depending on the synchronisation method used).
Adding or updating data through a web service API takes some planning too (whether you use an intermediate database or not). Approaches might include:
- Posting or updating the data into the intermediate database, and then making the “synchronisation” process two-way to push data back to the legacy system; or
- Accepting the data and turning this into a queued or batch data load job into the legacy database. In this case, the post or update request would return an “in progress” status and the calling system would not be able to access that data until the loading task had completed.
Inserting or updating data via a web services API raises other design questions, such as how to handle rejected data and other exceptions. Is there a roll-back mechanism or a way of discarding changes that were initially accepted but then found wanting? How do we notify the application that supplied the data if the API call was asynchronous? There are approaches to all these, but it certainly takes experience and makes for some robust design discussions!
Rezare Systems routinely undertakes this sort of work for its customers, and right now we’re working through a lot of these challenges with the Data Linker project. I welcome your thoughts, suggestions, or questions – do get in touch.