Going round in circles

I’ve just finished refilling my Ecover washing up liquid bottle at work. Here in our shared office facilities our landlord is trying to get us all to go green and so has invested in a large returnable drum of Ecover from which we can all recharge our plastic bottles and avoid yet more plastic into land-fill.

My eco-crusade doesn’t stop there. In the past month we’ve stopped buying milk from the supermarket and I’m now popping into our local dairy farm on the way home and refilling glass bottles from their state-of-the art milk dispensing machine (at twice the price I might add).

Now I haven’t done the carbon calculations on any of this but what I do know is the amount of plastic we are getting through as a family has reduced significantly with just these two simple changes in habit.

At a time when Greta Thunberg is making waves across the Atlantic (literally) and movements like Extinction Rebellion are on the front pages, we simply cannot ignore the fact that the planet is in crisis.

So what does this mean for agriculture?

Those who have far better crystal balls than I do are suggesting that the future of business and the economy will be in what’s known as Circular Design. Unlike our current linear way of living (design, consume, throw away – or at best recycle), Circular Design is based on the rationale of there being no more waste, only the recycling of nutrients with a goal of arresting resource depletion and exploitation. Global sailing icon Ellen MacArthur is one of the big names leading the charge.

If the recycling of nutrients and a sustainable approach to our use of natural resources is the ambition, then agriculture must be central to the mission. And that’s the bit as someone in the agtech sector that excites me.

In an increasingly data-driven world, the opportunities for machine learning and AI to help us rethink the way we do things are growing by the day. As producers of food we are already seeing the norms of food production being challenged – impossible burgers, vertical farming and insect protein to name three. Whether these are truly “circular” I can’t say but they do signal the start of a revolution that is challenging what the farming sector has done for generations – and to traditionalists it feels uncomfortable.

But the truth is there isn’t a future in comfortable. We have such an existential crisis in an environmental sense that the rule book must be ripped up and those that tear the hardest are likely to win out.

To me that means adoption of smart, data-driven tech is an obligation not a privilege. It means we need to start collecting data on farm as a matter of urgency to begin to understand the complex dynamics of food production and resource use, and to deploy the best minds and technologies to redesign how we produce what we eat, how we consume it, and how we recharge the environment throughout this process.

We have such an existential crisis in an environmental sense that the rule book must be ripped up and those that tear the hardest are likely to win out.

Myriad projects could and should emerge that can establish the best production systems optimised by machines (sounds scary but isn’t) that calculate the “circularity” of the on-farm choices being made and that could be tied to market incentives for those that are indeed truly circular.

Imagine a future where data (privacy compliant of course) from your car, home and elsewhere is all linked up to the decisions you make about what you buy. In other words, the way in which you acquire and consume a product (food and non-food) is a dynamic calculation based on its own production history and your subsequent behaviours with it. Your “circularity” could become a badge of honour.

Governments the world over must incentivise the farming sector to make a step change. It is not good enough (in fact shameful) that something like 75% of the UK’s farmers do no electronic data recording at all. That might be fine to run an individual farm but it’s a collective disgrace when you look at the lost opportunity in a sustainability sense. Instrumenting farms and gathering good data is essential.

So as I take my refilled bottle of Ecover home via the milk dispensing machine, I can’t help but wonder what things will look like in five to 10 years from now. If it’s more of the same then we will all have failed. But if I and my children become more enthralled by sharing on social media how “circular” we are rather than obsessing about Snapchat streaks and Instagram likes, then that might suggest the tide has turned.

Or to put it another way, MacArthur won’t be the only one going round in circles!

Person in field stares at the sky

Three steps to your agritech product vision

You’re creating the future: a product or service that does not yet exist. How do you ensure your team is on the same page? Do you tell them how they will build it? How it will work? Or do you paint the picture of how it will make people’s lives better?

Your investors and partners are the same. Your product vision is the core of your business vision, so you want key people to understand and love it. Still, formalising your product vision seems like another chore in your busy schedule. You might even fear your idea will lose some of its magic in the harsh light of the day.

A clear representation of your vision:

  • Helps you communicate the vision to your team, your partners, and potential investors. You can be confident that you’ve passed on the key elements and not missed anything out.
  • Inspires others! People who believe and buy-in to your vision will go above and beyond the call of duty to make it a reality.

Here are three keys to get you started on composing your agricultural product vision.

Start with the users

Identify who your intended users are and are not. What problems will you solve for them? How much do those problems matter to your intended users? Are you tackling their problems or forcing your own solution?

The Einstellung Effect is a psychological term. It describes how we can fall in to the trap of applying a familiar approach or solution to problems. When all we have is a hammer, everything looks like a nail. This is a great reminder to us to fall in love with problems, not our solutions. Your agritech product vision should describe the problems you want to solve.

Think big

The eventual picture is larger than your first release. Think about what your vision looks like over the next three, five, or even ten years.

At the same time, your first product release (or releases) don’t need to fulfil the entire vision. Break the plan into bite-size chunks that you can achieve and learn from them on the way.

Trace weak signals and trends

Take a view about the way we will solve problems in two to five years. You’re creating the future, so yesterday’s rules may not apply.

I’m not suggesting divorce from reality. Ensure you have time and space to read, research, and test. Seek to understand how people might work in the future, and how your product might contribute. If your product is a service (to some extent all are), how might it fit with the ways your users want to communicate?

Your product vision is not a product specification. It’s not an elevator pitch either. Whether it is a story or bullets in a slide deck, it’s the way you bring your team and partners with you on the journey. It should help them pull together and solve the problems that count.

As your Agritech product vision evolves, there is one more thing to do: communicate relentlessly. Share your vision with your team and your partners. Evolve it based on what you learn.

New call-to-action
Family eating dinner

Are you making authentic supply chain promises?

If you’re in the food business (whether that’s retail, food service, processing, farming, or supply), consumers are asking questions about your supply chain.

Of course, they may not be asking you directly, and they may not be asking your retail or food service partner, but they are asking: on social media, on recommendation sites such as TripAdvisor and Yelp, even over drinks at their local.

Are you providing the information they need to be confident about the quality and safety of your product? Do you have a substantiated story around provenance, animal welfare and the environment?

Safeguards such as DNA testing lasagna are “bottom of the cliff” activities, an attempt to rebuild broken trust and arguably too limited and late in the supply chain.

Future product preference and even acceptance relies upon a supply chain that can show ethical practices: in how environmental impacts are managed, natural biodiversity is encouraged, animal welfare is maintained, anti-microbial resistance is avoided, and workers and communities are treated.

Activist groups and the power of social media means that our response to these demands must be much more solid than a promise or a declaration form. We must have the systems and measures to back up our words – and to demonstrate as much to auditors and our supply-chain partners.

For those of us at the confluence of technology and agriculture, this means we must do more than just record activities and calculate gross margins. We must step up with tools that capture rich data in support of farming activities, and which actively encourage good decisions that improve both profitability and sustainability.

All this needs to be done with minimal additional effort by farmers and their staff, and aligned to real-world processes on farm.

I’ll be speaking at MobileTech 2017, the annual summit for technology innovations in the primary sector, reflecting on these challenges. I’ll summarise some of the work Rezare Systems is doing in this space, and suggest ways the industry could apply technology to the opportunity.

This article was first published at www.rezare.com/blog  

Dairy farmer with technology

Farmers love technology, fear misuse

Increasing numbers of farmers see technology as useful and important to their farming businesses, and farmers are looking to invest further in new technology over the coming years. Despite this, lingering concerns about data sharing, privacy and control remain.

According to the October 2016 Commonwealth Bank of Australia Agri-Insights Survey of 1600 Australian farmers, 70% of farmers believe that the digital technology available adds significant value to their businesses.

The Ag Data Survey published by the American Farm Bureau Federation (AFBF) also found that farmers are optimistic about technology, with 77% of farmers planning to invest in new technology for their farms in the next three years.

Farmers also see value in sharing and re-use of data, but privacy and control are the largest barriers to more widespread re-use.

The Agri-Insights Survey found that:

  • 76% of farmers think that there is value in sharing on-farm production information with others;
  • 58% of farmers currently share some on-farm production information with others; and
  • Of farmers who don’t see value in data sharing, “privacy concerns” at 28% is the largest reason.

The New Zealand Office of the Privacy Commissioner surveyed New Zealanders about privacy and their attitudes to data sharing in April 2016. They noted that:

  • 57% of respondents were open to sharing data if they could choose to opt out;
  • 59% were open to sharing if there were strict controls on who could access data and how it was used; and
  • 61% were open to sharing if the data was anonymised and they couldn’t be personally identified.

The US AFBF survey also highlighted some of these concerns in an agricultural context:

  • Only 33% of farmers had signed contracts with their ag-tech provider. Another 39% knew of their provider’s policies but had not signed anything;
  • When farmers were asked if they were aware of the ways in which an ag-tech provider might use their data, 78% of farmers answered “no”; and
  • 77% of farmers were concerned about which entities can access their farm data and whether it could be used for regulatory purposes.

Not just farmers

Confidentiality and control can be barriers to companies too. After all, much of the data is about their activities, products, or equipment as well as the farm itself.

It’s not always clear how other parties will behave when sharing data. Organisations generally make reasonable and effective use of data and meet confidentiality expectations, but there is always a risk that they won’t. So companies sharing data are forced to negotiate “iron-clad” agreements, keeping the corporate lawyers busy and making any new data exchange the subject of long-winded negotiations.

As soon as you get into negotiations like this, costs rise. If one of the parties is a smaller player with less negotiating power (company or farmer), they may never be able to conclude a useful data access deal. The end result? A slower rate of innovation, the benefits of information to the farmer and overall supply chain are not fully realised, and sharing data becomes a much more expensive exercise than you would otherwise expect.

Over the years, industry players have experimented with different ways to address these issues. Centralised industry-good databases and exchanges have been proposed, and these could be very effective. Unfortunately, concern about centralising large amounts of data, and the loss of control that this brings has led players to hold back some or all of their data from such repositories.

Other groups have posited that all data should be in the exclusive control of the farmer, and have built exchanges or created open API standards on that basis. We applaud this, but it doesn’t always reflect the significant effort that companies and service providers invest in creating and curating some data sets. The end result is that some data sets are often held back from such exchanges.

A collaborative approach

The New Zealand primary industry has worked on several approaches to this problem in a collaboration between the red meat sector, the dairy sector, and the Ministry for Primary industries.

The Farm Data Code of Practice is designed to encourage greater transparency between farmers and service providers or vendors about the data that is held, and the rights that each party has to the data. A straight-forward accreditation process gives farmers confidence that organisations have “got their house in order” when it comes to terms and conditions and data policies.

The DataLinker protocol builds on the standardised, open API approach to sharing data, but with three key considerations:

  • It provides a way for organisations to agree a Data Access Agreement without a protracted legal negotiation. Standard agreements are provided and encouraged, to reduce the overhead that all parties face in legal costs and time (that said, custom agreements are still possible where absolutely necessary).
  • Accepting a Data Access Agreement doesn’t give the recipient “open slather” to the data; for most data sets, explicit farmer approval is also required, requested and confirmed by the farmer using standard web authorisation protocols. Farmers grant permission to access data that covers their business, and can also withdraw that authorisation.
  • As an Open API approach is used rather than a central database or exchange, there is no “central service” that must be involved in each data transfer. This reduces the “attack surface” from a security perspective and enables organisations to retain control of the data they hold.

Organisations adopting the DataLinker protocols benefit in several ways:

  • Farmers see that they are playing their part in maximising the use of information;
  • Standardised APIs and Data Access Agreements reduce the time and money invested in negotiating and creating custom solutions for every interaction;
  • Data Access Agreements mean that companies still retain the necessary control over high-value data sets, and are able to meet the privacy and confidentiality terms they have agreed with farmers; and
  • Companies and farmers can efficiently use sets of data which otherwise might have been too expensive to collect, or required a level of farmer input which would have discouraged adoption.

Our hope is that this framework will help organisations and farmers to maximise use of farm information, reducing long-term costs and encouraging greater innovation.

What you can do about this:

  • Want to see which New Zealand companies are accredited under the Farm Data Code of Practice? Check out www.farmdatacode.org.nz and drop an email to your key information providers to find out when they will be accredited.
  • Interested in the DataLinker protocols and how they can be adopted by your business? You’ll find information at www.datalinker.org.
  • Planning your strategy in this data space, or considering next steps? Talk to us – we’re happy to provide you with background and advice.

Why invest in tech for farming?

Every second business writer today seems to be talking about ag-tech, big data, and the internet of things. If you’re an agriculture sector organisation, or a company servicing or purchasing from primary producers, you might be forgiven for thinking you missed a day in the office and overnight farming has become a connected, automated, artificially-intelligent system.

Reality of course is still far from that utopia (or dystopia, depending on your point of view). What the writers are telling us is that there are a range of interesting technical possibilities that might have application in agriculture: and that some of the very early adopters, enthusiasts, and visionaries are trialling these. In some cases, they’ve used technologies in interesting experiments and learned useful things about their supply chain or farming system.

All well and good; but if you don’t consider yourself a leading edge visionary (or perhaps you don’t have the same appetite for risk) should you just ignore the hype, and wait until the technology matures?

For the pragmatists among us, who are more interested in achieving practical goals and strategic goals than experimenting, here are some areas where I think there is value in leveraging technology into your business today.

Improving communication

We all know that good communication between people is what keeps the wheels of business oiled and turning. Technologies, systems, and business rules are no replacement for good people relationships.

Your customers, suppliers, and business partners see all of your interactions with them as part of that same interpersonal relationship. Are your reports or invoices late? Do they lack critical information your business partners need? Personal reassurances will go so far, but if you can’t achieve timely and data-rich information delivery that helps your partners, they may look elsewhere.

We’ve helped a number of our customers lift their communications with suppliers or customers. In some instances, we’ve delivered apps that help farmers access key pieces of information as they become available. In other cases, we’ve implemented reports and visualisations that are valued by business partners for the timely insight they provide. This isn’t just data: it’s business communication.

Understanding critical business metrics

Many businesses try and track too many metrics, and don’t always closely manage the key metrics that are leading indicators of success.

What drives your business? Primary production and processing businesses are often heavily influenced by factors such as weather and global market demand, but these factors are outside the control of most businesses and may have less impact on long-term profitability than we imagine.

Its typically our response to outside factors, and our ability to continue to produce value despite them that determines long term success. Measures of productivity per unit of input, and effectiveness at delivering high-value products are better indicators than raw dollar returns or kilograms of product shipped.

For some of our customers, this has meant improving alignment between their financial data and physical data records. For instance, benchmarking kilograms of product produced, farm working expenses and profit against the potential pasture or crop production for that season, allowing more effective comparison of improvements across seasons. For others it has meant focusing on the proportion of product meeting specification for high-value markets, regardless of whether the market actually delivered the desired price premium in that particular season.

Data integration can help bring these disparate data sets together for timely comparison, and in-field monitoring technologies or remote sensing can deliver the physical data needed to make sense of the product and financial outcomes.

Responding to changing conditions

How do you assess and respond to the risk of changing weather and markets?

Studies of farmer responses to drought and similar challenges indicate that we tend to respond too late, and in a conservative, “piecemeal” fashion. We seem to bet that things will trend back to normal sooner rather than later, and under-do our response.

Monitoring tools such as climate stations and market data visualisations allow us to understand trends and risks early – before their impacts really start to bite. Of course, responding to these is still a challenge: will I reduce stock numbers only to see the weather change?

Mathematical models don’t yet give those definitive answers some futurists might lead you to expect, but they allow you to ask the “what if” questions, looking at potential decisions and impacts. These allow you to build a plan for your business and understand what your critical review and decision points need to be.

Time to start learning

The pace of change in technology is only likely to accelerate in coming years, and the technology we use in farming in ten years may be very different than what is now available. It is tempting to just “wait and see” what evolves, but advanced agricultural businesses choose to embrace technologies that can deliver concrete benefits and which align with their goals. Consider ways that technology can help you achieve more effective communication, improved understanding of business metrics, and the ability to assess and respond to change.

What technologies are you embracing in your business, and why?

Is connectivity interrupting your agricultural vision?

The last few years (even the last few months) have seen a surge in organisations exciting us about what farmers will do with sensors and mobile devices. We’ll be able to collect data with drones flying above our fields and tiny devices scattered across our farms. We’ll collect data with smart ear-tags and boluses in livestock, and we’ll crunch it all with powerful cloud-based analytics and smartphones will be a convenient way to tap into these analytics so that we can make decisions on the move.

That’s the promise.

Given time, much of this is achievable, though being savvy business people, farmers will only adopt technologies that deliver value or reduce risk. Even if the technology does provide significant value the may also be a challenge in connectivity; how do all these devices connect together and to the cloud?

I was thinking about this just last week, when I was demonstrating livestock management software to a group of farmers. It was a wintery day with snow on the hills, and we were standing out in the sheep yards, protected from the sun (but not the wind) by the steel roof.

We had just demonstrated how trivially easy it was to capture information about animals and synchronise it onto a smartphone, and now we were going to show the same information synchronised to the cloud, through a web browser and a mobile data connection.

We pressed “refresh” and we waited – and waited.

Did I mention that I had plenty of time to think?

Of course, eventually the page loaded and the demonstration carried on, reasonably successfully too. The farmers of course took the opportunity to point out that their remote yards would have no coverage at all!

We face a variety of different connectivity challenges in rural environments:

  • Low bandwidth and high latency connections;
  • Connectivity black-spots and intermittent coverage; and
  • Areas with no connection at all.

Low bandwidth and high latency

In many cases, rural networks (fixed or mobile) provide significantly lower bandwidth than those in the cities, or they may have significantly higher latency (time to respond). For some applications, this doesn’t matter at all, but for others it may be a show stopper.

I watched a drone software manufacturer demonstrate the features of a very smart unmanned aerial vehicle (UAV). On-board software used the GPS to navigate the device over a flight plan we outlined using Google Maps. A multi-spectral camera captured high resolution images at 1cm per pixel, and the device transmitted the images to the cloud while it was still flying. That’s pretty incredible!

Transmitting images as they are captured is a great innovation. The intensive processing power required to stitch images together and analyse them is provided by a remote data centre and applied to data arriving from drones around the world. It also frees the operator from messing about with memory cards or thumb drives and processing software.

This works just great on the 4G mobile networks in the Salinas Valley, California. Not so much in rural Wales where the mobile networks drop back to GSM or EDGE, with significantly lower throughput. Still, there may be opportunities to address this. If the drone has sufficient memory, it could cache images until bandwidth improves, or when it returns to a connected base. Alternatively, flying at 400ft above the terrain may provide better coverage than we experience on the ground.

High latency rural connections provide a different challenge. We typically experience an increase in latency with satellite connections, as signal travels through a far-distant satellite, to and from ground stations. Satellite connections can still be very high bandwidth (high data throughput), but have a measurable delay, as you’ll notice if you use a satellite based phone service – “over”.

This latency won’t affect most applications to any noticeable extent, but consider the case of a livestock auction happening at a remote location. Smart auction software allows internet bids to be synchronised with the bids happening on site, right down to subscribers hearing and seeing the auction in real time. A delay of one or two seconds in this case becomes significant, so auction planning and operation needs to take this into account.

Mobility “black spots”

Much of the rural countryside lacks mobile coverage. Hills, trees, and buildings can all affect local reception, especially as distance from a cell tower increases.

This limits our dependence on these networks for “life and death” situations. There’s a reason why Search and Rescue services encourage hikers and hunters to use Personal Locator Beacons (PLBs) rather than rely on mobile phones when in remote areas, and the same should apply to tools that are triggered when an ATV rolls on the farm – the vehicle could well be in a gully outside of cellular network coverage.

We should also plan on intermittent connectivity when developing apps for the farming sector. Requiring an internet connection to start or use an application will limit its use to favourable environments close to home where that coverage exists. An answer of course is smart synchronisation of relevant data, when devices come back into network coverage.

I was quite confident demonstrating our livestock management application remotely for example, as the list of animals had already been stored by the mobile device and most importantly the application was designed to work disconnected and sync later. If there was no coverage, the new data I had captured would be sent to the server when a connection became available.

Bringing your own network

There are always spots with no connectivity: sheep yards in a valley between steep hills; installing nitrate sensors in a creek in a ravine; even equipment in the shadow of a large steel building. What options are there if we must install our technologies where there is no connection?

It turns out there are a range of options, but the need to be technology savvy and the cost of equipment rises in these cases.

  • Consumer networking such as Wi-Fi and Zigbee mesh networks operate at 2.4GHz, providing line-of-sight connectivity over relatively short distances. Trees, hills, and even buildings can block these signals. Wi-Fi’s real strength is that it is a relatively low-cost and easily deployed option. It’s less suitable where extreme battery life and low power is a requirement.
  • There are also specialist wireless networks. These often use lower frequency radio transmissions, operating at 433, 868, or 900-921 MHz. They require a larger antenna, but these frequencies can travel longer distances without increasing the power requirement, and may be less prone to blockage by trees. Some of these networks transmit point-to-point between sender and receiver, and others form a mesh communicating between multiple devices.
  • New standards are evolving for low-power, long-range wireless networks such as the LoRaWAN and SigFox networks. Telecommunications providers are starting to adopt these standards to provide specialist connectivity for “Internet of Things” devices and remote locations. Where these are used at low frequencies in rural networks, they may well provide better coverage for monitoring devices in previously poor locations.
  • Finally, we might consider using mobile phones to “collect and deliver” data from monitoring devices in places where there are no coverage, but where coverage spots are regularly visited or passed. A remote sensor with some memory and Bluetooth Low Energy capability might deliver summary information to your smartphone when you pass by. This is the same approach that is used by activity sensors you wear on your arm – data is captured, and delivered to your phone at intervals.

So there are possible solutions even to those areas with currently poor or non-existent coverage – but many of these require some skills, choices or trade-offs, and often a higher investment cost, and this makes adoption of technology more challenging. Companies developing new products can’t cater for every possible network technology, and may end up integrating a just subset of connectivity options.

Our team at Rezare Systems often discusses these challenges and the evolving set of solutions, because these will enable some of the software solutions we “brain-storm”. We’re not telecommunication providers or hardware developers, but we are very interested in how connectivity influences adoption of new technology and our ability to capture and leverage data.

Have you had experience with connectivity issues in rural applications? Can you share advice for farmers and rural professionals about some of the suggestions above and their alternatives? Let us know your thoughts.

Building APIs for your agricultural product

Providers of products, services, or data in the primary production sector are under pressure from their users and partners. Everyone wants to exchange useful pieces of information (data) rather than forcing farmers and customers to read information from one application and retype it into another. They not only expect data to flow, but applications to collaborate, providing benefits of automation, efficiency, and the whole being greater than the sum of the parts.

In this cloud computing and internet-connected world, we do that through web service interfaces called APIs. Application Programming Interfaces have been around for a long time (you may recall people using APIs to automate Excel and Word).

Today’s APIs are implemented through web servers, although unlike web sites, web service APIs are designed for other computers to consume, so there are no pretty user interfaces, just code. Web servers can scale to handle high loads and use secure protocols to protect data, so web service APIs are a logical step to make data available securely and to handle unpredictable demand from other systems.

Grasping the benefits

There are benefits to be gained over time by improving how your product, service, or data system integrates:

  • Increasing connectivity with the other systems that your customers or suppliers use, enabling them to make better informed decisions, improve product quality or productivity;
  • Leveraging the investment you have already made in capturing and cleansing data, turning that “hygiene” activity into something that adds value to your supply chain;
  • Reducing duplication of data entry with its attendant increase in errors and latency; and
  • Opening the door to two-way flow of information with your suppliers, collaboration partners, and others in the supply chain.

Challenges for existing systems

Connecting legacy applications through web services can be challenging if those applications were designed as standalone tools, client-server platforms, or even delivered through “green screen” terminal interfaces. These applications won’t have been designed for connection to the web.

  • Modern web applications are “stateless”, freeing the server from maintaining the state of a transaction or workflow between calls from the client. This allows web applications to scale to thousands or millions of users. In web applications, the web browser itself may hold state as it provides the user experience, but this may not be possible for other systems connecting to our API. Careful design of these APIs is needed to ensure they don’t just reflect the flow of the original legacy application.
  • Legacy applications may not support the technical features that would allow us to connect them directly to a web server. Often only file input and output or “scraping” data from an ANSI terminal screen are available, so that intermediate technologies are needed to collate and make data available.
  • One of the largest challenges we’ve found is that engineers who maintain legacy applications may have not been exposed to web technologies. They are great developers and grasp the concepts, but there is a significant learning curve to the new platforms, and these developers are already committed to maintaining and enhancing the core system.

Some approaches to consider

We’ve assisted our customers to connect their existing agricultural applications to the world of REST web services, and it is worth sharing some of what we have learned.

Can we access the database? Sometimes we’ve been fortunate to have access to the underlying SQL database used by the application, which makes it feasible to expose data through an API. Care needs to be taken though:

  • The security model of the application might not be fully represented through the database, so you may need to reproduce this or take other steps to avoid exposing data to which users should not have access.
  • Inserting or editing data may not be possible, or may require significant effort to enforce the business rules for that data. If you’re lucky, stored procedures or other server-side mechanisms might be in place to help with this.
  • The legacy database might not be able to provide the up-time or performance needed to handle hits from external systems. We worked with one database where the overall system performance was already marginal and adding more load would have brought the server to its knees.

Can an intermediate database help? With less accessible data storage, and systems where adding significant load to the existing database would be a problem, we have sometimes been able to use file export or database sync to take a copy of the data. We then use that database to provide data through the API. There are some additional advantages to this approach:

  • We can re-map the data to a simpler data model, or one which better reflects how people will use the data;
  • We can implement ways to secure the data that align with external access rules;

However, we also need to remember that this method will introduce latency – transactions that take place in the legacy system won’t immediately be available through the API (depending on the synchronisation method used).

Adding or updating data through a web service API takes some planning too (whether you use an intermediate database or not). Approaches might include:

  • Posting or updating the data into the intermediate database, and then making the “synchronisation” process two-way to push data back to the legacy system; or
  • Accepting the data and turning this into a queued or batch data load job into the legacy database. In this case, the post or update request would return an “in progress” status and the calling system would not be able to access that data until the loading task had completed.

Inserting or updating data via a web services API raises other design questions, such as how to handle rejected data and other exceptions. Is there a roll-back mechanism or a way of discarding changes that were initially accepted but then found wanting? How do we notify the application that supplied the data if the API call was asynchronous? There are approaches to all these, but it certainly takes experience and makes for some robust design discussions!

Rezare Systems routinely undertakes this sort of work for its customers, and right now we’re working through a lot of these challenges with the Data Linker project. I welcome your thoughts, suggestions, or questions – do get in touch.