The end of APIs?

The end of APIs

Babel fish (from the Hitchhiker’s Guide to the Galaxy): The Babel fish is a small, bright yellow fish, which can be placed in someone's ear in order for them to be able to hear any language translated into their first language.

The evergreen science fiction novel “The Hitchhiker’s Guide to the Galaxy” features a strange creature called the Babel fish. You can put the Babel fish in your ear, and instantly you can understand any and all languages spoken in the universe. It is like an extremely advanced version of Google translation.

If you were using the Babel fish, you wouldn’t even know what language has been spoken by the other person, as different languages have been “abstracted out” for you. Everything is presented in a way for you to understand immediately.

In October 2023, I had written about the real life experience of Brad Fruth and his team at Beck’s Hybrid, as they digitized the available data on their website, the performance trial results data, and data from other sources to push to an LLM.

They quickly discovered how the data was stored made it easier or harder to get the data into an LLM. For example, a text table in a PDF is easy to extract compared to a table in a PDF which is an image. Data could also be in other formats. It could be structured data in a spreadsheet, or it could be a document of notes based on a meeting between two individuals.

Powerful tools like GPT4 vision, and Gemini can extract out data from multiple sources and formats, and provide a consistent representation (in most cases).

There are three main technical challenges for APIs and data interoperability. (I wrote about this in edition The Babel Fish of Agriculture)

Technical: Infrastructure and protocols: the physical infrastructure is in place to transport bits of data.

This refers to the existence of APIs, and protocols to exchange data between two systems.

Syntactic (or structural): Common data structure: the ability to communicate and exchange data between two or more systems through a standardized structure and format; shared syntax.

For example, if one system sends yield information, can the other system receive it correctly?

Semantic: Common data definition: the ability to exchange data between systems and for the data to be understood by each system; shared meaning.

For example, if one system interprets “moisture” data in a certain way, does the other system interpret the “moisture” data in the exact same way?

In the case of the Babel fish, it caused many wars and fights, as it solved technical interoperability very well, did an okay job on the syntactic interoperability, and did a poor job on the semantic interoperability!!

The agriculture industry suffers from challenges (to a varying degree) on all three types of interoperability. For example, the models developed by equipment manufacturers is a big challenge, as it creates proprietary data formats for devices, and for the resulting data coming from or consumed by them. It poses challenges for technical, syntactic, and semantic interoperability. 

With some of the newer GPT4 Vision and Turbo capabilities (Gemini from Google has similar capabilities), domain specific LLMs will be able to ingest, process, and make sense of data in many different formats, with the importance of syntactic structure reducing in importance, as the structure of the underlying data could be potentially abstracted just like the Babel fish. 

For example, I was recently able to get data from an image in a PDF, text in a PDF, data from notes, data from a spec sheet, and also from a paragraph together in a single unified structure for comparison.

Last week I talked with some startups doing projects in the carbon removal and carbon sequestration space. One of the technical problems they had in common was how to collect good quality data, process it, and then run models to help with the MRV process. (Measure, Record, Verify). 

The startups decided to get together and were trying to figure out if they could all agree on a common process, structure, and format for data management. The difficult part for these groups is to agree upon sharing the data.

Once they agree to share the data, as long as the formats and structures used by these different organizations are not wildly off, it would be relatively easy to use an LLM to combine data from these multiple sources, and create a consistent structure and representation.

The expertise will be in building a consistent structure and data ontology underneath the LLM, and how to train the LLM to make sure it transforms data from multiple known and unknown formats in a consistent fashion.

For most APIs, the endpoint (the place with which one connects with to exchange information with an API), operates on a strictly defined structure. Deviating from the expected format even slightly, triggers errors and causes the APIs to fail or work in unexpected ways.

Generative AI models can address this problem to a large extent, as the variety of formats across different data sources will be less of an issue, compared to the current paradigm. APIs can and will become more adaptable and resilient by harnessing the capabilities of generative AI models like ChatGPT or Gemini from Google. 

In the age of LLMs, more and more APIs will be,

1. Fast, easy, and relatively inexpensive to build and deploy.
2. The development process will be evolutionary, and the APIs themselves will be self-healing and resilient.
3. Developer portals (used by human developers to build APIs) will reduce in importance, as most APIs will be machine to machine.
4. The technical aspects of scale, security, and privacy will become more challenging, and more important.

In the long run, there will be more machine to machine APIs, and fewer APIs used by humans (hence the title of this week’s newsletter!)

My hope is that as these human APIs get replaced by machine to machine APIs, it will be technically easier to build out ecosystems across multiple systems, though the technology will not solve the human challenges of trust, cooperation, and a desire to build ecosystems.

Wish you all happy holidays!