Hybrid transactional/analytical processing

Hybrid transaction/analytical processing (HTAP) is a term created by Gartner Inc., an information technology research and advisory company, in its early 2014 research report Hybrid Transaction/Analytical Processing Will Foster Opportunities for Dramatic Business Innovation. As defined by Gartner:

Hybrid transaction/analytical processing (HTAP) is an emerging application architecture that "breaks the wall" between transaction processing and analytics. It enables more informed and "in business real time" decision making.

In more recent reports Gartner has begun referring to HTAP as "augmented transactions." Another analyst firm Forrester Research calls the same concept "Translytical" while 451 Group calls it "Hybrid operational and analytical processing" or HOAP.

Background

In the 1960s, computer use in the business sector began with payroll transactions and later included tasks in areas such as accounting and billing. At that time, users entered data, and the system processed it at a later time. Further development of instantaneous data processing, or online transaction processing (OLTP), led to widespread OLTP use in government and business-sector information systems.

Online analytical processing (OLAP) covers the analytical processing involved in creating, synthesizing, and managing data. With greater data demands among businesses, OLAP also has evolved. To meet the needs of applications, both technologies are dependent on their own systems and distinct architectures. As a result of the complexity in the information architecture and infrastructure of both OLTP and OLAP systems, data analysis is delayed.

HTAP advantages and challenges

There are various interpretations of HTAP other than Gartner's original definition; an "emerging architecture". These interpretations suggest different advantages, one being a database functionality. Recent advances in research, hardware, OLTP and OLAP capabilities, in-memory and cloud native database technologies, scalable transactional management and products enable transactional processing and analytics, or HTAP, to operate on the same database.

However, Gartner's most recent reports suggest broader advantages than a single unified database can offer. Traditional application architectures separated transactional and analytical systems. Digital business, and the need to respond to business moments, means that using "after the fact" analysis is no longer adequate. Business moments are transient opportunities that must be exploited in real time. If an organization is unable to recognize and/or respond quickly to a business moment by taking fast and well-informed decisions, then some other organization will, resulting in a missed opportunity (or a new business threat). HTAP allows advanced analytics to be run in real time on "in flight" transaction data, providing an architecture that empowers users to respond more effectively to business moments.

The main technical challenges for an HTAP database are how to be efficient both for operational (many small transactions with a high fraction of updates) and analytical workloads (large and complex queries traversing large number of rows) on the same database system and how to prevent the interference of the analytical queries over the operational workload. This kind of operational workload is also commonly referred to as Operational Analytical Processing.

HTAP solves the issue of analytic latency in several ways, including eliminating the need for multiple copies of the same data and the requirement for data to be offloaded from operational databases to data warehouses via ETL processes.

Most applications of HTAP are enabled by in-memory technologies that can process a high volume of transactions and offer features such as forecasting and simulations. New HTAP technologies use scalable transactional processing, and do not need to rely on keeping the whole database in-memory. HTAP has the potential to change the way organizations do business by offering immediate business decision-making capabilities based on live and sophisticated analytics of large volumes of data. Government and business leaders can be informed of real-time issues, outcomes, and trends that necessitate action, such as in the areas of public safety, risk management, and fraud detection.

Some challenges for HTAP include limited industry experience and skills, as well as undefined best practices.

In 2020, the first paper in the industry was published by the team of PingCAP describing the practical implementation of a distributed Hybrid Transactional/Analytical Processing (HTAP) database: TiDB: A Raft-based HTAP Database.