Scaling AI Inference at the Edge

In the rapidly evolving landscape of artificial intelligence (AI), the demand for real-time data processing has never been more critical. Traditional cloud-based AI inference often introduces latency by transmitting data to centralized servers for analysis. In a distributed world, a centralized inference pipeline becomes a bottleneck. This delay can be detrimental in applications requiring immediate responses, such as autonomous vehicles, production line monitoring or real-time analytics. This challenge can now be addressed by moving AI inference closer to where the inference is used, distributing AI across your network to local devices near the data source. This approach significantly reduces latency, enhances data security, and improves overall system efficiency.

However, implementing AI inference at the edge presents its own set of challenges, particularly concerning data availability and consistency across distributed environments. This is where pgEdge Distributed PostgreSQL becomes indispensable. By providing a distributed PostgreSQL architecture optimized for the network edge, pgEdge ensures that data remains consistently accessible across multiple nodes, even in the face of network issues or hardware failures. This multi-master (active-active) setup guarantees high availability, lowers response times, and facilitates seamless data replication and synchronization, which are crucial for maintaining the integrity of AI models and their inferences.

Key Benefits of pgEdge for AI Applications

Ultra-High Availability

pgEdge's multi-master (active-active) architecture ensures that read and write operations can occur at any node within a geographically distributed cluster. This design eliminates single points of failure, providing continuous data availability even during maintenance or unexpected outages. This resilience is crucial for AI applications that demand uninterrupted access to data for real-time processing and decision-making.

Distributed Processing

By enabling data to be stored and processed across multiple locations, pgEdge facilitates distributed AI workloads. This allows for parallel processing of large datasets, enhancing the efficiency of tasks such as training machine learning models or executing complex inference algorithms. For instance, in a three-node cluster managing a 900,000-row table, each node can process 300,000 rows concurrently, significantly reducing overall processing time, with the resulting data automatically distributed across all nodes without needing to repeat the computations.  This can be especially valuable adding and maintaining embeddings for large data sets with constant and geographically distributed read/write activity.

Data Consistency Across Nodes

pgEdge employs advanced replication and conflict resolution mechanisms to maintain data consistency across all nodes in an active-active Multi-Master configuration. This ensures that AI models operate on accurate and up-to-date information, which is essential for generating reliable predictions and insights. The platform's support for synchronous read replicas within regions further enhances data integrity, making it a dependable choice for mission-critical AI applications.

Flexibility in Deployment

pgEdge's architecture supports deployment across various cloud regions and data centers, as well as on-premise or in air-gapped deployments. This unparalleled flexibility and resilience is particularly beneficial for AI applications that require scalability and adaptability to different operational environments. By integrating pgEdge into their AI infrastructure, organizations can effectively overcome the data limitations associated with centralized AI inference, thereby achieving faster decision-making processes and enhanced user experiences.

Considerations for Distributed AI Compute

While an ultra-high availability, multi-master distributed data environment is an essential foundation of a distributed AI inference implementation, it's only half of the story.  The AI Compute itself also needs to be distributed to realize the full benefits.

When a distributed database is integrated with a single, centralized AI compute environment, you can still encounter latency, as all nodes are required to send data to and await responses from the centralized compute resource. This bottleneck undermines the key advantages of a distributed database.

Implementing Localized AI Compute Instances

To mitigate this issue, you can deploy a localized AI compute instance in proximity to each database node. This approach ensures that AI processing, such as vector generation, occurs locally, thereby minimizing latency and reducing the need for data transmission over potentially congested networks. By processing data closer to its source, your system can achieve faster inference times and improved overall performance.

Parallel Processing Through Distributed AI Compute

Distributing AI compute resources across multiple nodes not only alleviates centralized bottlenecks but also enables parallel processing of large datasets. For instance, a smart city project can process sensor data (e.g., traffic, weather, or transport) at nodes near data sources, enabling real-time decisions like rerouting traffic or adjusting bus schedules, with global synchronization of critical updates. This parallelism accelerates data processing and leverages the inherent scalability of distributed systems, leading to more efficient AI workflows.  New data received at one node can be vectorized locally for immediate use by the local application, with other nodes receiving the new or updated embeddings through logical replication.

Integrating AI Compute with Databases

The goal of course is to get the Compute as close to the data, at the point of usage, as possible.  The question is just how close do you want to get?

In-Database Processing: Integrating AI capabilities directly within the database using extensions like PostgresML (PGML) allows for execution of machine learning tasks without the need for an external compute system. This tight integration reduces data movement and can enhance performance for certain workloads, but there are quite a few caveats with this approach.  Extensions like PGML are usually limited to running on very specific OS flavours, and can come with heavy Python library and version dependencies.  The additional AI workload will have a direct impact on hardware capability considerations for the database instance, especially if you intend to use GPU acceleration, as this will also require NVIDIA CUDA libraries.

In a distributed environment, other limitations are likely to emerge.  For example, in our local testing and experimentation with PGML, we found that it was using Primary Key Sequences in its model storage. With active-active replication, this will cause duplicate Primary Key conflicts on insert-insert scenarios (where two nodes insert a new row with the same Primary Key value).  To make PGML function in a distributed environment, pgEdge patches it to use our snowflake sequence extension, so that if a HuggingFace model was first used on one node, after download, it would replicate successfully to the other nodes. PGML also caches the model as part of its usage, making it necessary to invalidate the cache on the receiver node so that it would be rebuilt and function correctly.

Sidecar Deployments: Implementing AI models as sidecar services using frameworks such as ONNX or OLLAMA enables AI processing to occur alongside the database. This configuration offers flexibility, allowing for the use of specialized hardware or software environments tailored to AI tasks while maintaining close proximity to the database. This can be seen in extensions such as localAI (which uses ONNX), pg_vectorize, and TimescaleDB’s PGAI (both of whom use OLLAMA), that support remote calls to openAI or calls to a local framework/API. Maintaining a close proximity with the Compute environment also enables the ability to add asynchronous processes to update vectors when the underlying information is added or updated, without having to use triggers to invoke and wait for the return of the updated embedding.

Practical Use Cases

This post provides a rather high level overview, but let's end with a few practical use cases:

Accelerating Vector Search

Vector search is pivotal in AI applications, enabling similarity comparisons essential for recommendation systems and semantic search. pgEdge has integrated the pgvector extension, providing efficient storage and querying of vector embeddings directly within a distributed PostgreSQL database. This integration facilitates low-latency, distributed access to embeddings, ensuring that AI-powered search operations are both swift and scalable.

Parallelizing Vectorization of Large Datasets

Handling large datasets is a common challenge in AI workflows. pgEdge's distributed architecture enables an accelerated parallel vectorization of data across multiple nodes. For example, a global e-commerce platform can use a multi-master database to process transaction logs locally on regional nodes, identifying fraud or issues in real-time, while replicating key insights across the cluster for global visibility.

Real-Time Updates to AI Models and Embeddings

In AI applications, especially those involving real-time data processing, the ability to update models and embeddings promptly is crucial. pgEdge's multi-master replication ensures that updates made to AI models or vector embeddings on any node are propagated across the entire cluster in near real-time. This capability guarantees that all nodes operate with the most current data, enhancing the accuracy and reliability of AI-driven insights.

Enhancing AI Inference at the Edge

Deploying AI inference closer to end-users reduces latency and improves responsiveness. pgEdge's support for the pgvector extension allows AI inference and similarity search requests to be processed nearer to users, delivering faster search results regardless of their location.

Implementing Edge AI for Real-Time Analytics

Edge AI enables real-time data processing and analysis without constant reliance on cloud infrastructure. By bringing computation closer to the source of data, edge AI reduces latency, optimizes bandwidth usage, and enables faster decision-making.