In his initial blog piece my longtime friend and 2-time co-founder Phillip Merrick talks about why we started pgEdge.   Here I would like to discuss some of the technical background behind our product, pgEdge Distributed PostgreSQL, and the Spock Postgres extension that makes multi-active fully distributed Postgres possible.

When Phillip called me up a year or so ago and asked me to think about how you could support distributed Postgres at or near the network edge I was pleasantly surprised (and not because I always enjoy getting a call from him :-)  That’s because along with a few members of my team we had already been working on Spock and distributed Postgres as a side project for several years after I left AWS.

Over my time at EDB, OpenSCG and then at AWS as Postgres product manager I saw the need among customers for multi-master (active-active) distributed Postgres. It’s the answer to a number of pressing database challenges that customers struggle with, like data latency, high availability, data residency and geographic scaleout (for when you hit the Postgres CPU wall).

The problem was that while multi-master is supported in the MySQL ecosystem nobody has been developing and commercially supporting multi-master Postgres, at least not in an open and standard way that is based on the standard Postgres engine. 

Also the big cloud providers have all taken their “Postgres++” offerings like Aurora in a different direction (e.g. with rewritten storage layers) that don’t easily lend themselves to logical replication, which is a prerequisite for active-active distributed databases.

Folks we spoke to in the “cone of silence” about what we are doing reacted very positively, often saying “Oh, you’re working on multi-master Postgres? That’s awesome, we all need that.  How can I help?”.  That led to us adding more great people to our engineering team, including another longtime friend Jan Wieck, who was the creator of Slony, the first logical replication solution for Postgres. (Jan also added important features to Postgres like stored procedures and referential integrity.)

As a starting point we looked at pgLogical, a Postgres extension that included some support for multi-master bi-directional replication (BDR) but was missing some important safety features, among other things. In fact the multi-master code carries warnings that it is no longer supported and should not be used. The original authors at 2ndQuadrant (now part of EDB) continued development of the BDR capability in a proprietary closed source extension and product. 

One of the first things we did with pgLogical was bring it up to date with current and recent versions of Postgres, and then went to work on making the asynchronous multi-master replication feature and conflict resolution algorithm robust and reliable.  This became the basis for Spock.  

After that work we added “Delta Conflict Avoidance”, which is our lightweight alternative to CRDTs (Conflict-free Replicated Data Types) for running sums and counters.  We put in support for replication of partitioned tables, to enable what some folk like to call “geosharding” i.e breaking up a database into separate geographically distributed pieces. And we incorporated better management and monitoring, making various operational metrics available to software operating at higher layers in the stack (e.g. our pgEdge Cloud dashboards). 

Another Spock feature allows you to identify certain columns containing PII and then ensure their values are never replicated (e.g. across a national or regional border).  This feature, along with replication of partitioned tables, provides an elegant solution to data residency.  It can also be used to more safely replicate production data into a testing database.

The design philosophy we’ve tried to adhere to with pgEdge is one Alan Kay espoused: “The simple things should be simple and the complex things should be possible.”  Things can get complicated pretty quickly with distributed databases, so we look for the simplest (yet safe) way of solving for the majority of use cases while providing mechanisms for more complex situations. Our “delta conflict avoidance” feature is an example – we avoid the complexity of a full implementation of CRDTs for the common use case of running balances.  For more advanced use cases we plan to provide user-defined functions to handle them in an application specific way.

Let me wrap up this blog post by speaking about our open source strategy. We are huge fans and supporters of open source, and we have several decades of experience contributing to and working with the PostgreSQL community.  I mentioned above that it’s our desire to approach distributed Postgres in an open and standard way.  

In an ideal world this would mean we’d make all the source code we’ve developed for pgEdge Distributed PostgreSQL available under a license like Apache or the Postgres/BSD license. However, like many open source companies we need to protect our business from major cloud providers or other competitors simply standing up our code in the cloud, offering a service that directly competes with us using code we originally developed. As a result we have adopted the pgEdge Community License, which is virtually identical to the Confluent Community License (only the company name and venue are changed).  This allows customers, or anyone else for that matter, to use the source code for any purpose, except for the one prohibited purpose of directly competing with our pgEdge Cloud offering.

We believe making our source code available to our customers and other interested parties is extremely important.   First of all, customers know they are free from vendor lock-in and can fire us if they feel so inclined. (We will be working hard to avoid that of course!)  And this is true even for customers using pgEdge Cloud, since the BYO Account model means the databases and infrastructure are all installed in cloud accounts the customer owns and controls. But just as crucially our source code is available for both customers and distributed systems researchers to examine to understand behavior and safety characteristics. This is just not possible with vendors that have proprietary distributed database solutions.

If you would like to work on the Spock source code to make it even better please reach out. We’d love to hear from you and we can talk about you becoming a contributor.

We are proud of what we are bringing to the market with pgEdge Distributed PostgreSQL and Spock.  And we are grateful to the early customers and collaborators who’ve helped us get to this exciting launch.  We look forward to working with many more customers to solve the tough database challenges that are best addressed with an open and standard multi-active distributed implementation of Postgres.  

Viva pgEdge!