It seems like this post is a response to the discussion on this previous HN post[0].
There are two main criticisms of "Event [sourcing|driven]" patterns. First, that storing a full history of events becomes a backwards-compatible schema support hellscape nightmare after 12-18months. Second, that the patterns encourage eventual consistency in areas that could really benefit from strong consistency.
This post does a good job of explaining differences between Event Sourcing and Event driven to address problem 1. But does nothing to help problem 2.
Even the event-driven example in the post uses a shopping application where multiple "OrderPlaced" events could be produced without the necessary inventory to fulfill those orders.
I'm just not convinced that most people need this kind of complexity, when a simple database with strong consistency would get a lot of folks a lot of the way there.
I recently had a brief brainstorming with someone building an AI data enrichment tool. I found myself suggesting the 'entities' in the tool would just be semi-structured json blobs; instead of normalising the data into some relational database schema.
I couldn't find good arguments against just slinging the data into a blob and getting AI Models to make sense of it.
This will work, so long as people trust the results are not (too) skewed by paid ads.
I recently used Claude and ChatGPT for exactly one of the examples; comparing different bikes to buy. They could both look up the bike specs and geometry online and tell me what the 1 degree difference in head angle or 5mm difference in reach would feel like to ride. They both did really well.
But I used them only (with cross checks) because I was fairly sure they were giving me unbiased info. As soon as the "discovery" phase of this shopping research becomes polluted with adverts, the product becomes much less useful. The same as "no one trusts online reviews anymore".
> You can't blame event sourcing for people not doing it correctly, though.
Perhaps not, but you can criticise articles like this that suggest that CQRS will solve many problems for you, without touching on _any_ of its difficulties or downsides, or the mistakes that many people end up making when implementing these systems.
Anyone who's built, run, evolved, and operated any reasonably sized event sourced system will know it's a total nightmare.
Immutable history sounds like a good idea, until you're writing code to support every event schema you ever published. And all the edge cases that inevitably creates.
CQRS sounds good, until you just want to read a value that you know has been written.
Event sourcing probably has some legitimate applications, but I'm convinced the hype around it is predominantly just excellent marketing of an inappropriate technology by folks and companies who host queueing technologies (like Kafka).
> CQRS sounds good, until you just want to read a value that you know has been written.
This is for you and the author apparently: Prating CQRS does not mean you're splitting up databases. CQRS is simply using different models for reading and writing. That's it. Nothing about different databases or projections or event sourcing.
This quote from the article is just flat out false:
> CQRS introduces eventual consistency between write and read models:
No it doesn't. Eventual consistency is a design decision made independent of using CQRS. Just because CQRS might make it easier to split, it doesn't in any way have an opinion on whether you should or not.
> by folks and companies who host queueing technologies (like Kafka).
Well that's good because Kafka isn't an event-sourcing technology and shouldn't be used as one.
Yes, I don't know where the misconception that CQRS or Event Sourcing automatically means eventual consistency comes from. We have built, run, evolved, and operated quite a few reasonably sized event sourced systems successfully, and these systems are running to this day without any major incidents. We added eventually consistent projections where performance justified it, fully aware of the implications, but kept most of the system synchronous.
Yup. It's a shame as amazing as event sourcing is it does come with complexity.
On the other hand CQRS + single writer pattern on their owncan be a massive performance win because it allows for efficient batching of views and updates. It's also much simpler to implement than a fullblown event sourcing system.
If you segregate your read and write, i.e put a (P)artition between them, and require writes to be (A)vailable when reads are down (and vice-versa), then you're either inconsistent, or eventually-consistent.
Please explain how you intend to use different models for reading and writing without there being some temporal separation between the two?
Most all CQRS designs have some read view or projection built off consuming the write side.
If this is not the case, and you're just writing your "read models" in the write path; where is the 'S' from CQRS (s for segregation). You wouldn't have a CQRS system here. You'd just be writing read optimised data.
I created a side project ~3 years ago based on a similar idea.
It was before LLMs were a big thing, and AI could render the code relationships for you.
I started with go and java (the two languages I was using in my job) and built AST parsers that would extract the relationships between the classes and functions and draw them with graphviz. Then I created a filter syntax based on regex to filter the graphs.
I never followed through on the idea to a production ready version, but it helped massively as a personal tool when working on unfamiliar codebases.
The site is still here, but lots of it is probably broken by now..
I did something similar but for non-classes based language (Go) and in 3D [1]
But I saw it as next step towards shifting programming from sitting and scanning texts into something more tangible, where developer has broad overview of software, and can work on smaller parts while seeing context of how these parts are connected. Ended up concluding that this stuff should work in VR.
This reads like a bit of a smell. I'd be pretty suspicious of why you have tasks and reminders in the first place. The question also reads like the volume of these tasks and reminders is large enough to be a problem.
You should check out the "toil" section of the Google SRE book
Looks like it uses Postgres Logical replication to share changes made on one postgres instance to another. Conflict resolution is last-write-wins based on timestamp. Conflicting transactions are logged to a special table (pgactive_conflict_history), so you can see the history, resolve, etc.
Sounds like "yes, with an if" where the "if" is "if you don't really care about data consistency".
"Last write wins" sounds like a recipe for disaster IMO.
This is still one of those things that keeps people on MySQL - there are not one, but two open-source solutions available that provide synchronous cluster replication, allowing for "safe" writes against multiple primaries.
Out of curiosity, what conflict resolution options exist in mysql and/or mysql cluster (never checked / exp. in PG)? Because you'll always have to address conflicts of course - we come to CAP / PACELC. Hm [1][2] - looks like they support more strategies (possibly) but I mean none of them are somehow magical, and timestamp comparison based methods comprise the better part of offered strategy set (looks like?) - and "latest timestamp wins" at least used to be the default (did not read thoroughly mind you, was just curious)?
But I could be totally wrong - (1) curious if someone could link to things / explain, and (2) fyi ('stephenr) last write wins based on timestamp is a thing im mysql world as well (though again maybe set of options / different conflict resolution methods available is larger in mysql?)
For reference those two pages are both about NDB cluster.
The two "options" I was referring to are MySQL group replication and the Galera replication plugin for MySQL. Both provide synchronous replication, so the write either succeeds to a majority of the cluster or is rejected.
Did Postgres ever get a built-in, blessed replication offering? It's been a while since I set it up, but I remember this was always a big missing feature compared to Mysql.
The basic replication mechanisms have been built-in for quite a while. What’s not there is cluster management (replica launching, leader election, load balancing, that sort of thing) that makes it practical in a nontrivial situation. There are several 3rd party solutions to that. [0]
Same situation as, e.g., backups. You can just use pg_dump, but to be serious you need a 3rd party solution that does log shipping and so on.
It's eventual consistency: Latest-write wins after the dust settles.
As I understand it, this is a wrapper on top of Postgres' native logical replication features. Writes are committed locally and then published via a replication slot to subscriber nodes. You have ACID guarantees locally, but not across the entire distributed system.
It all feels like they expect developers to sift through the conflict log to resolve things manually or something. If a transaction did not go through on some of the nodes, what are the others doing then? What if they can not roll it back safely?
Typically applications will have some kind of logical separation of the data.
Given this is targeted at replication of postgres nodes, perhaps the nodes are deployed across different regions of the globe.
By using active-active replication, all the participating nodes are capable of accepting writes, which simplifies the deployment and querying of postgres (you can read and write to your region-local postgres node).
Now that doesn't mean that all the reads and writes will be on conflicting data. Take the regional example, perhaps the majority of the writes affecting one region's data are made _in that region_. In this case, the region local postgres would be performing all the conflict resolution locally, and sharing the updates with the other nodes.
The reason this simplifies things, is that you can treat all your postgres connections as-if they are just a single postgres. Writes are fast, because they are accepted in the local region, and reads are replicated without you having to have a dedicated read-replica.
Ofc you're still going to have to design around the conflict resolution (i.e. writes for the same data issued against different instances), and the possibility of stale reads as the data is replicated cross-node. But for some applications, this design might be a significant benefit, even with the extra things you need to do.
I think I understand the use case. Like, we have in fact several regional Postgreses, but we want them to be one physical database for the sake of simplicity.
Probably this should be in the motivational part of the README.
There’s no free lunch. The rabbit hole is only worth going down if the benefits are worth the operational pain. I view this as a building block, not a checkbox feature that magically just works all the time.
For someone who has these requirements out of the gate, another datastore might be better. But if someone is already deeply tied to Postgres and perhaps doing their own half assed version of this, this option could be great.
What are good off-the-shelf distributed databases? We looked at MongoDB but it wasn't worth giving up SQL. To reiterate the no free lunch point, no one has figured out how to outsmart the CAP theorem yet, so all you can do is design around it.
I work for them so take with a pinch of salt, but Oracle DB. It gives you a fully multi-master horizontally scalable database with ACID transactions (not sharding), full SQL, elastic scalability, built in queues, JavaScript stored procs, automatic REST API construction, many other features. Its pricing is competitive with a cloud hosted Postgres, believe it or not (the clouds are making a lot of money off customers who are wedded to Postgres). I work through some of the numbers for an extreme case here [1].
Behind the scenes, the way it works is by combining software tricks with special hardware. You rent a (part of a) database cluster. The cluster is running on high end hardware running customized kernels, with a private Infiniband RDMA-capable interconnect between the nodes separate from the front-side network that clients connect with. A lock manager coordinates ownership of data blocks, which can be read either from disk nodes or directly out of the RAM of other database nodes. So if one node reads a block then writes to it, the only thing written to disk immediately is the transaction log. If another node then needs to write to that block, it's transferred directly over the interconnect using RDMA to avoid waiting on the remote CPU, the disk is never touched. Dirty blocks are written back to disk asynchronously. The current transaction counter is also poked directly into remote nodes via RDMA.
In the latest versions the storage nodes can also do some parts of query processing using predicate push-down, so the amount of data to be transferred over the interconnect is also lowered. The client drivers understand all the horizontal scalability stuff and can failover between nodes transparently, so the whole setup is HA. A node can die and the cluster will continue, including open transactions.
If you need to accelerate performance further you can add read-through coherent cache nodes. These act as proxies and integrate with the block ownership system to do processing locally.
Other than financial reasons (I own some stock), I've started making this argument here on HN because it's unintuitive but correct, which is just enjoyable. A lot of people in the startup world don't realize any of the above, thinking that horizontally scalable fully coherent SQL databases either don't exist or have severe caveats. E.g. one reply to you suggests FoundationDB which is great, but it's a KV store and not a SQL database.
By the end of the day it’s not black or white there are trade offs. So special hardware is simply a no go zone for me. What happens if you want to leave the cloud and host on premises its the activation of the lock-in mechanism. Thanks I can manage one solution or another using one of the open source technologies. It’s all trade offs.
You don't have to use the special hardware, it's just faster and easier if you do. And you the customized hardware deal pre-dates the cloud: you can buy the pre-built racks and have them wheeled into a self-hosted datacenter if you want. That's called ExaData.
But if you want to run the DB on your own VMs or bare metal, you can do that. It doesn't have any DRM so from time to time you'll be expected to run some scripts that check your usage and reports back, to ensure you've been paying for what you use. But otherwise it's operationally no different to an open source DB.
The open source aspect makes a difference in terms of who you pay for support (if anyone), what quality of support you get, things like that.
Well, no moreso than for any other RDBMS. It implements standard SQL, exporting your data is easy. The only lockin comes from using features that other databases don't have, but that tradeoff exists with any kind of software including open source.
My guess is that you want to change your entire design philosophy a little bit with regards to table design, moving some entities to use a composite GUID+timestamp as PK's and replace most updates with inserts to avoid conflicts and instead resolve things at query-time (Basically a CRDT modelling philosophy contained within a relational schema).
Ideal? Not entirely but it should still give most query benefits of regular SQL and allows one to to benefit from good indexes (the proper indexes of an SQL database will also help contain the costs of an updated datamodel).
I think this is more interesting for someone building something social media like perhaps rather than anything involving accounting.
Are there any Datomic-like query layers on top of Postgres for approaches like this where you're recording immutable occurrences rather than updating mutable records?
No, this would be more be about using different ways of mapping data to the database from an applications point of view. So for an developer it would increase the load a tad in terms of exploring data when debugging simple things.
On the other hand, the increase in exploration costs should be more than offset by having most data changes logged to be able to track changes.
In our case, we're designing around INSERT-only tables with a composite primary key that includes the site id, so (in theory) there will never be any conflicts that need resolution.
I think that's absolutely true in the happy scenario when the internet is up.
There's a requirement that during outages each site continue operating independently and might* need to make writes to data "outside" its normal partition. By having active-active replication the hope is that the whole thing recovers "automatically" (famous last words) to a consistent state once the network comes back.
But if you drop the assumption that each site only writes rows prefixed with its site ID, then you're right back to the original situation where writes can be silently overwritten.
Not silently overwritten: the collision is visible to the application layer once connectivity is restored and you can prompt humans to reconcile it if need be.
That's correct: when the network comes back up we'll present users with a diff view and they can reconcile manually or decide to drop the revision they don't care about.
We're expecting this to be a rare occurrence (during partition, user at site A needs to modify data sourced from B). It doesn't have to be trivially easy for us to recover from, only possible.
One way we dealt with this in the past was assigning an "affinity" to each tenant and basically routing their writes/reads to that host, excepting if that host was down.
You would still get weird replication issues/conflicts when requests failed over in some conditions, but it worked fairly well the majority of the time.
These days I'd stick to single primary/writer as much as possible though tbh.
In principle you could use CRDTs to end up with a "not quite random" outcome that simply takes the conflict into account - it doesn't really attempt to "resolve" it. That's quite good for some cases.
This is a kind of CRDT. CRDT is just some papers defining reasonably clear terminology to cover the kind of eventually consistent replication that has been done for decades, including this kind (timestamp-based last-writer wins).
I'd agree. There's so many footguns involved in multi-master setups, that most organisations should avoid this until they're big enough to hire distributed systems engineers to design a proper solution for the company. I personally don't love any of the Postgres multi-master solutions.
You can scale surprisingly far on a single-master Postgres with read replicas.
IANAL but, you should be in contact with your airline about your specific flight.
You are entitled to be re-booked on the next available flight or get a refund. If you take a refund the airline has no obligation to you anymore. You might find after taking a refund, the price of an equivalent flight is now much more.
I know - but I'm going to a concert in London, if the flight isn't going at its original time then there's no point going. I'll just take a refund and drive instead, but so far it looks like flights will operate as normal tomorrow.
There are two main criticisms of "Event [sourcing|driven]" patterns. First, that storing a full history of events becomes a backwards-compatible schema support hellscape nightmare after 12-18months. Second, that the patterns encourage eventual consistency in areas that could really benefit from strong consistency.
This post does a good job of explaining differences between Event Sourcing and Event driven to address problem 1. But does nothing to help problem 2.
Even the event-driven example in the post uses a shopping application where multiple "OrderPlaced" events could be produced without the necessary inventory to fulfill those orders.
I'm just not convinced that most people need this kind of complexity, when a simple database with strong consistency would get a lot of folks a lot of the way there.
[0]: https://news.ycombinator.com/item?id=45962656