TIL

My New Reality of Software Engineering

Mon, 02 Mar 2026 18:00:00 +0000

My New Reality of Software Engineering

Software engineering as a profession is going through an unprecedented phase shift, not just another tool cycle. In the past month I have written almost no code by hand and still shipped more than ever, because agents now handle large parts of the implementation loop under my direction. That sentence would have sounded like breathless hype even a year ago. It does not now. The models and the tools have crossed a threshold where they are good enough, fast enough, and cheap enough to materially change how teams work and what they can deliver.

You do not have to like every part of this, and you can still have serious concerns about cost, quality, jobs, or broader social and economic impact. I do too. But this shift is happening whether we are emotionally ready for it or not, and the economics will keep pushing it forward. The practical question is no longer “is this real?” but “how quickly can I learn to use it well?” The people and teams who invest now will have a major advantage in output, scope, and speed over the next few years. There is real value in these technologies for you now, in a way that wasn’t really true before.

A Career That Suddenly Changed

I have been a software engineer for more than twenty years. Looking back, it’s amazing how many things have changed in that time, and how many things have not. But what has happened in the last year, especially the last six months, has created a fundamental shift in how I work, how I think, how much I produce, and really what it even means to be a software engineer.

I was lucky (I think) to have always known that I wanted to write software as a career. I knew I wanted to work with computers from the moment I heard about them, before I ever actually touched one. When I was around six, my parents got me a VTech PreComputer 1000, a simple device that had math and geography quizzes, a very non-standard terrible keyboard, and a single 20-character dot matrix LCD display. But it had a mode where you could write BASIC, and a very small instruction manual. Writing line numbers when you could only see a single line at a time was quite the challenge, but I spent hours writing programs in that thing. Worse yet, it did not have any non-volatile storage, so if you turned the power off, your program was gone. That did not stop me from writing programs over and over, debugging, improving, and just playing with it non-stop. I made monitors, cases, and keyboards out of cardboard boxes and markers, just to pretend. When I saw my first “real” PC at my dad’s work I was hooked. This was the future, and I wanted to be a part of it.

The idea that I could not just use but control these things was so empowering and enchanting for me. Like actual magic spells that brought my whims to reality. And there was always this intertwined tension between wanting to learn the languages and patterns and efficient ways to write the code, and the actual production of the thing I was trying to build so I could show other people how cool it was. Both brought me so much joy.

And it has been that way for my entire career. I have always loved the actual writing of the code, the sound of my hands on the keyboard, the diffs in version control, but most importantly the problem-solving challenge and the problems I was solving for my users and customers. I was making lives easier, bringing joy to other people, or making them and myself money. But it always required me to do all of the work: understanding the problem, challenge, or pain point; coming up with a solution; making that solution a reality; debugging and maintaining existing code; building; testing; iterating; showing it to other people; and getting it into production.

I was at my current job when ChatGPT was first released. It was such a novelty at first, then we started getting scared that our jobs were over. Then we settled into the reality that while these things were cool, they had major limitations. They hallucinated. They could not keep much information in context. There was no way they were going to be able to handle our hundred-thousand-line codebases. We were thinking about their knowledge cutoffs, figuring out how and if they might be useful, and some of us were imagining a future where they could do so much more, but that was not the current reality and was not near term.

And slowly but surely things improved. We got new models, better tools. When I started at my company we had lots of challenges where we needed to do things at a scale that would mean hiring dozens if not hundreds of people if we wanted to do it fully and repeatedly. There was no way we could hire that many people, much less manage them. But we started building toward that future anyway, thinking that one day it might be possible.

Then Claude Code came out. The climate started to change. People started saying it could do so much more. I decided to start using it a lot more, letting it write the code maybe 75% of the time. I was still using things like Cursor’s tab completion and writing large blocks of code myself, but I slowly gave it more and more. I was working on large-scale systems, some from scratch, some existing. Large, architectural, ambitious projects, the kinds of things I love and am really known for. And it was handling it all.

And then, in the last month we got Opus 4.6 and Codex 5.3. These two models specifically have changed the game. They are more thorough, make fewer mistakes, go for longer - they are just all-around capable. A memo went out to everyone that we should not be writing most of the code anymore, we should be using the agents. They are smart enough, fast enough, thorough enough. Good enough. I was relieved; I had already decided to try and go all in.

And now I have not written more than a dozen lines of code by hand in the last month. Codex writes my code, helps me plan, manages git, creates my commits, pushes my PRs, gets second opinions, responds to review comments and addresses them, manages my stories, just about everything, just with my control and guidance. I am shipping more than ever, getting more done than ever, while writing virtually no code to do it. I am talking to an agent like I would talk to a colleague in Slack. I am writing in languages that I am not nearly as fluent in, but it does not matter. I am still shipping quality code that does the job as well as if written by an expert in those languages. I’ve shipped over 30 PRs in the 20 working days of February, across four major systems. Features, tests, documentation, bugfixes. It’s a whole new world.

The New Reality

In some ways, this world has changed so fast, and in other ways it felt inevitable. I told someone a year ago that before long it will be like we are all engineering managers with full teams of employees. That is really the way I feel now. Across my career we learned new frameworks, new languages, and new patterns, but the fundamental shape of the job stayed mostly the same. This time is different.

The reality is not evenly distributed yet. At my job, and in my social feed, we are in a bubble where people are pushing these tools hard and adjusting their workflows around them. At the same time, there are many engineers still working mostly the way they worked last year, and the twenty years before that. I am not saying everyone has to move at once, but I do think everyone will have to confront this shift. Within five years the landscape will look completely different. Expectations for output, speed, and scope are going to move. Teams that adopt these tools well will compound output, and that will gradually reset what “normal” looks like across the industry.

Part of this reminds me of when source control was becoming mainstream. I remember having to proselytize why it was good, a necessary way of working that was worth learning and adopting. I remember people resisting at first because it felt awkward and slow. We take it for granted now, but back then many teams still copied files by hand before making changes. But with version control, nothing fundamental about the job changed. This new way of working is much more revolutionary.

“Revolutionary” is an interesting and apt word here. In tech, we usually use that word in completely positive terms. Who does not want a revolutionary new product? But actual revolutions are messy and disruptive. There are winners and losers, and there are people very happy with the status quo who will resist as long as they can. This should not be a holy war, and I do not think in moral binaries about it. But I do believe this shift is inevitable. I might be wrong about the exact timeline, but I do not think I am wrong about the direction. We need to start mentally preparing ourselves for what comes next because the economics alone will force the shift.

There are also new problems we have to solve. When the execution cost drops, teams take on more, and the bottlenecks move. PRs get larger. Review burden increases. Product and design can become the new constraint. The mental split between writing code and supervising/reviewing code gets sharper, and that cognitive load is real. The context switching is easier than it was before, but it still takes its toll.

I do not want to give the wrong impression: this is not utopia. Agents make mistakes. Quality still depends on tests, review discipline, and engineering judgment. There may be less demand for software engineers in the future. I am concerned about what this means for junior pathways, for hiring, and for the broader economics of software. Businesses themselves will be disrupted by these changes because the economics are changing, and expectations about software are changing with them. I don’t personally believe this is the death of SaaS, but I expect the nature of it to change significantly.

You may be revolting against this movement because you don’t think AI in general or LLMs in software engineering are actually positive for society. You may be concerned about the energy and water use, the way the data centers and graphics cards and model training are consuming the economy. You may be worried that by participating in it you’ll be somehow culpable if this isn’t the future you want. I understand those viewpoints, and may even agree with you, but for what this means for our reality as software engineers today I just don’t think those things matter. This is the future, and it is the present. We are here.

A concrete signal: Block laid off 40% of their workforce while explicitly tying that decision to AI-driven changes in how work gets done. I do not think this stays inside tooling conversations much longer. It is going to become a hiring standard. In my next interviews, one of my first questions will be inference budget and agent tooling flexibility, and I expect my fluency with these systems to be one of their first questions for me.

I want to tell you that it’s okay if you haven’t tried these things yet, or are still using agents in an IDE or tab completion, that’s fine. If you’ve thought that this stuff is changing too fast and have wanted some dust to settle before investing in it, that’s fair. If you already have a full-time job and learning all of this feels like another full-time job, I get it. If you’ve tried these agents in the past and were disappointed, I understand. But it’s time to try again. Things have changed. They’re going to continue to change, but they’re truly good enough to really put to work and a worthwhile investment of your time at this point. If you haven’t gotten started with these technologies yet you aren’t late - you are right on time. But no matter where you are today, there can be a real emotional cost.

Mourning the Loss

You may be somewhere in the five stages of grief. Realize that one day you’re going to get to acceptance, but that may take time. It is completely normal if you’re still on an earlier stage.

If you are a software engineer because you like typing on the keyboard, writing the actual code, and thinking about the control flow of the program through each successive line in a file, this sucks. It does. I certainly think I am one of those people. I have softened my attachment to the actual code over the last decade of working in startups where the usefulness of the thing I write might not last long as we try to find the elusive product market fit, but I have always cared deeply about not just the outcome but also the formatting, organization, and elegance of the code I produced. I tried to leave code better than I found it with every PR. I cared deeply about names, had debates about comments, fought in tabs-vs-spaces wars. I had opinions on the aesthetics of files. I am not happy about leaving that behind - it certainly felt like it mattered at the time. I worry that my ability to read and write code will languish over time, and that somehow that will affect my ability to reason about software itself.

But if you care about solving problems and shipping, this might be the best thing that has ever happened. In so many jobs in my career I advocated for spending time on our tech-debt column in the project management system. There were so many things we wanted to do but could not find the time for. There was never enough time, never enough people to get everything done. We needed to ship the new feature, fix the priority bug, and do the next thing. Going back to maintain old code, or make that improvement that wasn’t in the critical path, was something you only got to do if you were lucky or you snuck it in. We can do those things now. Most of the time those things were not difficult. They did not need a genius to implement; they just required time we did not have.

Now you can give your intent and constraints and opinions to an agent that can do those things for you, including a lot of the bookkeeping around the change that used to make it not worth it. How many times have you looked at what would be a change of only a few lines and chosen not to do it because those few lines were actually hours of work when you considered making the story, creating the branch, writing the tests, creating the PR, asking for reviews, iterating on review comments, and deploying it? Those things are materially easier now.

At the end of the day, we are paid for the things we ship. Shipping has to be the goal. It has been great for the past twenty years that I have been paid to do this thing I love because I was shipping value to customers, but it was the value I was paid for, not the symbols I wrote. But I will mourn the loss of artisanal hand-written code.

When I am really honest with myself, though, I am happiest when I am productive. That is such a hard word to define, almost an emotion. But when I go to bed at night, I feel best when I feel successful at getting something new out into the world, a desire made into reality. The mechanics of doing that gave me pleasure, but that was never the thing that mattered most. What matters is shipping.

My Suggestions for How to Get Started

You may not know how to get started and the options may be overwhelming. If you are curious but still uncomfortable, the answer is staged adoption. Ask the agent to do one contained change, something that you already know exactly how you would do it. Review the diff. Run your tests. See what it got right, what it missed, and where your instructions were vague. Then try again. When you get more comfortable, push it further, give it a little more. Just start talking to it. I think you’ll be surprised.

The biggest unlock for me was moving from build mode to explicit planning. Before any change, have the agent produce a plan first. Then force it to ask clarifying questions until there is no ambiguity. I often tell it to be relentless about asking questions, questioning assumptions, and raising edge cases. Give it your plan and ask it to elaborate on it and challenge it.

Trust builds from repetition. At first, keep your hand on the wheel for anything that touches external systems: deploys, migrations, production settings, billing surfaces. Over time as results become predictable you can delegate more of the full loop: branch setup, implementation, tests, commits, PR creation, review response, and cleanup. The goal is not blind trust. The goal is reliable delegation with verification.

If you only take one practical step this week, make it this: install opencode, launch it in your current project, leave everything as the default, and just talk to it. Put it in plan mode and tell it what you are about to do and see what it thinks. Have it describe to you how a subsystem works. Ask it to find a bug. Just start talking to it and I think you will be surprised. Then, when you get a little more comfortable, get an OpenAI subscription and switch the model to gpt-5.3-codex and ask it to do the thing you were about to do.

But do not have brand loyalty in any way. Try and choose tools not produced by a single LLM provider (the main reason I like opencode right now). Don’t fall in love and get attached to any one model. Don’t dive too deep into integrations with any one software because a better one will come out tomorrow. This is hard in a lot of ways because it feels like you’re never going to “catch up” much less get ahead, but that’s okay. Part of this new reality is being prepared to learn new things all the time. We are so early. Everything is going to keep getting better. We are still figuring out what these things can do and how to work with them every day. If you can be part of that research that’s awesome, but you don’t have to blaze new trails, use these tools for the value you can get out of them and then replace them as soon as something better comes along.

You have to change your mindset and judge this workflow by shipping outcomes, not by how much code you personally typed. I still care about code quality and design choices, but I spend less energy on the bookkeeping around changes. That lowers the activation energy for all the little improvements we used to postpone forever. Tech debt gets burned down. Side quests that never seemed worth the overhead actually get done. Use it to be curious - those ideas you’ve had but it would take too long to write out and try. Create a branch, tell it what you’re thinking, and see what happens. Who cares if you have to throw it away - you will have learned something.

A normal day for me now is usually one core project I am driving end to end, plus smaller improvements running in parallel that would have sat in the backlog before. While I am iterating quickly on the main thread, an agent can be tightening tests in one area, fixing a flaky edge case in another, or preparing a cleanup PR for a naming inconsistency we have ignored for months. None of those tasks are individually hard. They were just never worth the setup cost before. I have the agents read my logs and find problems or opportunities for improvement. I have it check my understanding of subsystems or database tables and where they are used. I treat it like a coworker who already knows all the answers or knows where to find out faster than me.

I’ll give you an example. Recently my main goal at work has been creating a large system using agents to process and make sense of our data. I am spending the majority of my effort on this, more than enough for a full-time job. Along the way I am seeing bugs or opportunities to improve the UI in other systems, so I fire up another session and tell it what I want and let it cook in the background. Later when I get a break I go and check on its progress, look at the changes and try it out, and tell it to iterate or to cut a detailed PR. I’m shipping several of these side quests a week, things I never would have had time for.

The agents will make mistakes. You’ll have to be diligent to catch them. Tests and code reviews and actual QA on your part continue to be as important as ever. The agent might write a few thousand lines of code and you’ll realize that it’s gone off the rails and you’ll have to throw it all away and try again. Frequent commits will help. You’ll have situations where you weren’t specific enough or the agent misunderstands you and it goes and does the wrong thing. You’ll tell it to fix a failing unit test and it will delete the unit test (“fixed the glitch” office space style). These are all things you’ll have to navigate and get comfortable with and learn how to avoid. But - it’s worth it, and it’s time to learn these skills.

Do not expect that the agents will feel amazing when you first start. It takes time to learn how to prompt them the same way it took time to learn to code. You can write subpar code and get results and you can write better code and get better results. Prompting is the same way. It takes practice and experience, and the only way to truly learn is by doing.

You do not have to stop being an engineer to use this model. You still own the problem framing, architecture decisions, constraints, and quality bar. You are still accountable for what ships. You still need to read every line of code it produces. The difference is that execution becomes collaborative and parallel. In practice, that means you can do more at once, if your team can absorb the review load and process throughput that comes with it.

There Is So Much More to Say

Maybe you are worried about the costs, maybe you want more opinions on which LLMs to use, which tools to use, and just more examples of what you can do with it and how to fit it into your teams and your processes and your day-to-day. I hope to write more about these things soon (reach out on Twitter if you would be interested). Don’t believe everything you see on social media about the topic, but do believe some of it. The signal-to-noise ratio is terrible right now, there are lots of voices trying to make money and drive public opinion. Ignore most of them, but when you start to see themes emerge, pay attention. Remember that not everyone is working on the same kinds of things. Devs building frontends and webapps don’t have the same needs as those building operating systems, and so their experiences with the agents won’t be the same. Don’t assume that someone else’s experience applies to you.

If there is one takeaway from all of this, it is that the leverage is real right now, and the people who learn to direct it well will shape what software engineering looks like next.

I’m going to leave you with some tweets and articles from the past couple weeks that I think help make my points. I’d love to hear what you think - if you agree or disagree.

the truth is no matter how hard you try you’ll never be able to keep up with 100% of what’s going on in AI right now

there’s just too much action right now
— GREG ISENBERG (@gregisenberg) February 26, 2026

It is hard to communicate how much programming has changed due to AI in the last 2 months: not gradually and over time in the "progress as usual" way, but specifically this last December. There are a number of asterisks but imo coding agents basically didn’t work before December…
— Andrej Karpathy (@karpathy) February 25, 2026

This matches my experience too. Around last November, Agentic Opus 4.5 with Claude Code was probably the biggest step-change since GPT-4. Now Opus 4.6 is somehow even better again.

It’s honestly bananas.

There is a huge tacit knowledge factor in supervision though which is… https://t.co/Z64GLMaFIO
— Machine Learning Street Talk (@MLStreetTalk) February 26, 2026

I empathize with this a lot, feel so torn between “AI is incredible and I’m having so much fun” and a nostalgia for the way things were. I don’t fault anyone for wanting to take a break. It’s so hard for me to put the genie back in the bottle, personally, bc I do think I can move… https://t.co/D28p362WRF
— Adam (@adamdotdev) February 26, 2026

i've spoken to a dozen people this month who have deep anxiety to get ai-native.

they are expected to be ai-native and they want to get ai-native, but they feel like it's a full-time job to do so on top of their already full-time job. https://t.co/qB9kDAedUk
— Alex Lieberman (@businessbarista) February 25, 2026

Reflecting that at a personal level I feel like I'm firing on all cylinders in terms of day-to-day productivity and impact.

I have never felt so capable of spinning so many plates at the same time without much failure. AI tools have redefined what my baseline is... by a mile.
— Dmitry Lyalin (@LyalinDotCom) February 25, 2026

Something that I think goes under-emphasized is how much AI coding demands a 'lead dev' mentality.

If you spent your pre-AI career trying to level up your teammates

(through API design, feedback loops, architecture)

Then working with AI will feel natural.

If you only focused…
— Matt Pocock (@mattpocockuk) February 24, 2026

my job has gotten at least 2x more difficult in all of the “this makes my brain hurt” sorts of ways and 10x easier in all of the “digging ditches” sorts of ways
— jeffypoo (@grepmoney) February 22, 2026

It’s easier than ever to write code…

And yet the hard parts of software engineering are still *very hard*

Don’t get discouraged by the hype. There’s still so much to learn and build.
— Lee Robinson (@leerob) February 22, 2026

AI lowered the cost of typing code. It didn’t lower the cost of thinking.

Architecture, trade-offs, debugging distributed systems, understanding users, those are still hard. The leverage increased. The responsibility did too.

The craft isn’t dying. It’s just moving up a level.
— Ashok Sahoo (@ashoKumar89) February 22, 2026

your timeline convinced you AI is in a bubble. talk to a boomer above the age 35 for 5 minutes.

most people don’t even know what claude is.

kind of wild when you zoom out. pic.twitter.com/fCeqxaUnpk
— Damian Player (@damianplayer) February 21, 2026

it's so weird watching people continuously pledge allegiance to multi billion dollar corporations and then experience heart break

these are just things you use when they're useful save your heart for real people
— dax (@thdxr) February 19, 2026

Nearly every ambitious person I know who has dived into AI is working harder than ever, and longer hours than ever.

Fascinating dynamic tbh.

I have NEVER worked this hard, nor had this much fun with work.
— Nat Eliason (@nateliason) February 6, 2026

Bulk insert into postgres using a single parameter

Tue, 24 Jun 2025 12:00:00 +0000

motivation

At my job recently we had a need to bulk insert lots of records quickly and efficiently, but we had some limitations we needed to work around. We have been using Prisma for a while, which has a createMany function to insert multiple rows, but something we have changed or upgraded recently (we aren’t sure what exactly) was causing it to actually insert rows one at a time instead. This isn’t too big of a deal if you are only inserting a few rows at a time, but we needed to insert hundreds to tens of thousands of rows at a time and doing them one by one wasn’t going to cut it.

We have also very recently started using Prisma’s typedSQL functionality to allow us to write real SQL and still get type safety as well, which is great, except it doesn’t handle a variable number of parameters.

So I had the idea to instead send a single parameter into the statement as JSON, an array of objects, then parse that into a recordset, and use that to insert into the database.

example

Here is what that looks like:

WITH raw_input(json_blob) AS (
    -- In production you would pass $1 here;
    VALUES (
        $$[
            {
              "id":           "9f2ddc93-0db3-46d2-a2cb-e6df4418faad",
              "key":          "status",
              "value":        "active",
              "created_at":   "2025-06-24T12:34:56Z",
              "updated_at":   "2025-06-24T13:45:00Z"
            },
            {
              "id":           "b8f9152e-a203-4d0a-b530-0a4e45c9b0a9",
              "key":          "priority",
              "value":        "high",
              "created_at":   "2025-06-24T12:34:56Z",
              "updated_at":   "2025-06-24T13:45:00Z"
            }
        ]$$::json
    )
)
-- Expand the JSON array to a proper recordset
, parsed AS (
    SELECT *
    FROM raw_input
    CROSS JOIN LATERAL json_to_recordset(json_blob) AS r(
        id          uuid,
        key         text,
        value       text,
        created_at  timestamptz,
        updated_at  timestamptz
    )
)
-- Bulk-insert the rows in a single SQL statement
INSERT INTO target_table (
      id, key, value, created_at, updated_at
)
SELECT
      id, key, value, created_at, updated_at
FROM   parsed
RETURNING *;

You can try it for yourself here: https://dbfiddle.uk/Am4xRgbz

There are only two parts to really understand here:

json_to_recordset takes a json blob and a schema for the resulting recordset and parses your data into a recordset. You don’t have to reference or use all keys in the json, and if you provide a column that doesn’t have a matching key you will get null for those values.

The only other thing to know is that you need to use CROSS JOIN LATERAL, this will parse each record individually giving you one row per object.

And that’s all there is to it. Now we have a statement that takes a single input, but that input can contain thousands of rows. The limit for a single parameter into a statement is an entire gigabyte of data, so you can use this to insert a lot of data at once.

The difference for us was significant. When inserting 1k records individually it was timing out after 60s. Even inserting in batches of 250 at a time using the previous createMany functionality it was taking 10’s of seconds. This was able to do it in less than a second every time, usually less than a 1/4 of a second.

Of course you could use this with ON CONFLICT DO UPDATE for upserts, or just as a way to get data into a statement for joining with other data. And you can enrich the json_to_recordset definition with default columns, CHECK constraints, or even a WHERE clause inside the CTE to filter out data early before you insert. Even join with other data in the database.

full SQL example

Adding the full SQL example here in case the dbfiddle link ever stops working:

create table target_table (id uuid, key text, value text, created_at timestamptz, updated_at timestamptz);

WITH raw_input(json_blob) AS (
    -- In production you would pass $1 here;
    VALUES (
        $$[
            {
              "id":           "9f2ddc93-0db3-46d2-a2cb-e6df4418faad",
              "key":          "status",
              "value":        "active",
              "created_at":   "2025-06-24T12:34:56Z",
              "updated_at":   "2025-06-24T13:45:00Z"
            },
            {
              "id":           "b8f9152e-a203-4d0a-b530-0a4e45c9b0a9",
              "key":          "priority",
              "value":        "high",
              "created_at":   "2025-06-24T12:34:56Z",
              "updated_at":   "2025-06-24T13:45:00Z"
            }
        ]$$::json
    )
)
-- Expand the JSON array to a proper recordset
, parsed AS (
    SELECT *
    FROM raw_input
    CROSS JOIN LATERAL json_to_recordset(json_blob) AS r(
        id          uuid,
        key         text,
        value       text,
        created_at  timestamptz,
        updated_at  timestamptz
    )
)
-- Bulk-insert the rows in a single SQL statement
INSERT INTO target_table (
      id, key, value, created_at, updated_at
)
SELECT
      id, key, value, created_at, updated_at
FROM   parsed
RETURNING *;


-- example so you can see what the parsed data looks like
WITH raw_input(json_blob) AS (
    VALUES (
        $$[
            {
              "id":           "9f2ddc93-0db3-46d2-a2cb-e6df4418faad",
              "key":          "status",
              "value":        "active",
              "created_at":   "2025-06-24T12:34:56Z",
              "updated_at":   "2025-06-24T13:45:00Z"
            },
            {
              "id":           "b8f9152e-a203-4d0a-b530-0a4e45c9b0a9",
              "key":          "priority",
              "value":        "high",
              "created_at":   "2025-06-24T12:34:56Z",
              "updated_at":   "2025-06-24T13:45:00Z"
            }
        ]$$::json
    )
)
, parsed AS (
    SELECT *
    FROM raw_input
    CROSS JOIN LATERAL json_to_recordset(json_blob) AS r(
        id          uuid,
        key         text,
        value       text,
        created_at  timestamptz,
        updated_at  timestamptz
    )
)  
select * from parsed;

select * from target_table;

Precedence Level ordering with Resets using window functions in SQL

Sun, 26 Jan 2025 12:00:00 +0000

Motivation

One of my larger projects at Vendr in the last couple years has been a document extraction system. It uses LLM and other technologies to extract structured data from contracts and other documents, allowing us to build a sizable dataset of all of the actual pricing and negotiation for SaaS software for thousands of companies.

But as we develop our extractions to get new details, or to improve the accuracy of the information, we may extract a document multiple times. We also have human review, correction, and people adding more information from context that may not be in the document. We may be extracting dozens of fields across the extractions, and so we needed a way to determine the precedence of different extractions on a per field per document level.

The problem at hand: Precedence Level

There is more that goes into how to choose which extraction is the best for an individual document than what I am going to go into here, the whole process is actually quite involved, but I wanted to talk about a particular challenge to allow us to set a precedence level on each extraction (or change it after the fact), to help guide that process.

To start with, we needed two precedence levels - DEFAULT and IMPORTANT. We would typically set an LLM extraction as DEFAULT, and human involved extractions as IMPORTANT. An IMPORTANT extraction beats a DEFAULT one, even if the DEFAULT comes after an IMPORTANT, but within the same precedence level the last one wins.

But then sometimes we may have an advancement in the LLM extractions that would lead us to want to let it take precedence over any previous extractions, even human guided IMPORTANT ones - but from then we would want any future extraction to take precedence again, either a new more recent DEFAULT or a new human corrected IMPORTANT extraction.

So I wanted to support another precedence level called RESET. An extraction with RESET should beat anything that came before it, but be treated like DEFAULT after that point, with a new DEFAULT or IMPORTANT taking precedence.

We store all extractions in the database and never get rid of them - we want to keep track of every value that was ever extracted for a field in a document and retain that history, and also we want to be able to adjust the precedence (and other attributes that affect which extraction is the best) at a later date.

To calculate the best extraction I use a SQL function in postgres - so I needed a way to implement this precedence logic using nothing but SQL.

The SQL challenge

Ordering information by precedence level with just a priority (IMPORTANT > DEFAULT) is easy in SQL - you just have to assign a sort value to those levels using a join against those values or a case statement plus a timestamp. But implementing something like RESET means that a normal sort wont work - the ordering is dependent on a timestamp and if a RESET comes before it.

So to do this, you need window functions - in this case LEAD or LAG - to look at the records before the current row and figure out if there was a RESET preceding it. But not just for the record immediately before it, all of them.

Here is an example of the query (much simplified and reworked to try and focus on just the precedence level for clarity):

with data (id, document_id, value, precedence_level, updated_at) as (
	values
		( 1, 1, 'Z', 'DEFAULT', '2025-01-01'::date)
		, ( 2, 1, 'Y', 'DEFAULT', '2025-01-02') -- later DEFAULT beats previous DEFAULT
		, ( 3, 1, 'X', 'DEFAULT', '2025-01-03') -- later DEFAULT beats previous DEFAULT
		, ( 4, 1, 'E', 'IMPORTANT', '2025-01-04') -- IMPORTANT beats DEFAULT
		, ( 5, 1, 'F', 'DEFAULT', '2025-01-05') -- previous IMPORTANT beats this record
		, ( 6, 1, 'D', 'IMPORTANT', '2025-01-06') -- later IMPORTANT beats previous IMPORTANT
		, ( 7, 1, 'C', 'RESET', '2025-01-07') -- RESET beats everything else
		, ( 8, 1, 'B', 'DEFAULT', '2025-01-08') -- bc last record was RESET this DEFAULT now takes precedence
		, ( 9, 1, 'A', 'IMPORTANT', '2025-01-09') -- IMPORTANT overrides DEFAULT
)
, data_with_resets as (
	select *
		, count(*)
			filter (where data.precedence_level = 'RESET')
			over (
				partition by document_id
				order by updated_at asc
			) prev_reset_count
	from data
)
, calculate_ranking as (
	select
		dense_rank() over (
			partition by document_id
			order by
				(prev_reset_count * 10) +
				(data.precedence_level = 'IMPORTANT')::int * 5
					desc
				, updated_at desc
		) as rnk
		, *
	from data_with_resets data
	order by 1
)
, data_with_prev_value as (
	select *
		, lead(value) over (partition by document_id order by rnk) prev_value
	from calculate_ranking
)
, final as (
	select
		dense_rank() over (ORDER BY rnk, updated_at desc ) rnk_final
		, *
	from data_with_prev_value
	order by 1 asc
)
select *
from final
;

You can play with this query and the parts of it here: https://dbfiddle.uk/HBKDOaR3?highlight=16

The key parts that make this work start with the data_with_resets CTE. Here we count how many preceding rows are a RESET precedence level using a window for the same document ordered by the timestamp. It isn’t necessarily obvious from this code unless you are familiar with window functions, but when you use count with a window (the over clause) it is a running count up to and including the current row.

Then the actual logic happens in the calculate_ranking CTE. we use a dense_rank which gives us a consecutive ranking without gaps. We partition by the document and then we order by the precedence level. We add 10 for each RESET, DEFAULT is treated as 0 and IMPORTANT is treated as 5, then ordering by the timestamp.

The next CTE isn’t required for our purpose here, but allows us to figure out what the previous value was, based on the same precedence ordering, so we can tell if something changed.

Technically the last final CTE isn’t strictly necessary, but it allows us to get a single ranking field to order the final recordset by.

Conclusion

The main takeaway I hope to convey with this post is that SQL traditionally has you think in terms of sets of data which is very powerful, but sometimes you need to be able to work iteratively and consider the previous or next row, or a window of rows not including the entire set. Window functions extend your capabilities of what you can achieve in a single SQL query and using only your database, keeping you from having to do this work in the application layer, increasing performance and lowering your memory and network utilization.

Optimizing Search with Parallel Queries in TypeScript

Sat, 19 Oct 2024 15:00:00 +0000

Recently, I encountered a situation where I built a search interface for a backend process. Users could search using various criteria—different identifiers or other metadata tagged to the records. However, the query I initially wrote to handle these searches didn’t perform well. It relied heavily on OR conditions or IN clauses (which are logically equivalent to OR), making indexing difficult and query performance sluggish^[1]

In this case, all the different search options were mutually exclusive — if results were found using one criterion, no other criterion would yield results. All our identifiers are UUIDs or globally unique formats, ensuring that only one search method could return valid results. Additionally, some queries were fast and efficient, while others were slower and more resource-intensive. Combining all these into a single query meant the user would always have to wait for the slowest part of the search, even if they were searching for something could return results immediately.

A Different Approach: Parallel Query Execution

To improve performance, I decided to run multiple searches in parallel and return results as soon as one of them succeeded. If none of the queries returned results, the system would fall back to showing the latest records. This solution is somewhat similar to Promise.race(), but with a twist. Promise.race() resolves or rejects with the first settled promise, regardless of whether it yields useful data. In my case, I wanted to inspect the resolved values and only return a result if it contained results.

After working with ChatGPT, Claude, and some friends, I developed the following solution. Below are two versions of the function—one that uses key-value pairs for better traceability and another that works with an array of queries. I have more to say about it below.

Implementation

/**
 * Runs multiple queries in parallel and resolves with the first query that returns records.
 * If no query returns records, it rejects with an error.
 *
 * @param queries An array of promises representing database queries.
 * @returns A promise that resolves with the first valid result or rejects if no results are found.
 */
export async function firstQueryWithResultsWithKey<T>({
  queriesAsRecord,
  resultTest = ({ result }) => Array.isArray(result) && result.length > 0,
  noResultsReturn = null,
}: {
  queriesAsRecord: Record<string, Promise<T>>;
  resultTest?: ({ result, key }: { result: T; key: string | null }) => boolean;
  noResultsReturn?: T | null;
}): Promise<{ result: T; key: string | null }> {
  return new Promise((resolve, reject) => {
    let completed = false; // Track if a query has returned results
    let pendingCount = Object.keys(queriesAsRecord).length; // Track the number of pending queries

    if (pendingCount === 0) {
      if (noResultsReturn) {
        resolve({ result: noResultsReturn, key: null });
      } else {
        reject(new Error("No queries returned any records"));
      }
    }

    // Define a function to handle each query independently
    const handleQuery = async (query: Promise<T>, key: string) => {
      try {
        if (!completed) {
          const result = await query;

          // If the result has records and no other query has resolved
          if (resultTest({ result, key }) && !completed) {
            completed = true; 
            resolve({ result, key }); // Resolve with the first valid result
          }
        }
      } catch (error) {
        console.error("Query error:", error); // Log query errors
      } finally {
        // Decrement pending count and check if all queries are exhausted
        pendingCount--;
        if (pendingCount === 0 && !completed) {
          if (noResultsReturn) {
            resolve({ result: noResultsReturn, key: null });
          } else {
            reject(new Error("No queries returned any records"));
          }
        }
      }
    };

    // Start all queries in parallel
    Object.entries(queriesAsRecord).forEach(([key, query]) =>
      handleQuery(query, key),
    );
  });
}

export async function firstQueryWithResults<T>({
  queries,
  resultTest = (result) => Array.isArray(result) && result.length > 0,
  noResultsReturn = null,
}: {
  queries: Promise<T>[];
  resultTest?: (result: T) => boolean;
  noResultsReturn?: T | null;
}): Promise<T> {
  let queriesAsRecord = Object.fromEntries(
    queries.map((query, index) => [index.toString(), query]),
  );

  const { result } = await firstQueryWithResultsWithKey({
    queriesAsRecord,
    resultTest: ({ result }) => resultTest(result),
    noResultsReturn,
  });
  return result;
}

Both versions of the function execute a collection of promises in parallel, returning the first valid result. If no queries pass the resultTest, the function returns the provided fallback value (if any) or throws an error.

Example Usage

Here’s a unit test to demonstrate how the function works:

const wait = (interval: number) =>
    new Promise((resolve) => setTimeout(resolve, interval));

it("should return the noResultsReturn value if no queries return results", async () => {
    const query1 = async ({ id }: { id: string }): Promise<QueryResult> => {
      await wait(100);
      return [];
    };
    const query2 = async ({ id }: { id: string }): Promise<QueryResult> => {
      await wait(200);
      return [];
    };
    const query3 = async ({ id }: { id: string }): Promise<QueryResult> => {
      await wait(300);
      return [{a: id}];
    };

    let result: QueryResult | null = null;

    result = await firstQueryWithResults({
      queries: [query1({ id: "1" }), query2({ id: "2" }), query3({ id: "3" })],
      resultTest: (result) => result.length > 0,
      noResultsReturn: [],
    });
    expect(result).toEqual([]);
  });

In this test, three promises simulate query operations. The first two return empty arrays after 100ms and 200ms, while the third returns a valid result after 300ms. The resultTest function ensures that only non-empty arrays pass the validation.

Reflections and Considerations

This approach improves user experience by returning results as quickly as possible, but it also increases the load on the database since multiple queries run in parallel. For my use case — an internal back-office tool — I find the trade-off acceptable. However, I plan to monitor system performance to ensure this doesn’t impact the overall system.

An alternative approach would be to let users specify the type of identifier they are searching for. This way, the system would only execute the relevant query, reducing database load. However, the current solution offers a seamless experience since users don’t need to know or care about the type of identifier—they just paste it and search.

Although I used the term “query” throughout the article because of my use case, the function is generic and can be applied to any collection of promises, not just database operations.

I welcome any feedback or suggestions on this approach. While I’m not fully convinced this is the optimal solution, I found the challenge of reimagining Promise.race() to be an interesting exercise.

Here is a gist with the function and more unit tests so you can see how it works: https://gist.github.com/ryanguill/e89931f9d223e74dabcb070879a58298

[1] There are ongoing improvements in PostgreSQL to optimize queries with OR conditions. For more, see: https://www.crunchydata.com/blog/real-world-performance-gains-with-postgres-17-btree-bulk-scans

safe_cast() function for postgresql

Thu, 17 Oct 2024 12:00:00 +0000

I love using JSON in relational databases. When support for JSON types and functionality first started coming out in SQL I generally thought it was neat but that it wouldn’t be something I would ever want to use in production. I could not have been more wrong.

I could talk at length (and have) about all the ways that it is useful, but if you do you will find that the main way you pull information out of JSON will bring it out as TEXT. And frequently when you’re using JSON you can’t be sure that the data is exactly the right format you expect anyway. Lately I store a lot of JSON that comes back from LLMs, and while it gets it right most of the time, you can never really be sure - you need to trust be verify.

So I have been using this function for a long time to safely convert from text to a given datatype in postgresql. If the cast can be made successfully it will, otherwise it will return the second argument as a default - most of the time I use null but it can be anything.

/* 
  utility function to convert from text to various data types
    , such as when you are pulling values out of json 
  The function will cast the value of the first argument 
    to the type of the second argument.
  If the first argument cannot be convert to the target type
    the value of the second argument will be returned as the default.
  If you want to return null as the default, cast the null to the target
    type like `null::integer` or `null::date`
*/
DROP FUNCTION IF EXISTS safe_cast(text, anyelement);
CREATE OR REPLACE FUNCTION safe_cast(text, anyelement)
RETURNS anyelement
LANGUAGE plpgsql as $$
BEGIN
    $0 := $1;
    RETURN $0;
    EXCEPTION WHEN OTHERS THEN
        RETURN $2;
END;
$$;

The function itself looks a little terse and strange, but if you look and play with the examples you’ll get a better understanding of how it works. I have been using it for a long time, I can’t remember if I actually wrote it or got it from somewhere else - I believe I may have adapted it from this answer on stackoverflow: https://stackoverflow.com/a/2095676/7186

Below are examples, but you can play with the function and these examples here: https://dbfiddle.uk/3kAop9-Z

select
	  safe_cast('true', false::boolean) = true
	, safe_cast('false'::text, false::boolean) = false
	, safe_cast('yes'::text, false::boolean) = true
	, safe_cast('no'::text, false::boolean) = false
	, safe_cast('on'::text, false::boolean) = true
	, safe_cast('off'::text, false::boolean) = false
	, safe_cast('1'::text, false::boolean) = true
	, safe_cast('0'::text, false::boolean) = false
	, safe_cast(1::text, false::boolean) = true
	, safe_cast(0::text, false::boolean) = false
	, safe_cast('foo'::text, false::boolean) = false
	, safe_cast('3'::text, false::boolean) = false
	, safe_cast(3::text, false::boolean) = false
	, safe_cast('', false::boolean) = false
	, safe_cast(null::text, false::boolean) is null
	;

select
	  safe_cast('123', null::numeric) = 123::numeric
	, safe_cast('123.45', null::numeric) = 123.45::numeric
	, safe_cast('0', null::numeric) = 0::numeric
	, safe_cast('-1', null::numeric) = -1::numeric
	, safe_cast('-1.2', null::numeric) = -1.2::numeric
	, safe_cast('123x', null::numeric) is null
	, safe_cast('', null::numeric) is null
	, safe_cast('foobar', null::numeric) is null
	, safe_cast('123', null::numeric) > 1
    , safe_cast('123', 0::integer) = 123::integer
	, safe_cast('', 0::integer) = 0::integer
	, safe_cast('foo', 0::integer) = 0::integer
	;

select
	  safe_cast('2024-01-02', null::date) = '2024-01-02'::date
	, safe_cast('01-02-2024', null::date) = '2024-01-02'::date
	, safe_cast('2024-01-023', null::date) = '2024-01-23'::date --possibly surprising
	, safe_cast('2024-01-', null::date) is null
	, safe_cast('2024-01-123', null::date) is null
	, safe_cast('2024', null::date) is null
	, safe_cast('foobar', null::date) is null
    , safe_cast('2024-01-02', null::timestamptz) = '2024-01-02 00:00:00'::timestamptz
	;

Other databases have similar functionality built in, but most do not have the ability to set a default if the value cannot be safely cast baked into the function, most just return null and then you can use coalesce or similar to set a different default if you need to. (The following list is the ones I know off the top of my head, and is not meant to be exhaustive)

MSSQL has TRY_CAST: https://learn.microsoft.com/en-us/sql/t-sql/functions/try-cast-transact-sql?view=sql-server-ver16

Snowflake has TRY_CAST: https://docs.snowflake.com/en/sql-reference/functions/try_cast

DuckDB has TRY_CAST https://duckdb.org/docs/sql/expressions/cast.html#try_cast

Partitioning using UUIDs

Fri, 09 Aug 2024 12:00:00 +0000

Recently I had a situation where I needed to process quite a few records, lets say around 100k, and the processing of each record was around 2 seconds. The math on that wasn’t pretty, so I needed to find a way to partition the set of data so that I could run multiple instances of the processing at the same time - but where each instance would have their own data and I wouldn’t try to process a record by more than one instance.

Normally you would want to look for some natural key if you can, but in this case there wasn’t anything I could use, especially that would be well distributed. I also wanted to have the flexibility to partition for as few or as many instances as I needed - I might start with 5 instances but that might not be enough.

The one thing I did have is UUIDs. We use UUIDs for virtually all of our keys - an explanation of all of the benefits of that will have to be its own blog post someday.

So I had the thought to use UUIDs to do the partitioning. The easiest thing you could do is look at the last character, that would quickly give you 16 partitions without much work, and most of the versions of UUIDs the last character would be fairly well distributed.

But then I remembered - a UUID is really just a 128 bit number, or two 64 bit integers (bigints) in a trench coat - and assuming they are randomly generated (we use v4 for most things right now so I know they are), I can use modulus math to partition into however many buckets I need.

I was able to come up with the following that given a UUID will extract the high bigint and then give mod 5:

SELECT abs(('x' || translate(gen_random_uuid()::text, '-', ''))::bit(64)::bigint % 5)

This might look complicated at first, but while it has a few steps its fairly straight forward. A few things to notice first:

A UUID is a hex encoded number that has hyphens in particular places to break it up into pieces that make it a little easier to read. So to break it apart and pull out just the higher piece:

a) convert the uuid to text

b) remove the hyphens

c) prepend an ‘x’ to the beginning of the string - this gives us a string of hex characters that can be understood as hex encoded

d) convert that to a bit string using ::bit(64) - this part uses a fact that if you take a number bigger than a bit type can hold it will just take as much as can fit and throw away the rest, so this will only take the first 64 bits

e) take that 64 bits and convert it to a bigint using ::bigint

f) lastly take its absolute value because the number is signed and you might get a negative, but that doesnt matter for our purpose

With these steps done you only need to use % to take the modulus of that number and get the remainder. You can change 5 to whatever number you want to get the number of buckets - just remember that the answer will be zero based. If you use % 5 you will get 0 through 4 as possible answers.

Now when I create my instances I only have to put in the number of partitions and which partition this instance is working with, and a simple where clause in the query to get the population it is working on will restrict it to only the records that belong to that instance and no two instances will try to do the same work.

For clarity, here is what this might look at in use:

SELECT
  id
  , ...
FROM foo
WHERE abs(('x' || translate(id::text, '-', ''))::bit(64)::bigint % 5) = 2

If you’re interested in exploring more about how UUIDs split into numbers and back again or proving to yourself that a random uuid is evenly distributed, you can check out this fiddle: https://dbfiddle.uk/yX-y7Ath

And one more bonus fact related to UUIDs - MD5 hashes are also 128 bit numbers stored as hex, and since UUIDs are also just 128 bit numbers stored as hex, you can convert MD5 hashes to UUID and store them using the UUID type in your database for space savings compared to storing it as text.

Aggregating event data into sessions using SQL

Mon, 24 Jun 2024 12:00:00 +0000

Lets say you have a table of events, each representing a point in time when an event occurred. Maybe it’s clicks in your applications, or pages that a user visited. You might like to see how long users are spending in your application, or what they are doing in a given session.

So the task at hand is to use SQL to analyze this event data and combine these points in time into groups of events - I will call them sessions for this post.

The first thing you must do is decide how much time you would allow someone to go before another event happens for it to still be considered the same session. Anything less than this duration between events, and the events are connected into the same session; anything more, and we will call it a new session. The value you use here is up to you and your needs - if the actions you are looking at take a lot of time to perform, or maybe someone is spending time reading a lot of content before continuing to the next thing, you might want to make this threshold higher, maybe tens of minutes or even more. On the other hand, if you expect they should be clicking around fairly quickly, you might want to go much shorter, maybe only seconds. For this example I am going to use 5 minutes between events as my threshold.

The next thing you need to decide is, if the session only lasted for one event, how long would you want to say it lasted? For this example, I am going to use 1 minute.

So there are several SQL features we can use to our advantage - this is all PostgreSQL; everything here works at least as far back as version 9.5, possibly earlier.

The first feature is Postgres’ interval type. If you haven’t played with intervals, you are really missing out. It is a native data type that defines a period of time - perfect for our use cases.

The next thing to understand is window functions. I have been writing SQL long enough that these still feel “new”, but they have been around for a long time now! Window functions are great; they let you aggregate against different partitions (groups, or in this case windows) in the same query, which is perfect for our needs where we want to consider each user independently.

Let’s look at some example data to get our bearings:

DROP TABLE IF EXISTS event_log CASCADE;
CREATE TABLE event_log (
	  ts timestamptz
	, name varchar
);

INSERT INTO event_log ( ts, name ) VALUES
  ('2024-07-01T12:00:00.000Z', 'Alice')
, ('2024-07-01T12:01:00.000Z', 'Alice')
, ('2024-07-01T12:02:00.000Z', 'Alice')
, ('2024-07-01T12:03:00.000Z', 'Alice')
, ('2024-07-01T12:04:00.000Z', 'Alice')
, ('2024-07-01T12:05:00.000Z', 'Alice')
, ('2024-07-01T12:12:00.000Z', 'Alice')
, ('2024-07-01T12:14:00.000Z', 'Alice')
, ('2024-07-01T12:17:00.000Z', 'Alice')
, ('2024-07-01T12:21:00.000Z', 'Alice')
, ('2024-07-01T12:26:00.000Z', 'Alice')
, ('2024-07-01T12:30:00.000Z', 'Alice')
, ('2024-07-01T12:00:00.000Z', 'Bob')
, ('2024-07-01T12:02:00.000Z', 'Bob')
, ('2024-07-01T12:05:00.000Z', 'Bob')
, ('2024-07-01T12:09:00.000Z', 'Bob')
, ('2024-07-01T12:14:00.000Z', 'Bob')
, ('2024-07-01T12:20:00.000Z', 'Bob')
, ('2024-07-01T12:25:00.000Z', 'Bob')
, ('2024-07-01T12:35:00.000Z', 'Bob')
, ('2024-07-01T12:44:00.000Z', 'Bob')
, ('2024-07-01T12:48:00.000Z', 'Bob')
, ('2024-07-01T13:05:00.000Z', 'Bob')
;

Here you can see we have two users, Alice and Bob, and they have several events. Of course, in a real world scenario, these names would probably be IDs to a users table, you would have an event type and probably lots of other attributes for the event, but none of that is necessary for this demo. If you have trouble extending this example to a more real world scenario, reach out in the comments or on Twitter.

Here is the final query, we will walk through the different parts of it to see how it works:

with inputs as (
  select
      '5 minutes'::interval threshold -- change this to however long you want your session timeouts to be
    , '1 minute'::interval min_session_duration -- if there is only one event in the session, how long do you want to say it lasted?
)
, prep as (
  select
      ts
    , name
    , row_number() 
      over (partition by name order by ts) row_num
    , coalesce(
      ts - lag(ts) 
        over (partition by name order by ts asc) <= inputs.threshold
      , false) is_connected
  from
    event_log
  cross
    join inputs
)
, session_detail as (
  select
      ts
    , name
    , max(row_num) 
      filter (where NOT is_connected) 
      over (partition by name order by ts asc) session_seq
  from prep
)
, sessions as (
  select
      row_number() 
      over (partition by name order by min(ts), session_seq) session_seq
    , name
    , count(*) event_count
    , min(ts) start_ts
    , max(ts) end_ts
    , greatest(inputs.min_session_duration, max(ts) - min(ts)) session_duration
  from session_detail
  cross
    join inputs
  group by
      session_seq
    , name
    , inputs.min_session_duration
)
select
    session_seq
  , concat_ws('-', name, session_seq) unique_id
  , name
  , event_count
  , start_ts
  , end_ts
  , session_duration
from sessions
;

If you arent comfortable with SQL that might look like a lot, but trust me, its really not bad - and we will walk through all the different parts.

Here is what the query returns (I have removed the date and seconds and milliseconds for the timestamps for brevity, they are all the same for the example):

session_seq	unique_id	name	event_count	start_ts	end_ts	session_duration
1	Alice-1	Alice	6	12:00	12:05	00:05:00
2	Alice-2	Alice	6	12:12	12:30	00:18:00
1	Bob-1	Bob	5	12:00	12:14	00:14:00
2	Bob-2	Bob	2	12:20	12:25	00:05:00
3	Bob-3	Bob	1	12:35	12:35	00:01:00
4	Bob-4	Bob	2	12:44	12:48	00:04:00
5	Bob-5	Bob	1	13:05	13:05	00:01:00

Lets start with the two pieces of SQL that make this work.

First theres this line:

, coalesce(
    ts - 
    lag(ts) over (partition by name order by ts asc) 
        <= inputs.threshold
    , false) is_connected

Forgive the formatting - there is no great way to format it that will make everyone happy, but hopefully I can walk you through the pieces.

The goal with this line is to get true or false, is the row we are looking at part of the same session as the previous event for the user.

Starting from the inside out - for the current row, we are taking its ts (timestamp), and subtracting the timestamp of the row before it. This is a window function - you know that because of the over () clause. The window is how we decide what “before it” means.

We are using partition by name, meaning that our window will always only consider rows that have the same values for name as our current row - and we are using order by ts asc, meaning “before” will be a row earlier than the current row.

partition by works just like group by, you are defining the individual groupings of columns whose combination of values are considered unique to define your different groups. In our case, we only want to consider rows by the same user.

lag() is just one option, theres its dual lead(), which if you were to order by desc instead would work just the same.

We do our subtraction and get an interval - a duration of time since that previous event. Now we just need to know if that interval is less than our threshold - if so we are still part of the same session, if not then this is the beginning of a new session.

But our lag(ts) ... could return null - if we are looking at the first event for a user, there is no previous value. And so if we take our current row’s timestamp and subtract null, the result will of course be null. And when we do our comparison with the threshold, we will also get null. So we coalesce() our value to false in that case - if it’s the first event for a user it certainly cannot be part of a previous session.

So now lets look at the other important part of this approach:

, max(row_num) 
      filter (where NOT is_connected) 
      over (partition by name order by ts asc) session_seq

This part is a little more complicated, so bear with me:

Earlier in the query we assigned a row number to each row with the row_number() window function, using the same partitions as we previously discussed.

Our goal is to get the highest row number that was the beginning of the current session - so we are only interested in rows where is_connected is false.

The most important thing to understand is that max() as a window function that includes an order by clause will not consider the entire window, but only the window frame up to the current row. I’m going to quote the postgres documentation here: ¹

There is another important concept associated with window functions: for each row, there is a set of rows within its partition called its window frame. Some window functions act only on the rows of the window frame, rather than of the whole partition. By default, if ORDER BY is supplied then the frame consists of all rows from the start of the partition up through the current row, plus any following rows that are equal to the current row according to the ORDER BY clause. When ORDER BY is omitted the default frame consists of all rows in the partition.

So our max() is only considering rows where is_connected is false, meaning the events that were the beginning of sessions. Then we want to pick the highest row number that came before the current row - so if we are in the second session we want to pick the beginning of the second session.

These are the two most important parts of this solution - how to figure out if the current row is a continuation of a session or the start of a new one, and then how to find the start of the session for the current row, so we know which session we are a part of.

The one other thing I want to call out here is the CTE at the top I call inputs - this is a pattern I use a lot - it pulls the “variables” of a query to the top, and just selects them as literal values - then I can cross join them where I need to in the rest of the query, making it dynamic to those inputs.

Here is a database fiddle if you want to play with this query to understand it better. I really recommend pulling out parts of the CTEs and understanding how each part works: https://dbfiddle.uk/M_iWoTG-

One last thing to mention is that this query is useful if you want to slice and dice your event data in a few different ways to explore it. You might get different results and gain more insights into your data if you tweak the thresholds. However, if you already have these thresholds set in stone, this would be an easy thing to calculate as your data comes in, or in an intermediate table, ingesting the event data on some period and calculating the duration from the previous event and which session they were a part of.

My PostgreSQL wishlist

Mon, 08 Jan 2024 00:00:00 +0000

If you know me you’re aware that I love SQL and relational databases. Of all of the databases I have used, PostgreSQL is by far my favorite. SQL is the one technology I have seen in common across every job I have had over the past two decades. I would bet you could say the same - even if you aren’t writing it directly (which would be a shame in my opinion), it is still there, behind the scenes, handling all of your data. But if you’re fortunate enough to work directly with SQL and a database, it’s hard to find a better tool.

But while I have never seen a better database than PostgreSQL (and you can’t beat the price!), that doesn’t mean that there aren’t things that could be improved. My goal with this post is to start a conversation and throw out some ideas that I have gathered over the years of things I yearned for. Best case scenario, someone tells me that these things already exist and I learn something new! Second best, these ideas resonate with others and we find ways to get some of these in future versions. But I would also love any conversation about why any of these are bad ideas or technically infeasible - I’ll learn something then too. And if anyone has ideas about how I can contribute to these things directly, please let me know!

This list is somewhat organized into sections and subsections, but is not really in any order of importance or ease of implementation at all.

DDL Enhancements

So lets start with DDL, how we create and manage our schema. I have to start this section off with praise for transactional DDL - if you have ever worked without it you’ll know how amazing it is. But I find myself wanting more, especially around describing the schema in more detail, and using that to generate or check code using the schema. So lets get started:

I wish I could set a column as insertable but not updatable, so immutable.

There are times where I want to have a column able to be written to on insert, but immutable from then on. There are ways to do this with triggers, or permissions but I want to be able to specify this on the schema itself, to be able to retrieve that a column is immutable.

Bonus points if I could say that it could be inserted as null, and updated from null, but not updated if the value is anything other than null.

I wish I could set a column to be generated on update.

The easiest example here would to set an updated_at column automatically if a row is updated in any way, but without having to specify it. I want to be able to do this when I create (or alter) the table, but never have to think about it again. created_at is easiest enough with a default value. But would be better if it could be made immutable.

I wish I could have a "weak" or optional foreign key.

Something like REFERENCES foo ON DELETE IGNORE
I want to be able to say that if this key exists, it is in this table, but it might not exist any longer.
The easiest example here would be for an audit log. The FK ties to the underlying table, but if the record for the underlying table is deleted I wouldn't want anything to happen to the data in the audit log.
I could see a use for a version that checks that the reference row exists at time of insert but then is ignored after that, and a version that doesn't care at all, I just want to be able to describe in my schema that this is how these tables are related (and you probably want to use an outer join).

I wish I could have a foreign key that used an operator other than equality.

I think if this was possible it would open up several possibilities, but the main motivation here for me is bitemporal tables. I want to have a FK that takes an ID and a timestamp in one table, and ties it to a row in the target table with the same ID and a timestamp range that includes my timestamp. I have a whole set of articles coming on bitemporal modeling in postgres and this would be amazing for that work.

I wish that when you created a foreign key that it would create an index on the target column by default, and if necessary give a way to opt out of it.

This is a very common source of performance problems, and one that surprises a lot of developers coming from other databases.

I wish I could assert a non-empty constraint the same way I assert a not null constraint.

I can do this with check constraints, but I wish it was something described in the schema itself. And you could take this further, assert that it must be a certain length (or range of lengths), or match a certain regex. But I want this in the schema so it is available to codegen tools.
I'm certain you could do this too with custom types, but those are also very opaque to the schema. I want something that is easy to build other tools on top of.

I wish you could specify JSONBArray vs JSONBObject vs JSONB.

A common constraint is to say that a JSONB column must be null or an object, or null or an array. most of the time if you are working with json columns it is either an array or an object, not either or some other JSON primitive. Of course you can do this today with json(b)_typeof but constraints don't show up in the schema. I want the schema to describe the shape of my data.

I wish that jsonschema was supported natively and we could write constraints using it

Basically I wish pg_jsonschema from supabase was built in

I wish that there was a - operator or jsonb_diff() function for JSONB like HSTORE to get a diff of objects

Here is an example. Super useful for audit logging.

I wish you could specify in the schema that view columns were not null.

All view columns are forced to be nullable, even if they aren't really. I don't know how you could do this or what the syntax should look like, but it is frustrating to have all columns in a view be nullable. PG should be smart enough to figure out if it is possible in most if not all cases.

I wish you could alter the definition of an existing generated column.

Columns change, it shouldn't require you to drop and recreate a column. I might want to leave previously generated data alone and change the definition going forward.

I wish you could do something like CREATE INDEX CONCURRENTLY IF AVAILABLE ...

It's a common problem that I run migrations inside of a transaction where possible, but I want to be able to copy those same statements and run them manually if necessary to get concurrent index creation. I just want to make the syntax of the statement work in either situation.

Schema Metadata

This section is a little more abstract, so I want to give an introduction to my perspective. My goal is to be able to describe and understand my schema, both manually but also with tools. There are patterns, conventions, and other approaches we bring to our databases that we aren't currently able to describe. And there is many times that we are doing exploratory analysis on a database and it's data, and there are things we can do to aid in that understanding. Such as:

I wish that we could control the default order of columns without having to rewrite the table.

Store this order separate from the schema itself. I don't care how it is stored on disk. I want to be able to add a new column to a table near the beginning, so it that is obvious to anyone who comes and explores.
Similarly, I wish I could "group" (not that group, no pun intended) columns together with metadata.
As an example, I want to create a column group called "name" that is actually multiple columns, like salutation, first, last, middle, suffix, etc.
I want this mostly for documentation purposes, but this would be even more amazing if you could select the column group and get all columns in the group.

I wish columns could be case insensitive, but created as mixed case, stored that way so that codegen could use the mixed case, but retrieved using any case.

Basically what it does now, store it as lowercase on disk, but keep a mapping to the way it was created (and allow it to be altered), and return the mapped column name when returning results unless an alias is specified.

I wish we had better hash indexes

There were updates in a relatively recent version to make them crash safe, which was a great step in the right direction, but we still can't use them as a unique key or on multiple columns. We use UUIDs, IMO it should be the default. For the vast majority of cases, a btree index still works just fine, but it has things we don't need. A hash covering index would be perfect for indexing tables using UUID primary keys.

DQL Enhancements

These would be the crowd pleasers I believe

I wish I were not required to specify a GROUP BY clause for most aggregate queries.

Why must I group by columns manually? Why can't the default for a query such as SELECT COUNT(*), name FROM data not automatically know that I want to group by name? It gives me an error that I must do something with name, so it knows that it is missing. I could group by more than name, but this is rare.
But if not this, then supporting GROUP BY ALL like some other databases like duckdb or snowflake would still be a welcome improvement.
I can concede that this might be confusing for beginners to SQL. But in all the years that I have taught SQL, this is the number one confusing thing already.

I wish I could use ORDER BY ALL.

Order by the columns ascending in the specified order. Like duckdb

I wish I could use column aliases in WHERE/HAVING

Like duckdb, We can do this with GROUP BY today, but not in a WHERE or HAVING clause.

I wish I could have leading and trailing commas for column lists!

This may be my number one thing on the entire list. I am a comma preceding kind of guy. But even if you're the opposite, I bet you have wished for this too.
Please, allow these two queries to be valid:

SELECT
  , a
  , b
FROM foo;
 
SELECT
  a,
  b,
FROM foo;

I wish that multiple WHERE clauses were treated as AND conditions.

A frequent pattern of mine is to write analytical queries like this:

SELECT
 ...
FROM foo
WHERE true
--AND condition = false
AND other_condition
AND yet_another_column = 'bar'

This lets me comment out conditions during testing. I wish I could instead do things like

SELECT
 ...
FROM foo
WHERE condition = false
-- WHERE other_condition
WHERE yet_another_column = 'bar'

I wish that there was an EXCEPTION JOIN.

I used this in DB2 decades ago. Select from the left table where there is no matching row in the right table. Basically the same thing as:

SELECT
...
FROM a
LEFT OUTER
  JOIN b
  ON a.b_id = b.id
WHERE b.id IS NULL

But far more explicit:

SELECT
...
FROM a
EXCEPTION
  JOIN b
  ON a.b_id = b.id

I wish there was a "percent" function built in.

I could totally write this as a custom function and have, but I wish it was there by default. Something like: round((a / nullif(b::**numeric,0)) * 100, 2)

I wish that if you used an INNER JOIN on a table with a clause using = that it would understand that the values in those columns must be identical, so that it wouldn't treat that column name as ambiguous.

It is able to do this today if you use a NATURAL JOIN (which I've never once seen in the wild and don't really recommend), so I believe this should be possible.

Miscellaneous

I wish we had more uuid types.

uuid v4 / v5 are great. But theres lots of other implementations out there including ones that are smaller in length. It would be awesome to have more options, and for them to be built in and not require an extension.

I wish that the conversion from uuid to text could be automatic.

And I wish that the planner could be smarter with this and pick better indexes.
See also HASH INDEXes above.

I wish columns like created_at and updated_at were automatic.

Nobody should ever need to create these columns on a table in my opinion. I am not sure how to do this in a backwards compatible way, but every record written to every table should capture the timestamp at which it was inserted and when it was last updated. If users need something different they can capture it themselves like they do today, but the vast majority of use cases would be handled automatically.

I wish that bitemporal features could be built into the database by default.

This deserves and will hopefully get it's own blog series soon.

I wish I could have identifiers longer than 63 characters

It is rare, but when naming things well, especially compound things like table + constraint type, it happens. You can recompile PG to get it, but it is an unfortunate limitation.

I wish there was a built in scheduler. PG_CRON is great, but it should be built in.

A scheduler would allow us to build many things we need separate tech for today, such as automatic job or event creation, summarization of data into an append-only log, rebuild statistics and indexes, see also async queries below.

I wish there was a way to define an automatic update frequency on a materialized view

Or better yet, create implicit dependencies on the tables and possibly rows (where filter) that he db can detect that the source has updated and it automatically updates the materialized view.

I wish I could use hints.

Sometimes I want to force PG to use an index. Let me select it, let the performance to be bad and let me understand why the planner isn't using that index. Bonus points if could tell me why it didn't use that index in the first place in plain english.

I wish I could easily set an identifier when starting a transaction and have that identifier automatically added as metadata to the row as who inserted and updated the row last.

I wish I could use nested transactions.

Savepoints sorta work for this, but nested transactions would be cleaner

I wish I could create a point in time and roll back to that point in time, regardless of transactions.

This would be amazing for unit tests. Let me set the DB into a known state, set a save point, run a bunch of statements, and then throw everything away after the save point.
Even if this required putting the database into a single user mode or similar, this would be so valuable.
But this has to work across all transactions, so that you can roll back to a known state. Transaction save points dont work for this.

Bonus: I wish that I could create a point in time and then run many transactions (directly, or through an application, whatever) and then get the DDL/DDM necessary as SQL statements to go from the saved state to the current state.

I would love to be able to take a seed database, hook it up to an app and use the UI to set up scenarios that I could then use in my unit tests. This isn't bad in isolated cases, but when setting up a scenario takes coordination of tens of tables, this would be much easier.

I wish there were different storage engines similar to mysql, specifically I want to have a columnar store format in PG.

This is huge, I know, but it would be so so valuable, and look at all of the columnar stores out there that are popping up.

I wish we could have new ways to compose queries.

Allow me to create functions like query partials that can be named and composed. Similar to macros. Look at what DBT is doing and steal most of it.

This could have really been in the DQL section but I wish we could build a query pre-processor that would let us implement extensions and experiments on top of SQL.

This is a bigger thing, so let me explain. There are many things that require different syntax to achieve. Things like the composition functions I mention above, or things like GROUP BY ALL or ORDER BY ALL. These are things we should be able to extend ourselves, but we need a place to put this pre-processor in a place that all of the ways we talk to the database can go through. So best case scenario, that would be the database. Make us include a pragma or similar at the beginning of the query and opt into using it. But imagine if we could instead write a query like this:

FROM table
INNER
  JOIN other_table
  ON table.x = other_table.x
WHERE
  NOT condition
SELECT
  , x
  , count(*) n
ORDER BY ALL

This would be so much easier to teach people how to reason about SQL. You could run everything up to the WHERE clause, leaving off the SELECT and have it transformed into SELECT *. A pre-processor would allow you to explore these alternative syntaxes and see how they really work. Or build your own DSLs that get turned into standard SQL.
Bonus points if it is a full environment that has access to all of the schema (and the extensions ive wished for) and can not just transform but build queries.

I look forward to PG supporting SQL/PGQ for graph queries

This is brand new in the SQL:2023 specification (no surprise PG already has just about everything else), but there is a new graph query abilities that would be awesome to see in PG.

I wish we had better connections in PG

I know there are reasons, and I know that this might be the hardest thing on this list, but connections are one of the biggest pain points in modern stacks. Connections take too many resources, and needs for pooling and connection sharing is one of the hardest parts about connecting your serverless app to your database. I don't know what the solutions really look like, but all of the existing solutions suck.

I wish I could submit an async query

Years ago using DB2 on an AS/400 I would submit a query to "batch", meaning we would provide the query and specify a library (schema) and table to save the results to, and then set a job priority level and kick it off. Back then it wasn't uncommon to be running queries that took hours to days to finish.
It isn't common that I need to run queries that take quite that long today (and most of those times would be better suited for a data warehouse), but still there are times where I want to run a query but not hold a connection open in some client to allow it to run.
I wish I could submit a query to postgres in such a way that I don't want to wait on the result or have a connection open. It would just also require a way to inquire about the status of a job.
Note: give us a job scheduler like pg_cron built in and we could build this ourselves.

I wish that the documentation had better SEO

Oftentimes when you search for something you are taken to documentation for an old version. I wish that they would update their indexes to point to the latest version by default. Especially for features that have been around for a long time, ending up in 9.x documentation and not realizing it might mean that you are missing the latest information.

And last but not least, I wish we had a standard AST parser.

Let me use the parser built into PG and get out a standard AST that we can use for formatting, transformation, or knowing what queries are touching what parts of the schema. I could write an entire post about the use cases.

Other resources

You should also read this post by Johnathan Katz about his thoughts about postgres in 2024 and PostgreSQL and SQL:2023 by Peter Eisentraut

Conclusion

I want this to be a living document. I can't wait for someone to tell me how wrong I am in the comments. When that happens I'll update this list. And tell me what you thing I missed, the things you wish for - I'll add new ideas as well. And will be very happy to celebrate when any of these are addressed in PostgreSQL! I really hope this post is received in the spirit it is intended - the only reason I am making this post is because I believe and have seen amazing new capabilities added to PG over the years. PG release notes are one of my favorite things! These are not at all intended to be criticism. There are many many years of reasons why things are the way they are. But PG has so much potential to be bigger and better than it already is. Special thanks to Jacob Elder, Ian Burns, and Bryce Hipp for the help with this article.

Notes on PostgreSQL Explain Analyze

Fri, 25 Aug 2023 12:00:00 +0000

One thing you’ll want to learn if you use PostgreSQL for any length of time is how to use EXPLAIN. At my job at Vendr, like my previous roles, we are no exception. The good and bad thing is that in many cases you can go pretty far before you start having issues. But that means that not everyone has had the opportunity to learn the dark art of reading explain output. And as great as PG is, it is not as user friendly in this area as other RDBMs like MSSQL. But with this primer, a few free online tools, and a little bit of time, you can quickly learn how to think about the performance of your queries and where to start when optimizing.

The first thing to know is that you almost always want to use EXPLAIN ANALYZE, but its important to understand that the ANALYZE keyword will cause PG to actually execute your query so it can give you an execution time. This has two consequences:

If you use EXPLAIN ANALYZE on DML statements (that is INSERT, UPDATE, DELETE, statements, calling functions which do these things, etc), it will actually run those statements. So you want to make sure you are in a transaction that you can roll back.

And even though EXPLAIN ANALYZE runs the statement, it throws away the actual results.

But returning data to the client can be a significant part of the execution time of any given statement! Here is an example:

explain analyze verbose
select n
, repeat('abcdefghijklmnopqrstuvwxyz ', 100)
from generate_series(1,2000) as x(n);

When I run this the explain returns:

+------------------------+
|QUERY PLAN              |
+------------------------+
|Planning Time: 0.038 ms |
|Execution Time: 0.459 ms|
+------------------------+

But if I run the query (without the explain) I see:

2,000 rows retrieved starting from 1 in 2 s 305 ms

Which is a difference of more than 2000x. The network costs completely swamp the complexity of the actual statement.

So two things to draw from this: make sure you understand the difference between how long it takes to plan and execute a statement on the database vs how long it takes to return the data to the client; and this is why you want to be as selective as you can about what you return to the client. This is why SELECT * is bad (well, one of many reasons). If you can, only returning the data you actually need (both rows and columns) can make a significant difference to the speed of your queries and thus application.

The next main topic I want to mention is about the cost values you see in EXPLAIN output.

cost values should be considered unit-less, and should only be used to compare queries doing similar things, and only against the same database instance. Costs are calculated using many different things, such as the data you have in your database, the statistics it has gathered about that data, the parameters in that instance such as work_mem, etc. You cannot compare costs directly from dev environment and production, they can be two completely different things, even if they look comparable.

What you can do is compare a query to a different version of the same query (against the same instance) to see if you are improving things or not and on what scale. You can also compare them inside of a single query plan to understand the highest cost parts of your query.

But don’t get hung up on the cost numbers themselves. Use them as a relative measurement, they are not hard and fast numbers.

Also you may need to run your query against the production database to get accurate information like you see in your application. This is a sliding scale - sometimes you can get a good idea just from a development database, sometimes you need a bit more realistic example from a staging or similar server that has a copy of the production data, but sometimes you need to go to production to really see representative data. Another thing to consider is if you use read-replicas and primaries, you may need to go to whichever is going to be the place your query is actually going to be executed against in your application.

As far as what you are looking for when reading EXPLAIN output, that is a much larger topic. But the place to start will always be to look for Seq Scan. Nine times out of ten that is where you should focus your efforts. The one time out of ten it might actually be faster than an index because the table is small or the indexes aren’t very useful, but most of the time you want to be using an index. Ideally you want an Index Only Scan. This means that it was able to get all of the data from the index and didn’t have to go to the actual table. Next best is an Index Scan.

The next thing most common you are looking for when trying to improve query performance is Nested Loop. These will be places where things cant be done in a batch and has to be looped over. The fix here isn’t always straight forward, but oftentimes is a matter of doing a better filter earlier in the execution of the query.

This information is of course nowhere near exhaustive, just a place to start. I am far from an expert in understanding query plans, I learn more all the time. If you need more you should look at resources such as use-the-index-luke.com, or use explain visualization tools such as explain.dalibo.com, PEV, depesz or pgexplain.dev

But like most things the best thing you can do is practice, play with queries and explain output and keep reading to understand what is happening. In my experience you can generally get great performance with a little work in most cases.

Postgres pivot table using JSON

Sun, 15 May 2022 12:00:00 +0000

Something I have a need to do often but can be difficult to do at times in SQL is to create a pivot table. As an example imagine wanting to see customers and their revenue by month. It is straightforward to create a normal data set where the dates are the rows and you have a revenue amount for each. Something like this:

dte	total
2022-01-01	22030
2022-02-01	22753
2022-03-01	0
2022-04-01	9456
2022-05-01	7798
2022-06-01	38278
2022-07-01	18736
2022-08-01	6794
2022-09-01	21033
2022-10-01	28576
2022-11-01	10172
2022-12-01	41901

But you quickly come up to two obstacles as you try to take it further - you either want to have the months as columns like this:

jan	feb	mar	apr	may	jun	jul	aug	sep	oct	nov	dec
22030	22753	0	9456	7798	38278	18736	6794	21033	28576	10172	41901

or you want to see multiple customers, which as a column can be difficult, or even harder is having the months as columns and the customers as rows:

cus_id	jan	feb	mar	apr	may	jun	jul	aug	sep	oct	nov	dec
1	0	10170	0	5399	0	14821	7927	0	14	15466	3675	14447
2	22030	12583	0	4057	7798	23457	10809	6794	21019	13110	6497	27454

The term for this is pivot table - which is something you may have done many times in Excel or other spreadsheet applications.

But this is difficult in SQL, because SQL requires you to have a static column list. You can’t ask SQL to give you whatever columns are necessary, you must declare them in your query. (SELECT * may seem like an exception to this rule, but in this case the SQL engine still knows what the columns are going to be before the query is executed).

Luckily Postgres gives you a way around this. If you want actual columns you still have to specify them, but the hard part of aggregation into these pivoted columns and rows are made much easier. The key is using the JSON functionality, which allows you to represent complex values in a single cell. It allows you to aggregate the values into what really represents multiple values, and then pull them back apart after the fact.

Here is an example of what this looks like:

WITH columns AS (
  SELECT
       generate_series dte
     , customer_id
  FROM generate_series('2022-01-01'::date, '2022-12-31', '1 month')
  CROSS JOIN
    (
      SELECT DISTINCT
        customer_id
      FROM
        invoice
    ) AS customers
), data AS (
  SELECT
      columns.dte
    , columns.customer_id
    , SUM(COALESCE(invoice.amount, 0)) total
  FROM columns
  LEFT OUTER
    JOIN invoice
    ON DATE_PART('year', invoice_date) = DATE_PART('year', columns.dte)
    AND DATE_PART('month', invoice_date) = DATE_PART('month', columns.dte)
    AND columns.customer_id = invoice.customer_id
  GROUP BY
      columns.dte
    , columns.customer_id
), result AS (
  SELECT
    customer_id
    , JSONB_OBJECT_AGG(TO_CHAR(dte, 'YYYY-MM'), total) pivotData
  FROM
    data
  GROUP BY
    customer_id
)
SELECT
    customer_id
  , (pivotData->>'2022-01') "jan"
  , (pivotData->>'2022-02') "feb"
  , (pivotData->>'2022-03') "mar"
  , (pivotData->>'2022-04') "apr"
  , (pivotData->>'2022-05') "may"
  , (pivotData->>'2022-06') "jun"
  , (pivotData->>'2022-07') "jul"
  , (pivotData->>'2022-08') "aug"
  , (pivotData->>'2022-09') "sep"
  , (pivotData->>'2022-10') "oct"
  , (pivotData->>'2022-11') "nov"
  , (pivotData->>'2022-12') "dec"
FROM
  result
ORDER BY customer_id

Which gives results like we were after above.

If you would like to see how this is done, I have an interactive fiddle you can play with that shows you step by step how each of these parts work:

https://dbfiddle.uk/?rdbms=postgres_14&fiddle=39e115cb8afd6e62c0101286ecd08a3f

This example is using PG 15, but this functionality works all the way back to PG 9.5.

This query is also a great example of using generate_series to generate a set of data to join against, so that you can find any holes and represent all the data points you need (months in this case), even if there is no actual data for that point.

In conclusion, the JSON functionality built into todays relational databases are more than just schemaless data stores and complex values in cells. They can also be powerful as intermediate steps to help you manipulate and transform your data in useful ways.