Philip Potter

  • Keep database deploys separate

    Posted on 04 April 2018

    Context

    A while ago, I tweeted that you shouldn’t deploy database migrations at the same time as your app code. I thought I’d write something about why I feel this way, and the situations where I feel strongest about this.

    This post has the structure of an Architecture Decision Record, because context matters. I think ADRs are an excellent way to force you to think about the context in which you make a decision, and I’m hoping to use the format to think about the context in which my recommendation applies.

    The team that this recommendation applies most to has the following things going on:

    1. You are building a user-facing application, quite probably a web app for consumption by a browser or mobile native application.
    2. You have enough users that, whenever you deploy, most likely someone is using the app. Therefore, you employ techniques to deploy your app with zero downtime, such as deploying one app server at a time rather than all at once.
    3. You deploy your app multiple times a week (possibly even multiple times a day).
    4. You don’t have maintenance windows (for deployments or otherwise).

    Many teams, having read the Continuous Delivery book, will set up an automated deployment pipeline to ensure there is a clear, repeatable path for incremental code changes to go through various testing stages and end up in production. As part of this, they are probably using a database migration tool (such as liquibase or active record migrations) so that the database schema and data structure can be incrementally updated as the application is worked on. And in the final step of the deployment pipeline, the latest code and migrations are deployed to production in a single deployment stage.

    I anticipate problems with this setup.

    The main issue is that database migrations are inherently riskier types of change than code changes. Problems with database migrations might only show up when you deploy to production, for a number of reasons.

    A migration’s success is contingent on what data existed in the database previously. Development databases are often routinely emptied out between tests. You can trivially add a column with a constraint to an empty table, but a single record will cause the migration to fail. More complex migrations involving rewriting data from one form to another (string to int, or int to date) might fail on corner cases that only show up in “real world” data. More mundanely, a database migration might perform a lot of updates all at once, suddenly filling up a disk.

    Another problem with a production (or production-like) environment, that often doesn’t appear in development environments, is that you will at some point end up with an active user hitting a new version of your app while the old version of the database schema is still active (or vice versa). You are doing deployments in a way that avoids downtime, and whichever way you do it, you will have concurrent active requests hitting old and new code. Development and testing environments often don’t cover this case, because tests are run between deploys rather than during them, and so code is often not tested for schema backwards- or forwards-compatibility. In an environment where developers don’t see the hazards with creating commits which atomically change code and schema, they will do so, causing problems only when people try to use the system during a deploy.

    A final issue is that database changes are much harder to back out than code changes. There’s often no “reverse” migration script available. Even if you have written code to reverse the migration, you might not trust it, especially if there was a problem with the forward migration script. Even if the reverse migration was automatically generated, you might have irrevocably lost data already. (This is not an absolute rule: code changes can also lose data; but their risk profile is lower. It’s easier to lose data on a mass scale with an UPDATE IN statement than in a simple web request handler.)

    Decision

    We will not deploy database changes in the same step as code changes.

    This means that a single deploy pipeline stage should not deploy both code and data without a manual trigger in between the two. Each deploy should be smoke tested, to test that a code change without a data change doesn’t break anything, and similarly a data change without a code change also doesn’t change anything.

    This setup means that every step of the deployment process is verifying the backwards- and forwards-compatibility of your code against your database schema.

    There are a number of ways of achieving this. One simple change you could do is change your deployment steps to deploy code or migrations, based on a parameter in the build. I have seen this work successfully for GOV.UK’s deployments – their deployment step template has a parameter for code or migrations.

    Another way would be to have totally separate deployment pipelines for code and migrations. To the code deployment pipeline, the database becomes an external dependency (like any third-party service). Then database deployments can be explicitly scheduled on a slower cadence, such as weekly. John Allspaw has described this approach at etsy and flickr (see below).

    A very low-tech way to do this would be simply to have a convention that releases can either contain database changes or code changes, but not both, and rely on code review or release management to catch contraventions.

    Note that there are several things that are important to keep, even as we separate out code and database deployments:

    • database schemas should be built from a source-controlled definition, not hand-crafted
    • database schemas should be managed using a migration tool
    • database schemas should be deployed using a deployment pipeline, testing changes first in development and testing environments before being run in production

    Consequences

    Developers will have to adapt to this environment, and understand both that database changes are considered riskier, and that if they want to introduce database changes, they will have to consider how to orchestrate the rollout. A typical scenario would involve three steps: first, release a new app which can support the new schema when it arrives; second, deploy the database migration; third, release a new app that removes the old code for compatibility with the old schema.

    This is what developers should be doing anyway, because otherwise their code will cause problems for users when deployed to production. However in practice they either don’t do this phased rollout, or they sometimes forget. Under the new system, if they don’t write forwards-compatible code changes, their code deployments will fail smoke tests. As a result, there should be fewer instances of failures in production due to mismatched code and schema versions.

    Database changes will become more expensive to push through the pipeline. This is by design, but it does mean that there is a risk of perverse behaviour such as writing fixes in code because it’s easier, when a fix in the database would be more correct.

    This change will not fix everything about database migration riskiness. Trying to update every record in a large table to add a new column will still be slow and fraught.

    A brief survey of other people’s views

    I think it’s worth reviewing other people’s thoughts and experiences here too. Many smart people seem to disagree with me, including the Continuous Delivery authors.

    InfoQ recently published an article about including database migrations in the deployment pipeline, although I actually agree with most of the points they make: database schemas should be defined in source-controlled code, using a migration tool, and deployed as part of a pipeline. The schema should be defined by the application team, not a separate DBA team. The benefits of automation are incontrovertible. The only thing I disagree on is that code and database changes should not be deployed in the same automated step.

    On the flip side, John Allspaw has tweeted: Decoupling schema changes from deployment of the code making use of those changes had huge advantages at Etsy. He has spoken about this in this interview by Jez Humble. People ask him “if you deploy 50 times a day, how do you change the database 50 times a day?” and his answer is “well, you don’t”. At flickr, they had a window every other Tuesday to do database changes.

  • Dad Weeknotes, volume 7

    Posted on 25 February 2018

    Previously, on Phil’s dad weeknotes..

    Planning for return to work

    It feels like I only just started these notes but it will soon be time for them to end (as we know them at least): I only have a week and a half before returning to work. I’ve really enjoyed and valued the time I’ve had caring for Luke and it’s going to be a hard adjustment to go back to full-time work and not see nearly so much of him.

    Sonia and I have had some conversations about the new routine. One practical thing we have had to address is that the nursery doesn’t have enough space during March – Luke will be in for 3 days a week but we wanted 4 – so we need to work out how we will care for Luke in that month. Thankfully, I have plenty of annual leave left so I can take some leave to care for Luke on some of those extra days.

    We have tried to replan Luke’s morning routine. The current routine we’re in means that Sonia often doesn’t start work til around 9.30, because she has to look after Luke while I have my shower, and she has her shower afterwards. This has been okay for the moment but isn’t really sustainable.

    We have also had some difficult conversations about hazards that Sonia anticipates. One is that, with Sonia working from home, it may well fall to her by default to ensure Luke’s dinner is ready for him when he gets home from nursery.

    It’s important for us to keep talking about this. We won’t come up with the perfect plan on day one of me going back to work, but we can reflect on what works and what doesn’t and iterate our routine so that it works for all three of us, without placing unfair burden on anyone.

    Language development

    Luke has definitely shown some signs of developing language. He has already been babbling for some time, but recently he has started to seem more deliberate in his use of sounds. When I go and get him up in the morning, he says “Dad! Dad Dad Dad Dad Daaaad” with a variety of tones and inflections. This is exciting, but my feelings of pride are tempered somewhat because this week he has also said “Dad” when looking at: his mum, his granddad, and a piece of cucumber.

    One thing that is far more clear is that he is starting to recognise things that we say and respond appropriately. During mealtimes, Luke often focuses on his food and forgets to drink his water. When I say “have a drink, Luke”, though, he knows to pick up his cup and (hopefully) have a drink from it, or (all too often) see what sound it makes when he bangs it on the table, or turn it upside down and dribble the water out. We have some books which have a common item on each page (such as “That’s Not My Snowman” which has a mouse on each page, or “This Little Dinosaur” which has a ladybird, or “Wow! Said The Owl” which has an owl), and Luke has learned to go hunting for it when we say “Where’s the mouse, Luke?” or “Where’s the owl, Luke?”.

    Milk ladder

    As mentioned last week, we’ve started on the milk ladder. He’s done very well with the biscuits baked by grandma, and so we moved on to muffins. I baked some carrot and dairy-free cheese muffins (made with a specific amount of milk) and he has been having half a muffin and some biscuits with his breakfast over the past few days. So far, we haven’t seen any definite reaction, so this is really good news.

    Luke continues to drink some coconut milk, though not as much as we would like. He had about 40ml of coconut milk this morning, which isn’t too bad, and is far more than the oat milk we were trying before.

    Supporting mum’s work trip

    Luke’s mum Sonia returned to work at the start of January, and this week she needed to be in St Andrews in Scotland for some work meetings. Since Luke is still breastfeeding, this is somewhat complicated. We considered a few options for how I might look after Luke in London while Sonia went to St Andrews: I could substitute with formula while she was away, or Sonia could express breast milk and store it up in the freezer. However, as mentioned in previous weeks, we haven’t found a dairy-free (or extensively-hydrolysed) formula that Luke will reliably drink. We have also had significant spoilage problems with freezing breast milk – often half of the batches I defrosted would have gone foul in some way.

    As a result, by far the easiest option was for us all to travel to St Andrews. I’m still on Shared Parental Leave, after all, so I have nothing tying me to London, and if we all go then Sonia can continue to breastfeed Luke in the morning and evening and do her work stuff during the day. We can also stay with Sonia’s parents who can provide additional support with cooking, cleaning, and cuddles.

    In other house-husbanding news, I very much enjoyed sewing some buttons on to mine and Sonia’s coats while watching Star Trek Discovery.

    Nursery

    Luke missed his usual full-time days at nursery on Thursday and Friday, but I was able to take him in to nursery for an hour on Monday and Tuesday. There were 3 parents with babies in on Monday, which was slightly more than the nursery had chairs for, so on Tuesday they asked us to come in at staggered times. This meant I brought Luke in in the afternoon instead, which is something I’ve never done before. Normally I take him in for an hour or two between 9 and 11, but on Tuesday it was 2.30 to 3.30.

    It was almost like Luke was a different baby! When I take him in around 9, he usually stays very close to me, and even if he crawls off to explore or to play with a toy he’s spotted, he frequently checks to see I’m still there and still looking at him. But on Tuesday, he was off, crawling quite far away, to places which were almost out of site, not looking back nearly as frequently.

    It’s hard to know exactly what your own baby is like at nursery when you’re not there, when all you have to go on is how they are when you drop them off or pick them up. I worry about Luke, but this made me worry a little bit less.

    With the work trip to St Andrews, it means Luke is going to have quite a long break from nursery. We’re bracing for re-entry when we take Luke back. It’s probably going to take him some getting used to again.

    Quaker meeting

    On Thursday I took Luke to his first Quaker meeting, in [[http://www.quakerscotland.org/st-andrews][St Andrews]. It was a small meeting, with just 9 people (including Luke) and a dog. Luke behaved himself pretty well. I kept him strapped in his pushchair, and he sat and chatted away to himself. It wasn’t the most silent Quaker meeting I’ve been to, but I think people appreciated Luke’s presence anyway. The dog (called Gunnar) also made his presence known, occasionally walking around and making friends with those present.

    Two of the attenders were students who were attending a meeting for the first time. The welcomer, who had given them a brief introduction to the Quakers and a copy of Advices & Queries beforehand, explained to the students afterward that she hadn’t been to a meeting quite like that before either! 8 adults, a baby and a dog does make for a unique Quaker meeting experience.

  • Dad Weeknotes, volume 6

    Posted on 17 February 2018

    Previously, on Phil’s dad weeknotes..

    Health

    Last week, I wrote:

    Luke’s been having some funny days where he has a much lower appetite than usual. He just eats much smaller portions than we’re used to. But otherwise he’s absolutely fine: just as happy and interested in things as usual, if maybe a little subdued. He’s also been a bit snotty, so we’re hoping it’s just a cold or something.

    It was not just a cold. Luke had diarrhoea and vomiting on Monday morning. We called the doctor and were seen the same day: he checked Luke over and said it was a viral infection, and it should improve with Calpol, rest, and fluids, though we should call again if it didn’t improve. Luke slept a lot that day – I think he had 4 naps in total when he would normally have 2, and the naps themselves were longer.

    On Tuesday, when I got Luke out of bed and changed him, I saw he had a dramatic rash on his tummy and back. So we called the doctor again, and were seen that morning. The doctor took one look and said it was a typical symptom of a viral infection and that, although it looked dramatic, it wasn’t something to worry about. Luke himself didn’t seem terribly bothered by it: he was more active than on Monday and wasn’t itching at his rash.

    On Tuesday night, Luke didn’t sleep well. He was up multiple times, crying in his cot (which we hear loudly through the monitor). A note for non-parents on crying here: babies really do have different cries, and in this case it was the sort of cry which we didn’t feel needed a response from us. Normally he would get himself settled again and go back to sleep. However, this night, it happened several times.

    In the morning, we felt he was well enough to take in to nursery. It was almost exactly 48 hours since his vomiting spell, which is the minimum time to wait before returning him to such an environment. Sonia took him in and I had a bit of a break. However, in the afternoon, the disrupted sleep caught up with me. It was a real struggle to keep an eye on him, to give him the attention and interaction that he wanted and deserved. He was still recovering from his illness and was a little clingy; I just wanted to sit motionless and for him to amuse himself. There was a good moment in the afternoon where I put him down for a nap and went and dozed in bed myself.

    These sorts of moments are the ones I find the hardest: to be an adequate parent when you’re not functioning well yourself. But somehow we made it through the afternoon.

    Nursery

    Luke had his first full days at nursery this week: 9 til 4 on Thursday and 9 til 4.45 on Friday. I thought I would have loads of free time without him, but I spent most of Thursday catching up with laundry, cooking, and cleaning. At least on Friday I managed to get a lunchtime climb in at Vauxwall.

    Luke is settling in more and more. One of the things we have been a little concerned about is how Luke’s sleep schedule doesn’t align with the nursery schedule: Luke generally naps mid-morning and mid-afternoon, but the nursery structure has a single post-lunch nap. But the staff have been very flexible about letting Luke keep his existing schedule. And on Friday, he actually slept just the once, just after lunch.

    The nursery staff are really good about giving a debrief at the end of the day. When I pick him up, they take me through his day: when he slept, how long he slept, how much milk he had, what size portion he ate at lunch, how many nappies he got through, and maybe some highlights of what he did that day. This has been really valuable to understand how well he’s doing and what he gets up to when we’re not around.

    Milk

    Luke has cow’s milk protein allergy, so we currently avoid feeding him dairy at all. Luke has continued to drink coconut milk with variable enthusiasm. He drank 2 oz (about 60ml) on Thursday at nursery, but only tasted it on Friday. He still eats his afternoon coconut yoghurt, so that’s something.

    We are going to start a milk ladder: a structured process of reintroducing dairy to Luke’s diet to see how he reacts. We got the details from the dietician. Step 1 is a biscuit made with a dough that has a teaspoon of milk powder in it, and step 6 is drinking cow’s milk (or formula). The first few steps (biscuit, muffin, pancake) are homemade recipes so that the amount of milk can be carefully controlled. My mum is visiting this weekend and she has kindly offered to bake the biscuits for us.

    Food

    Since recovering from his bug, Luke has had a phenomenal appetite. We did a vegan vegetable curry which is perfect finger food for Luke, and he wolfed it down. At one breakfast he had 1½ weetabix and wanted bread as well. Today, we had a tuna pasta bake and he basically had an adult portion. We wonder if he’s going through a growth spurt of some sort.

    It’s nice that Luke has reached a stage where he is able (and insists!) to feed himself, rather than us having to spoonfeed him. It’s at least one thing that has gotten easier for us.

    Those things your parents told you to do

    The NHS recommends babies to have vitamin supplements. We have not been especially good at making this happen every day, and often gone days at a time without giving him his vitamins. But this week, we have managed to get more into the habit.

    Similarly, you can start brushing your baby’s teeth as soon as they come through, but we have not been good at doing this every day either. When we do manage, Luke wants to do it himself, with all the enthusiasm and inaccuracy one would expect from an 11-month-old. We were given a tip to have two baby toothbrushes: one for Luke to play with, and one to actually clean his teeth. Watch this space for whether this tactic actually works…

  • Dad Weeknotes, volume 5

    Posted on 11 February 2018

    I have successfully finished my fifth week of dadding. You may also want to see volumes 1, 2, 3 and 4.

    Shoes

    We bought Luke’s first pair of shoes this week. He hasn’t needed shoes before because he hasn’t started walking yet. He has been cruising for a while now, but this has been indoors. However, now he has started at nursery, and they encourage outside play, he really needs to be wearing shoes.

    We already had some hand-me-downs that were roughly the right size, which is what he’s been wearing at nursery for the past few weeks, but Luke didn’t look comfortable in them, and he either pulled them off or they fell off by themselves.

    We went to Clark’s and got his feet properly measured and bought a proper pair of crawling/cruising shoes. The difference was immediately apparent: he looked comfortable, didn’t try to take them off, and was happily cruising around the shoe shop bench straight away. I think they’ll make a big difference! It’s just a shame they won’t last long as he’ll grow out of them so soon.

    Nursery

    Luke has been progressing at nursery. Again, I took him in Monday to Wednesday to help him further acclimatize. On Thursday he stayed til 3pm again, and Friday til 4 – the latest he’s ever stayed.

    He’s been coming home covered in paint, pen, sand, all sorts. I think this shows he’s having a great time!

    Health

    This week, Luke had his 12 month health visitor check. It was actually one of the more fun appointments we’ve had. There were no needles or physical manipulation of joints. Luke had his weight, length and head circumference measured, then he was given some toys to play with, and the health visitor observed his reactions. The health visitor was clearly someone who loved her job: she was really good with Luke at keeping him interested in the examination and making everything into a bit of a game for him.

    Luke’s been having some funny days where he has a much lower appetite than usual. He just eats much smaller portions than we’re used to. But otherwise he’s absolutely fine: just as happy and interested in things as usual, if maybe a little subdued. He’s also been a bit snotty, so we’re hoping it’s just a cold or something.

    Milk

    We’ve still not found a great breastmilk substitute for Luke during the day. He didn’t take well to the wysoy formula (see volume 2) and we suspect he may also have a soya allergy. After taking advice from the dietician, we’ve been trying oat milk, but after some initial hopeful attempts, he’s shown less and less interest over time.

    This week, I tried to introduce coconut-based milk instead. Again, there was some early success – he managed around 60ml the first time – but I was struggling to get him to still be interested once the novelty had worn off. I have tried taking a drink from his cup myself, to show him how it’s done and to set an example, with varying success.

    Sonia made a breakthrough, though, when she offered him some milk in an open cup, rather than the sippy cup we had been trying. He seems to like drinking from an open cup, though he needs much closer supervision to avoid spilling it all! We shall see if his interest stays high or if it again wanes as the novelty wears off.

    Social stuff

    Once again, I got out to my Thursday night orchestra. With my new free time during nursery on Thursday and Friday, I managed to meet some friends for coffee and lunch.

    Sonia (aka mum) even managed to have three nights out: Monday, Tuesday, and Thursday. I’m really pleased we’re able to have some sort of “normal” social life.

  • Dad Weeknotes, volume 4

    Posted on 04 February 2018

    Slightly late, here are my notes from my fourth week of dadding. You may also want to see volumes 1, 2 and 3.

    New recipes

    I tried two new big-batch recipes, both of which Luke loved: lamb stew with root vegetables, and tuna pasta bake. The pasta bake did the whole family for three separate meals, and the lamb stew made 6 2-portion boxes for the fridge and freezer. I think both will become regular additions to the repertoire!

    Nursery

    I spent a lot of time at nursery this week! I took him in for an hour on Monday and Wednesday mornings, to get him more familiar with the staff, the routine, and the environment. Then Thursday was the big test: having had a bad time last Friday when we left him alone, we wanted to do everything to help him settle without his mum or dad there. We brought his pram to sleep in as something more familiar than the nursery mattresses, and we also brought his plushy duck comforter. And fortuitously, one of his NCT buddies started nursery that day too. The effort spent familiarizing him with the place seemed to pay off, and he managed 9.30am til 12 noon without too much drama. They both seem to have helped him settle in too.

    However, he was a very hungry boy on Thursday afternoon – eating a bigger afternoon snack than usual, and also eating a big dinner. At nursery he had had a “large”-sized lunch; I wonder if that’s still not as big as what he’s used to at home?

    I also learned my lesson from last week, and arranged to do something to keep myself busy rather than having nothing to do but worry about Luke. I went to meet an old friend for coffee and a catch up. She politely tolerated me showing her about 80 photos of Luke, for which I am most grateful. It was also comforting to know that our NCT friends were there for their own baby’s first day.

    On Friday, we tried 9am til 2pm. I dropped him off, and again went somewhere nearby and kept busy. He again did well, but he was incredibly tired when we picked him up. He had had 2 naps at nursery, but he had a third nap that afternoon – something unheard of for him! Having him so tired made the rest of the afternoon much harder work, but I still think it’s a good sign that he’s using up all this energy at nursery.

    Next week, I’m planning to take Luke in on the off days again (Monday to Wednesday) and hopefully he’ll stay a little bit longer on Thursday and Friday once more.

    Other stuff

    I’m struggling to find things to write about. The week has flown by, and although I tried to keep notes as it went, I just don’t feel like there’s much of note. There was plenty more routine housework: washing, washing up, that kind of thing. I took Luke to the Tuesday health visitor baby clinic to get some advice on some nappy rash. They also weighed him (he’s 11.10 kg, which is around the 90th centile). I found some time to do some filing, and noticed that our fixed-price gas & electric deal has expired, so it’s time to do some shopping around for that again. On Wednesday Sonia looked after Luke for the afternoon, and so of course I had a raging migraine all afternoon. We managed to do some sociable stuff. Sonia and I managed to go to our respective orchestra rehearsals on Tuesday and Thursday, we hosted Sonia’s book club on Monday, I managed to buy some climbing shoes at get back on the wall for the first time since 2016.

    I’m keeping busy, I suppose?