Category Archives: Software Development

Extending GraphQL Code Generation (Part II)

In the last post I said that even though I had used code generation throughout my career, the expressiveness of modern languages had reduced my usage. But, I had found that connecting code to external systems was better if I generated a type-safe API layer.

GraphQL Code Generation (which I mentioned in the last post) does this for GraphQL client code. You provide your queries and the endpoint to your server, and it will validate the queries and use the server’s schema to generate type-safe Typescript wrappers to use to execute those queries, including react hooks.

It doesn’t go far enough for me, but since it supports plugins, it was easy to add what I wanted. All I wanted was simpler, non-hook, type-safe functions to make GQL requests. The hook variants are very convenient when you are just populating a screen, but I had times when I wanted to make GQL requests and didn’t want to handle the result in the React lifecycle. This was even more true for mutations.

So, as a first step, I made a non-type-safe function to do any mutation given an Apollo client and document (NOTE: DocumentNode is a type that represents a GQL query).

export const gqlMutate = async (
  apolloClient: ApolloClient<object>,
  mutation: DocumentNode,
  responseProperty: string,
  variables: any,
  onSuccess: (response: SuccessResponse) => void,
  onError: (err: string) => void
): Promise<SuccessResponse> => {
  try {
    const response = await apolloClient.mutate({ mutation, variables });
    // handle response and call onSuccess or onError
  }
}

I could just stop here, but then when I write a call to gqlMutate, I get no code completion or type-checking.

But the GQL code generator actually generated concrete types for all of my GQL queries, so I could write a wrapper, like this:

export async function requestCreateOpening(
  apolloClient: ApolloClient<object>, 
  variables: gql.CreateOpeningMutationVariables, 
  onSuccess: (response: gql.SuccessResponse) => void, 
  onError: (err: string) => void): Promise<gql.SuccessResponse> {
  return await gqlMutate(
    apolloClient, gql.CreateOpeningDocument, 
    "createOpening", 
    variables, 
    onSuccess, 
    onError);
}

Which works great, but I have hundreds of mutations and I would have to write this code over and over and keep it up to date if anything changes. But, literally everything I need to know to write this is in the GQL query document. The code generator already generated this constant and type for me.

export type CreateOpeningMutationVariables = Exact<{
  teamId: Scalars['Int'];
  name: Scalars['String'];
  body: Scalars['String'];
  whatToShow: Scalars['String'];
}>;

export const CreateOpeningDocument = gql`
    mutation CreateOpening($teamId: Int!, $name: String!, $body: String!, $whatToShow: String!) {
  createOpening(
    teamId: $teamId
    name: $name
    body: $body
    whatToShow: $whatToShow
  ) {
    success
    error
  }
}
    `;

It’s not a stretch to get it to generate the wrapper around gqlMutate as well.

Normally, the drawback is having to parse the source language, in this case GQL. But GQL code generator already does that and already generates a lot of useful types for you. All you need to do is loop through your GQL documents (which are types) and write out the code. For details see the plugin documentation.

It’s too hard to explain my generator (write me if you need more help). But, if you are calling GQL with Apollo and typescript from your front-end, consider code generation. And, if you find yourself writing non-type-safe wrappers, know that you can extend it with plugins to make them type-safe as well.

Why I am Using Code Generation Again, Part I

In the nineties, I used Rational Rose and Perl to generate a lot of C++ code. I’m not sure if I knew the term at the time, but we also created DSLs (Domain Specific Languages) that probably fell into the “… poorly specified, half-implemented Lisp” trap.

Back then, I would say that I used code generation to overcome the limited expressiveness of my programming language. There was a lot of boilerplate in C++ (and later Java and C#) that I didn’t want to write—I didn’t even want it to be part of the source. I considered the DSL to be the source in these cases.

When I moved onto iOS development, I used code generation sparingly. It somewhat complicates the build process and the Swift language designers prioritized expressiveness enough to make it less necessary.

But, recently, I’ve begun to use code generation more while using Typescript. Typescript is a lot like Swift, so it’s expressive enough, but the main difference is that I am using Typescript to access things that don’t have types in their built-in API (SQL and GraphQL), but do have an underlying type system.

I decided to use TypeORM and TypeGraphQL, which allow me to define types that map onto the database and are passed around in the the GQL, and so on my backend, I have a strongly typed API to both. These systems are based on “decorators”, to tackle the problem of boilerplate code. But TypeORM generates the SQL to build the schema and each migration as I add to it, so there is code generation as well.

This is great for the backend. My server ends up exposing a GQL API which can also provide a schema. But, I still have the problem of having a typed API to GQL from my front-end. For this, I ended up using GraphQL Code Generator, which can connect to a GraphQL server, read the schema and your GQL queries, and generate a nice typed API, including React hooks for fetching your data.

Even with all of this, I didn’t think of myself as generating code. I was not writing parsers or generators—I was just using a system that happened to generate code. But having this system in my build flow already (and the fact that GQL code generator can have plugins) made it inevitable that I would eventually be generating my own code.

GQL Codegen is already doing all of the heavy lifting of getting the schema and parsing it—you can write a new plugin in a few lines of code. Once I wrote the plugin, I could delete hundreds of lines of boilerplate code around doing GQL mutations without having to resort to typeless wrappers. I’ll provide some details in Part II.

Seeing Code

There’s a saying that code is written to be read by programmers and incidentally to be run by a computer. I would argue that that doesn’t go far enough. The goal shouldn’t be readability, it should be to make it so you don’t have to read it.

When I have been working with a codebase for a while, I don’t need to read the code all of the time. I see it. I’m not perceiving it serially as lines of code—it’s more like seeing it all at once.

Some of this comes from familiarity. It’s in my head already. The code itself is a reminder. It’s something that comes from flow and extended time with the code.

But, it’s mostly a function of the code itself. Some code is easier to perceive this way.

What helps me is if the code is made up of memorable and predictable chunks. In a seeable codebase, I read one file and just know how the others would look without having to read them. When I open another one, it looks how I pictured it.

This idea is related to System Metaphor from XP

The system metaphor is a story that everyone – customers, programmers, and managers – can tell about how the system works. It’s a naming concept for classes and methods that should make it easy for a team member to guess the functionality of a particular class/method, from its name only.

and Convention over Configuration from Ruby on Rails.

The same goes even when you understand how all the pieces go together. When there’s an obvious next step for every change, we can scoot through the many parts of an application that is the same or very similar to all the other applications that went before it. A place for everything and everything in its place. Constraints liberate even the most able minds.

and the Principle of Least Astonishment

The principle aims to leverage the existing knowledge of users to minimize the learning curve […]. In more abstract settings like an API, the expectation that function or method names intuitively match their behavior is another example. This practice also involves the application of sensible defaults.

Astonishment is perhaps the best way to perceive code that resists being seen. It may be perfectly readable and understandable, but it should also already be predictable. If it surprises you, then you should ask yourself why.

The Unreasonable Effectiveness of Mathematics in Programming

(with apologies to Eugene Wigner)

My college didn’t have a CS major, but they let me put together one under a general engineering program. To fill up the requirements, I had to take a lot of math, which has been more useful than I expected a programmer.

I didn’t seek out jobs that required a lot of math. I optimized my job search around small companies doing product development for B2B, and didn’t care much about the specific technology they used. But, I was comfortable with the math, so it made my life easier.

The first eight years of my career was in FinTech, and the software I wrote was a nice UI around a lot of math. The core concept was options pricing (probability and statistics) and the sensitivity of that price to its inputs (calculus and differential equations). To do risk analysis, you have to build up huge matrices (linear algebra) for various purposes. Our company employed mathematicians, so we didn’t do the research, but we had to understand it to work on those parts.

Later, I contributed to a patent related in spreadsheets where graph theory was important. I also implemented numerical differentiation and root-finding algorithms as a way to run expressions backwards (numerical analysis and calculus). That patent expired, so I am reimplementing it in Swift and Typescript.

In 2005, I did a consulting project to implement a distributed monte-cargo engine for a decision support system. I would not have won this bid if I did not understand the math behind the engine.

From 2006-2013, I worked at an image processing tools vendor. This job was the closest to pure CS that I have had, and there was a lot of math, specifically linear algebra, but also some numerical analysis.

Every front-end position I have had uses at least a little linear algebra (for affine transformation). It’s not like you are doing the matrix multiplication yourself, but you’ll understand the more complex compositions better if you understand them. For example, if you know that matrix multiplication is not communicative, you’ll get why the order of the transformations matters.

Nearly every programming job now requires you to understand the analytics data that the software generates and to do statistical analysis on it. Forming a hypothesis and getting evidence to support or reject it is essential. At a bigger company, you might have a data-science team to do the heavy lifting, but it really helps if you can do it yourself—you also want to be able to read and understand their reports.

If you really want to go deep into the type theory behind type-safe languages (like Swift or Typescript), you have to understand set theory and maybe even HoTT. You don’t need it to use these languages, but if you had interest in compiler theory or implementing a language like this, it would help. Understanding set theory also helps with understanding relational databases.

When I was trying to find a Swift implementation of numpy a few years ago, I ended up finding Surge and contributed Eigen decomposition to it. I had to relearn it, but I would not have even tried if I hadn’t touched linear algebra since college.

Games are essentially physics simulators, which are ultimately math engines. I only write simple games as a hobby, but even for pong, I had to write vector math functions.

And, although I think my career has used a somewhat typical amount of math, there are programming jobs that require a lot of math. A deep neural network’s guts are a calculus and linear algebra engine (see this video from 3blue1brown). As I mentioned, data science makes heavy use of probability and statistics. I learned in Quantum Country that the “programming language” of a quantum computer is based on matrix multiplication with complex number entries. And while writing a game is a reasonable amount of math, writing a game engine is much more than that—and as more things (the blockchain, games, machine learning) have used the GPU for computing, the way you think of implementing solutions in those domains is more mathematical.

To be clear, you can do a lot of programming without understanding any of the math behind it. But, I have found it more enjoyable to go a little deeper, and it makes it easier to develop intuition about the higher level concepts.

So many choices, Part 2

Yesterday, I lamented that web development hadn’t come to any consensus on tooling. programming language, basic UI framework or anything else.

I was talking about front-end development, but it’s true on the back-end as well.

Here, I’m not so sure if we could ever get consensus. On the front-end everyone is running in a browser and ultimately has to provide HTML, CSS, and Javascript. I was hoping that would have driven the industry towards one obvious choice.

On the back-end, everyone has been using their tool of choice for decades, and using Javascript (via node) is a good choice now, but it’s not close to universal. And if you are only working on the backend there isn’t a requirement to know Javascript at all, so there are lots of teams that don’t care about matching languages.

For me, using one language on both the front-end and back-end is such a compelling benefit, that I’ve moved to node for anything new. In practice, I haven’t had much code sharing, but I am flipping back and forth so much while developing, that just having to have one language in my brain is reason enough.

And since I use GraphQL, again Apollo is an obvious choice.

Before this, I was using Django and Python for my backends, and I like working with an ORM for SQL data, so I decided on TypeORM. Here there seemed to be 1 or 2 other viable options as well. TypeORM can be DB engine agnostic (depending on what you use), so setting up an in-memory, sqlite version for unit testing is pretty easy.

With Apollo and TypeORM, TypeGraphQL is a nice addition. It lets me have a DSL that describes entities in one place that generates my DDL, queries and also encodes them for GQL/Apollo automatically. If you don’t do this, a lot of your GQL code isn’t going to be type-checked.

My backend is essentially just a DB with GQL in front of it right now, so there isn’t much else to decide. Since I use the same language as my front-end, I can use eslint, prettier, and jest here as well.

The one thing I miss from Django is the automatic admin console and user management. In the end, I do think my admin interface needs to be more custom, but it’s useful at the start until you get around to making a real one. I looked at AdminJS, but unfortunately I had already committed to some TypeORM choices that it couldn’t support. If I thought I’d be on AdminJS long-term, I might have backed out of those choices.

In any case, the choices on the back-end mostly depend on what language you choose. As an iOS developer, I never really had the option of using one language for the full stack—I usually want to make native apps and Swift isn’t ready for the server-side in my view. It’s nice to have this option, and would make me reconsider ReactNative if I need a mobile app for this project.

So many choices

Up until about March this year, I was primarily an iOS developer and had been for a while. But, this year I decided that the things I want to build would be best built on the web, and so I started to learn modern web development. I was struck by the lack of consensus about what that means.

My goal is to build a real web app, not just to learn. I already know HTML/CSS/JS going into this. If your goal is to get started or just to learn, I’d start with HTML/CSS/JS and maybe React or Vue and not worry too much about the rest yet.

Right now, I’d say, if you want to be an iOS developer, you download Xcode, write in Swift, use UIKit for the UI and standard components and libraries and you can build professional apps. There’s built-in support for most things that you need, and the Apple implementations are excellent. All of the apps I have worked on are not much more than this.

On the web, I’d say the one consensus choice is to use VSCode, but then it starts bifurcating quickly after that.

The default language of the web is Javascript, but you can use anything that transpiles to it. I have decided on Typescript, but I think Clojurescript is a reasonable choice. There are probably a few others.

At this point, you basically have nothing. The browser provides an HTML renderer, very basic components, and a limited library. You are going to need to add dependencies to do anything beyond the basics (you had to do this even to get Typescript).

The first choice I had to make was what UI framework to use. I chose React, but people I trust also use Vue. There are few others that seem reasonable, but React seemed a good fit for me. By the way, if you want to get started with React, I recommend Pure React, which is a book explains React in isolation.

It might be overkill, but I decided to use Redux to manage state. React has something built in now, but I like things that have been around a while.

I also needed a design system — something with ready-made components. I didn’t find anything that great, but chose Material-UI in the end as it seemed the best supported.

I wanted to like Microsoft’s Fluid, but I couldn’t get it to look right for me. I also could not get Ant to work at all (I’m a newb at this). Every big company seems to promote one of these — my old employer, Atlassian, has ADG, which looks ok. SalesForce has one too. But, they all seem to have a different idea about what a reasonable amount of markup is for default usage. Material-UI is usually a simple tag with a few attributes to get something on the screen — these other systems are way more complex. There is also a vibrant commercial market for components. I will probably revisit this after my MVP is done.

I decided on GraphQL as my API basis, and so I got another easy choice: Apollo. I also added GraphQL codegen because it can turn GQL into a Typescript API (with typechecking).

It doesn’t stop there. I have eslint, prettier, and jest as my code quality tools. I think those are default choices, but there were others. In iOS, code formatting and unit testing are built in.

And this is all just to get something minimal working. I have also had to add in some other components for specific features.

I have also checked in with people in my network that do this for a living, and I haven’t found two that have made the same choices. Except for VSCode — that does seem universal.

Tomorrow, I’ll talk about the backend, but there is no comparison with iOS, since you’d have to decide on this as well for a server backed app.

Outcome Story Templates

In yesterday’s post about Outcome Stories, I don’t think I quite got the story right.

I suggested changing the story from “As a writer, I want to publish posts, so I can share it with readers.” to “To get readers, a post must first be published.” I do think this does put more emphasis on the goal because it’s the first few words of the story.

But, it would be better if it were the subject of the sentence. The subject is what we’re talking about, and in my fix, we’re talking about the post, not the outcome.

So, tweaking it more, you could say “Getting readers requires that the post is published.” Maybe this is semantics. The more important thing is that the goal is early in the sentence, but if it can also be the subject, it will carry more weight.

But, if we go down this route, maybe we can develop a template around it

  • [Outcome] requires [the object] to [have a state]
  • [Outcome] requires [the persona] to [do an action] to [an object]
  • [Outcome] requires [the persona] to [do an action] to [an object] when [a trigger happens]

So, we end up with: “Getting readers requires the author to publish the post when they are done writing it.”

And again we’re clear that this is necessary, but not sufficient to accomplish the goal.

Outcome Stories

User Stories are centered around the user persona, which seems like a good idea, but it would be better if they were centered on a business outcome.

Consider the Publish button in WordPress.

One way to write a specification for this is to mock that screenshot I pasted above in Figma and say what the button does when clicked. Perhaps you’d use the typical User Story format and add “As a writer, I want to publish posts, so I can share it with readers.”

That works well in the sense that the production software will probably do what was specified. But, we’ve lost an opportunity. The story is focussed on a persona, but doesn’t emphasize the goal. I’m not even sure the persona matters at all here.

And just like no one wants a drill, I don’t want to publish. I want to influence readers. Others might want ad revenue or newsletter subscribers or to sell books. So, can we take the opportunity in this story to remind everyone about the business goals? The final state is not a published post.

It seems like a small tweak, but I’d word it “To get readers, a post must first be published.”

And now it’s obvious that this is insufficient. Yes, we need a Publish button, but that won’t accomplish the goal. This wording begs for more features to support the actual goal. It also implies that the metric for the success of the Publish button is the number of readers not (for example) the number of published posts. How would the design of publishing in WordPress be different if it was centered around the business goal of the users?

Publishing should lead directly to distribution, but in actual WordPress, the end of the publish step is a button to copy the link.

I guess I’ll go tweet it myself.

An Unnamed Programmer in Peopleware is One of my Heroes

I read Peopleware early in my career and revisit it every few years. Yesterday I wrote about what they meant about 10x programmers, and while doing that, I looked for this excerpt, which I think about all of the time (emphasis mine).

One of the upper managers buttonholed me to request that I assess […] his project staff. He was particularly curious about one woman. It was obvious he had his doubts about her: “I don’t quite see what she adds to a project; she’s not a great developer or tester or much of anything.”

With a little investigation, I turned up this intriguing fact: During her 12 years at the company, the woman in question had never worked on a project that had been anything other than a huge success. It wasn’t obvious what she was adding, but projects always succeeded when she was around. After watching her in class for a week and talking to some of her co-workers, I came to the conclusion that she was a superb catalyst. Teams naturally jelled better when she was there. She helped people communicate with each other and get along. Projects were more fun when she was part of them.

Peopleware

They go on to say “The catalyst is important because the project is always in a state of flux. Someone who can help a project to jell is worth two people who just do work.”

I don’t know who she was, but this programmer was one of the most influential role models of my career. I constantly ask myself what would she do. My post about How Senior Software Developers Think is my take on it, but it’s much more than that. Coding is certainly an important part of being a software engineer, but most projects can technically be done by a wide range of coders. The other skills, the work that makes the project succeed, are a lot more rare in my experience.

Waterfall is a Myth

I think everyone who programs has heard the story of the failure of the waterfall process.

Back in the olden days, to build software someone would first write up a document with all of the requirements. Then, the software architects would design a system and produce a document with that. And, then, I guess there would be some kind of implementation plan and that would be completed. Then, the programmers would code it up. And finally, the testers would test it. Then, you ship it.

It looked like this:

I’m going to say that I think this never happened in real life. I doubt anyone seriously tried it for anything big. Or, if they did, I hope they failed super-fast and started over.

This diagram comes from a 1970 paper called Managing the Development of Large Software Systems by Dr. Winston Royce. Right after this figure, he says: “I believe in this concept, but the implementation described above is risky and invites failure.”

Then he goes through a bunch of fixes to the process and ends up with:

The process as described seems completely reasonable to me (see the original paper) . It features

  • Code review
  • Automated tests
  • DevOps manuals
  • The idea of a spike implementation to learn and de-risk the real one
  • Including Customers in the process (every circle icon in the figure)

So, even in 1970, it was known that waterfall could never work, and the term was only coined later to describe Royce’s first diagram, which he had immediately rejected.

I think waterfall only really existed as a concept to immediately reject and fix, not a serious suggestion for what to do for a large software project.