I’m fairly active on StackOverflow, so when people started answering questions by copy-pasting from ChatGPT, I noticed. Unlike the AI generated code shared on social media (which appears to be cherry-picked), in actual use, the ChatGPT answers were nonsense. And so, StackOverflow banned its use.

The problem was not that ChatGPT was generating nonsense — the problem was that people were posting the nonsense without vetting it. That’s partly because the answerers were not even trying (they thought it was fun), but also, I would guess that many of them didn’t know if the answer was good or not. The answers were obvious nonsense to experts, but less so to beginners or laypeople.

Right now, AI generated code can’t be used without an expert. But is it useful if you are an expert?

Two years ago, I said that I wanted a Robotic Pair Programmer. In that post, I made some suggestions for what I’d want:

One way that seems fruitful to me is rare API calls. There will be times when I am using an API that appears very infrequently in the corpus or my own repositories. In that case, it should infer that I probably need more help than usual and offer up a tutorial or the top Stack Overflow questions.

And …

Another trigger might be my new comments. If I comment before I code, then it should be interpreted as a search query:

// Parse the JSON I get back from the data task

That should bring up links to likely API classes in the help pane (just like it would if I already knew the class). Maybe offer up imports to auto-add. Maybe offer a snippet.

One important thing is that this needs to be just in the IDE, not a chat interface. And last March, GitHub Copilot was released as a VSCode plugin. I ignored it back then, but seeing how far ChatGPT had come made me think that Copilot had a chance of being good.

The best thing about Copilot is its UI. It feels like autocomplete, but offers up more than usual autocomplete would. Sometimes it can complete a line — sometimes it can give you a few lines. In either case, since IDE users are already used to this interaction, it doesn’t get in the way. It’s also fast, which is vital.

It also does the comment trigger I wanted — meaning, I can comment my intention it will offer up snippets. Many of these are useful.

One worry I had in 2021 was that a system like this would offer bad suggestions often. And, that’s the main drawback to Copilot. Much of what it suggests is wrong. Some of it is scarily accurate, and the rest is in-between — not right, but still helpful. After a few weeks with it, I can’t decide if this is the right balance. I haven’t felt the need to turn it off, but also, I’m not sure I’d miss it once it was gone.

My main reason for trying it out was that it feels obvious to me that this is the future of programming. I program mostly for myself, so I really want the productivity gain. It costs $100/year, and I think that’s a no-brainer, because it pays for itself pretty quickly in productivity gains. If you use VSCode and one of the languages it supports, I’d recommend trying it out.