The benefits (and pitfalls) of GitHub Copilot

Most organizations we talk to are considering adopting AI-assisted coding tools, with coffee table discussions by developers already leaning heavily into the topic for a while (with some potentially using a personal license for their work). But, there are some concerns about intellectual property leaks–especially if organizations aren’t using business or enterprise licensing with AI-assisted tools.

GitHub Copilot is an AI-assisted pair programmer tool trained on billions of lines of code, mostly from open-source projects. Copilot turns natural language prompts into coding suggestions across dozens of languages, suggests auto-complete for code, and provides other features, like a chatbot (similar to ChatGPT).

As with any technology, team leads and scrum masters are hesitant to introduce possible distractions mid-sprint or release. There’s rarely a perfect moment for adding new tools into the development process, which often calls for a deeper look into the following topics:

• What changes does Copilot bring to the everyday life of software professionals?
• What are the risks of using Copilot, and how can they be mitigated?
• What short and long-term benefits does this new technology bring?

How does Copilot change the everyday life of software professionals?

Copilot isn’t just for application developers. Anyone who works with “code” can use it, such as data engineers or infrastructure administrators. There are plenty of use cases, and communities of software professionals are only starting to dive into the opportunities Copilot provides. It can also help us understand legacy code or any languages we’re not familiar with. Translating between coding languages is an area it’s especially powerful in. 

Based on our experiences within Eficode and with several of our customers, we recommend using AI-assisted tools to improve your existing ways of working. Copilot isn’t great at providing complete solutions or reverse engineering business logic from existing codebases. However, the chatbot that’s integrated within the Copilot user interface does add capabilities in designing the solution and understanding its end-user impacts. Software developers don’t become architects with these new tools, even though the initial feeling of force is strong when using AI-assisted tools for the first time. All the traditional software development roles are needed now and in the future.

Copilot excels in creating boilerplate code and reminding you of complex syntaxes. That alone will allow several hours in a week to be directed into more creative work. Not only that, but the possibility of creating a skeleton for your tests is absolutely central to overcoming those daunting moments of writing tests for your code. In the ideal case, you will begin by shaping your test scenarios—implementing the elusive and powerful test-driven development (TDD).

On a larger scale, improving quality is even more interesting than increasing speed, which is the primary thought when considering AI-assisted tools. Coming up with tests the developer would not have thought of directly improves product quality. It also gives developers new viewpoints on the product under development, as well as their own notions about what quality means and how it’s assured.

Mitigating possible risks from using AI-assisted tools

Since you’re suddenly able to complete your code functions in record time (and even write tests for them simultaneously), the feeling of omnipotence might be blinding. This is especially true when, in the safety of your own corner of a repository, everything seems to work perfectly, and tests keep on passing.

A real risk here is not understanding what you are creating. You may accept coding suggestions and tests that implement a different thing than what your solution needs. It doesn’t help to create something fast with enough test coverage if the code is doing something other than expected. Sometimes, Copilot provides solutions that are just wrong. It’s easy to get stuck in a loop in trying to make a suggestion work when the approach is misguided.

AI-assisted solutions should be taken into use in a controlled way. Guidelines and examples relevant to the company or organization are very powerful. A recent GitHub study reported that 81% of developers think AI tools will help them collaborate better.

When less experienced developers commit code, it makes sense that a mentor oversees if we have implemented the right thing, confirms that testing is sufficient, and ensures the developer is ready to explain the code they’ve created to others when needed. Everyone is responsible for the code they commit, even if it is mostly produced by an AI assistant. 

Some ideas to avoid risks:

• Paste the code function created back into the chat module of Copilot and ask it to explain it to you in layman’s terms.
• Ask the chatbot to explain code like a cooking recipe. It often sums activities up very effectively.
• Ask the chatbot to suggest tests for the code, including edge cases, negative testing, or mocking.
• Request suggestions to improve the style of the code. The tool will most likely write the improved code in practice, which you can diff and learn from.
• Take a moment to think about really good names for your function parameters, as Copilot can better understand your intent.
• Contact a real person to walk through any major body of your work and how you used AI-assisted tools to create it before passing it forward.

In general, our experience tells us that senior technical engineers can take Copilot into use without much introduction or external support. The time spent in taking it to use is won back usually within the same week.

For people just starting their technical careers, pairing up with a mentor will ensure that using Copilot doesn’t become a race against their former selves but rather a way to learn quickly and be more confident in making mistakes.

In short, anyone can take GitHub Copilot into use as long as they’re aware of the responsibilities that await them when pushing the produced functionality downstream. Be ready to walk through the solution when asked, and make sure you can explain each step and why you chose to implement it in that way.

What short and long-term benefits do AI-assisted tools bring?

The majority of the people we talk with aim at increased productivity and efficiency—basically producing value faster by using AI-assisted tools. Research on value chain analysis tells us that only a fraction of the time in delivering software to end users is spent in coding. If we consider even 20% of time spent in coding during a release, cutting 50% of that isn’t too significant in the overall picture.

A much more interesting area than productivity is Developer Experience (DX). The sheer enjoyment of work and the ability to learn while staying in the context of the program you are developing will definitely increase retention.

Developers in your organization are more likely to stay with you for longer if you provide them with tools that enable them to focus on the creative side of things and avoid the dull work. Most developers like designing solutions to novel problems, and Copilot helps leave time for exactly that.

We must take a holistic look at the software lifecycle to really get the short and long-term benefits of using GitHub Copilot beyond improvements to the daily lives of developers. In particular, we should push our wishlist for DevOps improvements forward. How can we provision test environments more quickly? Can we prevent problems from having different configurations between environments? What are the bottlenecks for release speed and quality? How will we ensure the outcomes we produce are valuable for our customers?

As for collaboration with product teams, can we ensure we’ve correctly understood a requirement and turned it into a matching technical specification? Combining the use of AI-assisted chatbot tools with coding assistants gives us a way to look at the challenges we’re solving in different ways. Writing technical specifications or rephrasing requirements with AI-assisted tools is very useful. Seeing the requirements from different angles can help us collaborate better and discuss their meaning with our end users.

In the short term, GitHub Copilot can help us learn while staying focused during our work in the integrated development environment (IDE). The ability to stay in the creative flow and avoid frustrating tasks will improve the days of our development teams. In the long term, using a combination of AI-assisted tools can help in almost every step of the design, specification, development, and delivery process. 

In this blog post, we haven’t even started to talk about operating software in production environments—an area that will eventually see the greatest changes. Telemetry data we get from production clusters, difficult to read and analyze by humans, can be turned into insightful configuration changes in our runtime environment, with humans using these tools in new, powerful ways.