Planning and Task-based Approach to Software Development

Last updated on 2026-10-22 | Edit this page

Overview

Questions

  • How can I use the planning agent to structure software development before implementation?
  • What are skills and how do I create reusable skills for common development tasks?
  • What is the format and structure of a skill definition file?
  • Why is testing important and what is a unit test?
  • How do I write and run unit tests using pytest?
  • How do I review and improve AI-generated skills and tests?

Objectives

  • Use the built-in planning agent to develop a basic Python application
  • Create reusable Copilot skills for specific development tasks
  • Describe the format and basic configuration of a skills definition file
  • Describe the purpose of a unit test
  • Write and run a unit test to run within the pytest unit testing framework
  • Interpret the output of running pytest

In this episode, we’ll move beyond simple code generation to adopt a more structured, task-based approach to software development. Rather than asking Copilot to build our entire application at once, we’ll plan our work in detail, then break development into discrete tasks using reusable skills. This keeps you (not the AI tool) at the center of decision-making, with Copilot serving as an assistant.

We’ll learn to use the planning agent to decompose requirements into steps, create custom skills for common development tasks, and write tests to verify our work. Throughout, we’ll emphasize the importance of reviewing and refining AI-generated content before committing it to our codebase.

Using the Built-in Plan Agent

Creating a Plan

Instead of “one-shotting” the development of code using an AI coding tool, we’ve seen that a better approach is to plan and implement our code in a step-wise, incremental fashion. So how should we go about this?

One way would be to use the built-in VSCode plan agent that helps developers break down tasks into clear and actionable steps before writing code. Instead of jumping straight into implementation, it researches the requested task, analyses any existing project code in read-only mode, and generates structured step-based plans for features or for other code modification activities. In short, it aims to guide users through a thoughtful planning phase prior to coding to reduce errors and encourage better design and implementation decisions.

Let’s try it out now.

  1. Select Local from the Set Session Target dropdown.

  2. Select Plan from the Copilot mode dropdown in the chat panel.

  3. Enter the following into the chat:

    Create a command line tool written in Python that reads in a single CSV data file contained in the data directory passed as an argument, and creates graphical plots saved as PNG images to visualise the mean, minimum, maximum and standard deviation across each column. The tool must use Numpy for statistical analysis and Matplotlib for generating the plots. Use the existing virtual environment invenvfor managing dependencies with pip, and create a requirements.txt file to hold the dependencies.

  4. Press Enter.

  5. Answer any clarifying questions from the planning agent (e.g. which might include how to select the CSV file to process, or how to arrange the plots)

  6. Observe the step-by-step thinking and actions undertaken by the agent.

  7. When the planning agent concludes, select the option to Open in Editor

  8. When the plan appears in the editor, select it then select Keep.

  9. Save the file that appears by selecting Save as prompt file which should appear in the bottom right of the editing window, then confirming the filename in the top text bar by selecting it and pressing Enter.

You should find you end up with something similar to this, saved as a prompt file (either in the repository root or in the .github/prompts directory), although the content will likely differ:

MARKDOWN 

## Plan: CSV Statistics Plotter

Build a Python CLI that selects one CSV from a supplied data directory, calculates per-column statistics with NumPy, and saves a combined Matplotlib plot as a PNG.

**Steps**
1. Add a CLI with a positional data-directory argument and required `--csv` filename. Validate the selected headerless numeric CSV and report useful errors for invalid input.
2. Calculate column-wise mean, minimum, maximum, and population standard deviation (`ddof=0`), and print the results in a readable table.
3. Plot all four statistics against 1-based column labels in one PNG. Save it as `output/<csv-stem>-summary.png`, creating `output/` as needed. Do not modify files in `data/`.
4. Add `requirements.txt` with NumPy 2.5.4 and Matplotlib 3.11.2, matching the existing venv.
5. Add focused `unittest` coverage for the statistics, input validation, and PNG generation.

**Relevant files**
- [`.github/copilot-instructions.md`](.github/copilot-instructions.md) — Python style and read-only data/output rules.
- New [`analyze_inflammation.py`](analyze_inflammation.py), [`requirements.txt`](requirements.txt), and [`tests/test_analyze_inflammation.py`](tests/test_analyze_inflammation.py).

**Verification**
1. Activate the existing environment and install with `python -m pip install -r requirements.txt`.
2. Run `python -m unittest discover -s tests`.
3. Run `python analyze_inflammation.py data --csv inflammation-01.csv`; check the printed values and `output/inflammation-01-summary.png`, and confirm the input data remains unchanged.

**Decisions**
- Require an explicit CSV filename because `data/` contains multiple CSVs.
- Generate one combined PNG; print statistics without creating an extra data artifact.
- Leave `.gitignore` unchanged.

These generated plans tend to follow a structure that includes steps, relevant files, verification procedures, decisions made and any other further considerations.

This now provides us with a planning document which we are able to review and amend as we wish. This is very reasonable approach:

  • Importantly, we are now moving from ad-hoc development to intentional development, which forces us to consider ways forward and make decisions and capture these within a defined plan that we validate and refine before moving to implementation.
  • We may extend this plan with other sections and further detail as needed.
  • It has created a concrete document we can discuss and refine with colleagues before we proceed.
  • It also provides a “checkpoint”: if the implementation is unsatisfactory we can remove the implementation, amend the plan, and ask Copilot to create the implementation again.

Take a look at the Problems tab in the terminal pane at the bottom. You may spot some errors regarding files that don’t exist. This is partly expected, since some of the files it mentions in the plan may not exist yet, but if you spot an error about a missing .github/copilot-instructions.md file, this is probably due to it getting the path to it incorrect, since it locates it relative to the location of the plan file.

e.g. you may need to change .github/copilot-instructions.md to ../copilot-instructions.md.

Challenge

Solo Exercise: Review!

10 mins.

As we know, we should always review the output from generative AI. So with a skeptical mindset:

  • Carefully review the generated plan and ensure it complies with the initial prompt.
  • Does the plan match what you want from this tool?
  • Refine the plan as needed.
  • Ensure that a README.md file is created to hold instructions on how to install the prerequisites in a virtual environment and run the tool. If there isn’t, add a new line about it to the relevant files section.
  • If you spot any references to creating unit tests, remove them for now - we’ll cover this later!
  • Add a new line to the Verification section: Do not create any unit tests.

When you’ve finished, add your thoughts about how well the built-in planning agent performed this task into the shared document, noting what it did well and what it could have done better.

Enacting the Plan

Once we’re happy with our plan, we may proceed with implementation either by selecting the ‘Start implementation’ option in the chat if it appears, or by selecting the plan file in the chat context and entering something like the following in the chat with the Agent mode enabled:

Implement the plan

The built-in agent mode will then create an implementation based on the plan. You’ll need to review and approve various actions the agent needs to take in order to follow the plan. If the agent seems to pause for too long, look for anything it wants you to select from the search dropdown at the top of the VSCode window.

Challenge

How did it do?

5 mins.

Run the tool on the command line, following the instructions in the README.md. Does it work as specified? Does it operate as you intended? For example (your command many differ):

BASH 

python csv_statistics_plotter.py data/inflammation-01.csv

Add your thoughts to the shared document.

Using Skills for On-demand Coding Tasks

With the aid of Copilot’s planning agent we’ve created an initial plan, reviewed and refined it, and had Copilot create an initial implementation for us based on the plan. However, there are other tasks typically conducted during development, and Copilot (and other similar generative AI tools and infrastructure such as Claude) allows us to encapsulate these tasks by creating our own custom skills. An AI skill is a packaged set of knowledge, instructions, and behaviours that extends an AI assistant’s ability to perform a defined task. By defining common development tasks as skills, when it comes to implementation, we can call on these skills to lighten the load.

Let’s create some skills using this approach to help us with some development tasks.

First, from within our coding directory, create a new directory .github/skills to hold our skills, e.g.

BASH 

mkdir .github/skills

VSCode also has a useful extension that’s useful for identifying issues with skills that we develop, so let’s install that now so we can use it later:

  1. Select the Extensions icon on the sidebar.
  2. Add @id:ms-vscode.vscode-chat-customizations-evaluations into the search box, which is a specific reference for the extension we want.
  3. Select Install

This extension - which actually acts as a Copilot skill - helps us find contradictions in skill prompt logic as well as other ambiguities and issues. This extension is also useful for identifying issues with other Copilot AI prompt files, such as .instructions.md, .prompt.md and .agent.md files.

A Docstring Writer Skill

Let’s start with a very common task in software development, that of adding descriptions of our functions (and methods and modules) to our code. These are commonly referred to as docstrings.

Let’s define a skill that adds docstring suggestions to our code. Our code may already have docstrings defined, but perhaps we want to ensure that all docstrings follow a common format, such as restructured text, e.g.:

PYTHON 

def concat(arg1, arg2):
    """Returns the result of concatenating ``arg1`` with ``arg2``.

    :param arg1: A first string to concatenate.
    :type arg1: str
    :param arg2: A second string to add to the first string.
    :type arg2: str

    :return: A string containing the concatenation of the two arguments.
    :rtype: str
    """
    # This is the actual function implementation.
    return arg1 + arg2

In VSCode, agents are typically defined in a file with a .agent.md suffix. Create a new add-docstrings directory in .github/skills, and add a new SKILL.md file with the following contents:

MARKDOWN 

---
name: add-docstrings
description: Writes docstrings for Python code in the reST format.
compatibility: Requires python3
metadata:
  version: 1.0.0
---

This skill adds docstrings to Python source code files.

## Constraints

- ONLY make modifications to Python source code files.
- ALWAYS create module, function and method docstrings.
- Docstrings should ALWAYS be in the reStructuredText format.
- Where docstrings already exist, ALWAYS convert them to the reStructuredText format.

## Approach

1. Analyse source code and identify where new docstrings are needed, or existing ones need to be updated.
2. Make the identified changes to the source code.
3. Provide a summary of what was done.

We’re being careful to define constraints on the behaviour here in addition to defining the approach to take as a sequence of simple, explicit steps.

Once you’ve saved the file, select the analyze_inflammation.py script file (or whatever it’s called) in the chat context, then run the skill by ensuring you have Agent mode enabled, and entering /add-docstrings into the VSCode chat. You should see reST docstrings added to your code.

Challenge

Solo Exercise: Review!

5 mins.

We should always review AI-generated content, so take a look at each of the newly added docstrings in turn and only select Keep if you agree with them! If not, correct them instead.

A Unit Test Builder Skill

Sometimes we want our skills to explicitly execute commands and do something based on analysing the output of those commands. Code linters are well established tools for statically analysing code and reporting common code issues. So how should we handle this?

A critical step in writing robust, reproducible code is to test your software, and in addition to testing software manually, the writing and running of unit tests is established practice that can save you time - for those types of test thare are amenable to automation.

Pytest is a popular Python testing framework that makes it straightforward to write and run automated tests. Unit tests help developers verify that individual functions and components behave as expected, reducing the likelihood of defects and making it safer to modify or extend software over time. By writing tests, teams can detect problems early and increase confidence that changes have not introduced old issues known as regressions. Pytest is widely used because it has a simple and readable syntax, requires minimal boilerplate code, provides detailed failure reports and integrates easily with continuous integration (CI) systems to automate testing as part of the development workflow.

A Quick Tour of Pytest

Let’s set up Pytest and create a directory to hold our tests:

BASH 

python -m pip install pytest
mkdir tests

Within the tests directory, we’ll also create an example test in a new test_example.py:

PYTHON 

def test_addition():
    assert 1 + 2 == 3

Ideally, we’d of course create a test that actually tests the code we’ve written, but since your code will likely differ in structure we’ve just used a generic example. We can now run this test (and any others that may be created in the future within this file) by doing:

BASH 

python -m pytest tests/test_example.py

And you should see the following:

================================ test session starts ================================
platform darwin -- Python 3.14.2, pytest-9.0.3, pluggy-1.6.0
rootdir: /Users/steve/Test
configfile: pyproject.toml
collected 1 item

tests/test_example.py .                                                       [100%]

================================= 1 passed in 0.01s =================================

Delete the tests directory along with its contents.

Challenge

Solo Exercise: Create a Unit Test Writer Skill

10 mins.

Based on what we’ve learned from writing our docstring skill, write a skill for automatically drafting Pytest unit tests in .github/skills/add-tests/SKILL.md. Ensure it has the following sections:

  • Constraints: what should it do? What should it not do? Where should tests reside?
  • Approach: list the steps the skill needs to take to generate and run the unit tests
  • Commands to Use: how to run the unit tests?

With skill agent definitions conciseness is good, so try to keep it short.

An initial try:

MARKDOWN 

---
name: add-tests
description: Create Pytest unit tests.
compatibility: Requires python3
metadata:
  version: 1.0.0
---

This skill writes Pytest unit tests.

## Constraints

- ONLY write tests and test files in the `tests` directory.
- ONLY write tests and test files in the `unit_tests` directory.
- Tests should ONLY be written for Pytest.

## Approach

1. Examine the code and identify suitable unit tests.
2. Write the identified unit tests.
3. Run pytest to verify the tests.
3. Provide a summary of what was created and the results of running pytest.

## Commands to Use

- Ensure the virtual environment is activated before running pytest

```bash
python -m pytest tests/
```

One way to help us identify any issues with our agent is to use the Chat Customizations Evaluations extension we installed earlier:

  1. Select Analyze that should have appeared at the bottom of the SKILL.md file editor pane. If you receive an error here, check the syntax of the SKILL.md file.
  2. A pop-up box will request permission to conduct the analysis. Select Allow (or similar).
  3. When the analysis is complete, select Show problems when the pop-up appears, which will display a list of issues in the PROBLEMS tab in the bottom VSCode pane.

You should see at least one issue classed as a contradiction, since we’ve specified two locations for the unit tests in the constraints.

Within the file code editor, you should see these issues highlighted with underlines. If you hover over either the entries in the PROBLEMS tab, or the editor underlines, a description will pop up explaining the issue.

Remove the unit_tests line in the constraints section.

Challenge

Solo Exercise: Review and Refine the Test Writer Skill

5 mins.

Go through each of the identified issues and fix them.

One way to do this such that you get an opportunity to review the suggestions before they are applied, is to first right-click on the entry in the PROBLEMS tab and select Explain which drafts a set of proposed changes.

The Chat Customisations Evaluations tool can be a bit overzealous, so if you think a suggestion is overkill, feel free to ignore it.

If you’re happy with the change, select the Apply to... icon at the top right of the suggestion to apply it to the skill file, then select Keep in the editor window. Otherwise, Undo the change and add your own fix.

Note: you may find that even once the identified problems are fixed, issues in the PROBLEMS tab still remain. This appears to be a bug, and restarting VSCode should fix this.

Now use the skill to create some unit tests, e.g. select Agent mode and enter /add-tests into the chat window.

Challenge

Class Exercise: How did it do?

5 mins.

Take a look at the tests that have been written, and then run the tests on the command line, e.g.:

BASH 

python -m pytest -m tests/

Add your thoughts to the shared document on the following questions:

  • Can you understand them?
  • How useful are they to you?
  • How could they be improved? What’s missing?
  • Did they pass?
Callout

Pros and Cons

We should consider how we use AI tools to verify the code it has written very carefully. AI-generated tests often focus on the most obvious behaviours and may miss important edge cases, error conditions, and domain-specific requirements that an experienced developer would consider. Tests may therefore provide a false sense of confidence while leaving significant defects undetected.

Another concern is that writing tests is itself a valuable design activity. Creating unit tests forces developers to think carefully about how code should behave, as well as its interfaces, assumptions and failure modes. If AI generates all the tests, developers may miss opportunities to deepen their understanding of the code and identify design weaknesses.

AI-generated tests can also suffer from the same issues as AI-generated code: incorrect assertions, unrealistic test data, brittle implementation-dependent checks, or tests that merely reproduce the logic of the code under test rather than independently verifying it. In some cases, the tests may pass while providing little meaningful verification.

As with development, the most effective approach is usually to treat AI as an assistant rather than an author. AI can help generate test scaffolding, suggest test cases, and identify potential edge conditions, but developers should review, refine, and validate the resulting tests to ensure they genuinely improve software quality.

Callout

Should we Even use AI at all for Generating Tests?

We’ve created a generative AI skill for this which seems useful. But using AI tools come at a cost - typically, they take a while to run and are prone to error. What we should also consider is whether a more dedicated tool on its own would be a better fit.

In this case, beyond using AI to set up the test scaffolding, there are established tools that are more specifically designed to create unit tests for software.

For example, Hypothesis is a property-based testing library for Python. It allows you to specify the domain of data values you wish to test for a given test case, and it generates a distribution of test values that select for edge cases within that domain that you test against high-level assertions.

A Documentation Skill

Although writing good software documentation is critical for understanding and using code, it’s often a task many developers find tedious. However, there are patterns to writing documentation that are amenable to AI assistance, albeit with some important caveats.

A typical tool for writing software documentation is MkDocs. MkDocs is a lightweight and open-source static site generator designed specifically for creating software documentation from Markdown files. Developers use MkDocs because it provides a simple and efficient way to write, organise, and publish documentation that is easy to maintain alongside source code in version control repositories. It automatically generates a professional-looking website from documentation files, supports navigation menus, search functionality, themes, and extensions, and integrates well with platforms such as GitHub for automated deployment. The process of authoring and generating documentation is straightforward compared to many other methods, and whilst this helps to reduce the burden on writing documentation somewhat, writing docs is often a mechanistic process.

A Quick Tour of MkDocs

For example, we can create a new MkDocs project in our code by installing the mkdocs Python libraries and invoking it:

BASH 

python -m pip install mkdocs mkdocstrings[python]
python -m mkdocs new .

This will create two files in your project: mkdocs.yml and docs/index.md. The first file mkdocs.yml is the configuration file for your documentation site, which serves as the central configuration hub for your MKDocs documentation. It tells MKDocs how to structure your documentation site, which plugins and themes to use, how to organize navigation, etc. docs/index.md is the main landing page of your documentation site.

Let’s first look at the mkdocs.yml file. It is almost empty, so let’s edit it to contain the following:

YAML 

site_name: Inflammation Data Analysis

nav:
  - Overview: index.md

plugins:
  - search
  - mkdocstrings

Here we give a name to our documentation site, and set up the navigation menu with one item Overview that links to index.md. We also enable two plugins, search to provide search functionality in the documentation site, and mkdocstrings to automatically generate API reference documentation from Python docstrings (i.e. like the ones generated by our add-docstrings skill).

We can try to render the documentation site locally and see what it looks like:

BASH 

python -m mkdocs build
python -m mkdocs serve

This will start to build a local static documentation site and serve it at a local web server. By default, it will be available at http://127.0.0.1:8000/, which will also show in the terminal output. You can open this URL in your web browser to view the documentation site.

Ordinarily, we could iterate on writing and verifying what’s built until a first version is ready, and if we had this repository in our own GitHub account, we could go ahead and deploy this to our repository with python -m mkdocs gh-deploy which would deploy it to a new gh-pages branch and it would be visible from the repository’s GitHub Pages website.

Challenge

Solo Exercise: Create, Review and Refine an MkDocs Skill

15 mins.

Write a skill for automatically generating documentation using MkDocs. Ensure it has the following sections:

  • Constraints
  • Approach: list the steps the skill needs to take to generate the documentation
  • Initial mkdocs.yml Template: the starting content for the mkdocs.yml file
  • Documentation Sections: a list of sections to include in the documentation, including an overview of the software, software prerequisites, and how to run the software
  • Commands to Use

Once finished, use the Chat Customisations Evaluations tool as before to review the skill and suggest improvements, and refine the skill until you are happy with it.

Finally, use the skill to create the documentation, e.g. select Agent mode and enter /create-docs into the chat window.

Here’s one solution:

MARKDOWN 

---
name: create-docs
description: Author and build software documentation using MkDocs.
compatibility: Requires python3
metadata:
  version: 1.0.0
---

This skill uses MkDocs to create and build software documentation using MkDocs

## Constraints

- ONLY make modifications to the `mkdocs.yml` file and files in the `docs/` directory
- If `mkdocstrings` is not installed or the project does not use Python docstrings, remove it from the plugins list and notify the user.

## Approach

1. Inspect the codebase to identify software prerequisites needed to run the software and the automated tests.
2. Create a `mkdocs.yml` file from the template below.
3. Write documentation sections as specified below.
4. Provide a summary of what was done.

## `mkdocs.yml` Template

```yaml
site_name: Replace with the actual name of the software being documented.

nav:
  - Overview: index.md
  - Software Prerequisites: prerequisites.md
  - Running the Software: running.md
  - Running the Tests: tests.md
  - API: api.md

plugins:
  - search
  - mkdocstrings
```

## Documentation Sections

- `docs/index.md`: a brief overview of what the software aims to do, its high-level architecture, and the key technologies or libraries it uses.
- `docs/prerequisites.md`: contains instructions on how to install the software prerequisites needed to run the software.
- `docs/running.md`: instructions on how to run the software.
- `docs/tests.md`: instructions on how to run the automated tests.
- `api.md`: API documentation based on the code and its docstrings.

## Commands to Use

- Ensure the virtual environment is activated before building mkdocs

```bash
python -m mkdocs build
```

Following the Chat Customizations Evaluation tool:

MARKDOWN 

---
name: build-docs
description: Author and build software documentation using MkDocs.
compatibility: Requires python3
metadata:
  version: 1.0.0
---

This skill uses MkDocs to create and build software documentation using MkDocs

## Constraints

- ONLY make modifications to the `mkdocs.yml` file and documentation files.
- If `mkdocs.yml` already exists, preserve any existing `site_name` and any nav entries not covered by the template, and merge the template structure into it rather than overwriting it.
- If `mkdocstrings` is not installed or the project does not use Python docstrings, remove it from the plugins list, remove the `API Reference` entry from `nav`, populate `docs/api.md` with manually authored API documentation extracted from the source code and note that it is not auto-generated, and notify the user.

## Approach

1. Inspect the codebase to identify software prerequisites needed to run the software and the automated tests.
2. Create a `mkdocs.yml` file from the template below.
3. Write documentation sections as specified below.
4. Provide a summary of what was done.

## `mkdocs.yml` Template

```yaml
site_name: Replace with the actual name of the software being documented.

nav:
  - Overview: index.md
  - Prerequisites: prerequisites.md
  - Running: running.md
  - Tests: tests.md
  - API Reference: api.md

plugins:
  - search
  - mkdocstrings
```

## Documentation Sections

- `docs/index.md` - a brief overview of what the software aims to do, its high-level architecture, and the key technologies or libraries it uses.
- `docs/prerequisites.md` - contains instructions on how to install the software prerequisites needed to run the software. If the project contains a requirements.txt, pyproject.toml, or similar dependency file, include virtual environment setup instructions. Otherwise, omit that subsection.
- `docs/running.md` - instructions on how to run the software.
- `docs/tests.md` - instructions on how to run the automated tests. If no automated tests are found in the codebase, state that clearly in the file and keep the page in the nav with a brief placeholder note.
- `docs/api.md` - API documentation based on the code and its docstrings, or manually authored API documentation extracted from the source code if `mkdocstrings` is not available; note when the documentation is not auto-generated.

## Commands to Use

- If a virtual environment is already available, activate it before running the build command.

```bash
python -m mkdocs build
```
Challenge

Class Exercise: How did it do?

5 mins.

Serve the built documentation and view it in a browser, e.g.:

BASH 

python -m mkdocs serve

Direct a browser to http://localhost:8000/

Take a look at the documentation, and add your thoughts to the shared document on the following questions:

  • Are the docs buildable and viewable?
  • Are they correct?
  • How could they be improved?
  • Would they be adequate for another developer unfamiliar with the project to understand (at a basic level) what it’s about, how to set it up, and how to run the code and the tests?

Summary

In this episode, we’ve explored a task-based approach to software development using Copilot’s planning and skill capabilities. We began by using the built-in planning agent to break down a complex feature into manageable steps before implementation, then demonstrated how to create reusable skills for common development tasks such as adding docstrings, fixing code style issues with linting, writing unit tests, and generating documentation.

Throughout, we’ve emphasized that AI tools are most effective as assistants rather than replacements for developer judgment. They excel at creating scaffolding, setting up boilerplate code for new technologies, and generating preliminary content that can be reviewed and refined. However, they should always be reviewed carefully, and for some tasks, dedicated specialized tools may be more efficient and reliable than AI-based approaches. The key is understanding when to use AI as a helpful tool and when to reach for more established alternatives.

Key Points
  • Use the planning agent to create structured plans before implementation.
  • Create custom skills to encapsulate common development tasks.
  • Always review AI-generated code, tests, and documentation carefully.
  • Consider whether dedicated tools might be more appropriate than AI for certain tasks.
  • AI is most valuable for scaffolding, boilerplate generation, and preliminary content. The real value is what you add yourself and how you refine it.