Ethics, Reliability and Security Considerations

Last updated on 2026-07-09 | Edit this page

Overview

Questions

  • What are some risks of biased, inaccurate, or unreliable AI-generated outputs?
  • How can the use of AI tools compromise data privacy, security, or confidentiality in research and software development?
  • What intellectual property and authorship issues emerge when AI contributes to code or written work?
  • What are the long-term consequences of researchers relying on AI without developing core coding skills?
  • What best practices can ensure that AI is used responsibly, ethically, and transparently in research workflows?

Objectives

  • Describe common sources of bias, inaccuracy, and unreliability in AI-generated outputs.
  • Explain data privacy, confidentiality, and security risks associated with using AI tools in coding and research contexts.
  • Summarize intellectual property, authorship, and citation considerations related to AI-generated code and text.
  • Analyze the potential long-term consequences of researchers relying on AI tools without developing foundational coding skills.
  • Examine ethical challenges introduced by AI-assisted research, including accountability, transparency, and reproducibility.
  • Assess the appropriateness of AI tool usage in specific research or coding scenarios.
  • Apply best practices to mitigate ethical, security, and skills-related risks when using AI in research.
  • Develop personal or team-level guidelines for responsible and ethical AI use in coding and data analysis workflows.

Overview

Understanding the risks and implications of AI is critical to using AI tools for coding safely, effectively, and with confidence. In this episode, we’ll take a brief look at issues related to:

  • Errors, biases and security issues in AI-generated code
  • Intellectual property of AI-generated code
  • De-skilling and overdependence on AI in research computing
  • Best practices for responsible AI use in research

Errors, Security Issues and Biases in AI-Generated Code

Errors

AI coding assistants can produce both random and systematic errors, threatening the reliability and reproducibility of your work. For instance, a 2023 study evaluating the code quality of AI-assisted generation tools found that ChatGPT generated correct code 65.2% of the time and GitHub Copilot generated correct code only 46.3% of the time. However, given the rapid improvements to generative AI over the past few years, it may not be fair to suggest these figures are representative of outputs generated by the current models used by ChatGPT and GitHub Copilot.

Nonetheless, this study underscores the potential danger of depending solely on AI tools for critical research tasks without carefully reviewing the outputs.

There are a few different reasons for errors occurring in AI-generated code. These include:

  • One reason is that there are likely to be errors in the training data. Many large language models have been trained on vast amounts of publicly available code, some of which contains mistakes. As a result, AI-generated code can inherit these errors without any indication that they exist.

  • Another reason for error is that when the model lacks relevant training data or encounters an unfamiliar task, it may invent code or logic rather than responding with uncertainty. This can produce outputs that are plausible but incorrect, a phenomenon often called a hallucination.

  • AI-generated code may be outdated. An AI model is trained on vast amounts of publicly available code, including code written many years ago, and is only aware of information up to the date that it was pre-trained, which may be several months in the past. Therefore, the AI may not produce code that follows the most up to date standards. For example, the AI might suggest a function from an open-source library that hasn’t been well-maintained over the past few years.

  • Using AI to write your code without having a structured plan can lead to messy and confusing code that difficult to understand and maintain, increasing the risk of error.

Callout

Vibe Coding

Vibe Coding is a term used to describe AI-assisted coding without a structured plan, proper design, or architectural considerations. Decisions are made on the fly, often based on intuition or immediate needs rather than a thoughtful development strategy.

Andrej Karpathy, co-founder of OpenAI and one of Time Magazine’s 100 Most Influential People in AI in 2024, has said about vibe coding: “There’s a new kind of coding, I call ‘vibe coding’, where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It’s possible because the LLMs … are getting too good.

“When I get error messages I just copy [and] paste them in with no comment, usually that fixes it … I’m building a project or web app, but it’s not really coding – I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.”

This can be fantastic for developing a quite prototype or trying out an idea. However, coding in this way can also lead to some major problems:

  • Without planning the structure of your code at the start, programs are likely to become messy and confusing, and this can introduce mistakes into the code.
  • Outputs are likely to appear mostly correct and, while obvious errors are usually caught, the subtle mistakes are easy to miss.
  • This approach is likely to lead to problems being discovered only during the build or runtime phase instead of during design, which makes them more time-consuming and costly to fix.

Security Issues

Data Privacy and Confidentiality

It’s really important to be cautious that you don’t accidentally share confidential code, sensitive datasets or proprietary research methods with an AI tool. Depending on the settings of your AI tool, the information you enter may be reused to improve the AI model and/or could resurface in future outputs, creating risks around intellectual property leakage, confidentiality breaches, or non-compliance with data protection regulations.

Security Risks in AI-Generated Code

A 2023 Stanford University study found that programmers who used AI assistants often produced less secure code but at the same time, felt more confident that it was secure - a risky combination!

Some of the errors in AI-generated code can pose security risks for your software.

For example, AI tools sometimes hallucinate non-existent coding packages in their outputs. A study by the security company Vulcan identified a cyberattack technique where criminals would publish a malicious package under the name of the hallucinated package, hoping that AI would suggest the same package to other users, who would then install the cyber criminals’ malicious package based on the AI tool’s recommendation.

This practice has become known as ‘slopsquatting’, a combination of ‘AI Slop’ and ‘typosquatting’ (the practice of registering domain names or software package names that are slightly misspelled versions of popular ones to trick users into visiting them or downloading malicious content).

Callout

Embed a ‘security conscience’ into the AI

A Security-Focused Guide for AI Code Assistant Instructions was written by the OpenSSF Best Practices and the AI/ML Working Groups. The guide suggests ways that you can improve the security of AI-generated code by deliberately embedding security expectations into the prompts. These might include:

  • Secure coding best practices that are relevant for your code (e.g. Input validation and output encoding, error handling and logging, secure defaults and configurations, testing for security)
  • Reminders of software supply chain security (i.e. security of suggested third-party libraries and dependencies)
  • Address relevant platform and runtime security considerations (e.g. operating system, deployment considerations, mobile app security)
  • Language-specific security considerations
  • Pointing the AI toward relevant security standards and frameworks

Note: Including security expectations in prompts requires knowledge of relevant software security practices, so is outside the scope of this novice course. However, it’s worth bearing in mind if you’re interested in developing research software.

Bias

AI-generated code can contain undocumented assumptions that reflect biases in the model’s training data. These assumptions may lead to code or documentation that unintentionally favours certain demographic groups over others.

Challenge

Biases in User Validation

Scenario

Sally wants to make a web-based tool that allows other researchers to create an account and log in to explore the animals data. She asked an AI tool to write a function that validates a name on a user profile, and the following code is generated.

What are some biases or assumptions in this code? Write your thoughts in the shared document.

Hint: If you do not understand the code, use the docstring to understand what the function does.

PYTHON 

import re

def is_valid_name(name: str) -> bool:
    """
    Returns True if the name is valid.
    A valid name contains a first name and a surname.
    It contains only alphabetic characters and starts with a capital letter.
    """
    pattern = r"^[A-Z][a-z]+(?: [A-Z][a-z]+)*$"
    return bool(re.match(pattern, name))
  • The code that only allows Latin letters and Western capitalization patterns, implicitly assuming names are formatted as “First Last”.
  • These undocumented assumptions exclude valid names from many cultures (e.g., letters with accents, apostrophes, non-Latin scripts, or single-word names), reflecting biases in the model’s training data.

Intellectual Property of AI-Generated Code

Intellectual property rights for AI-generated code are currently evolving.

Currently in the UK, if a person creates some work using AI, the content is the human’s own intellectual creation and the copyright belongs to the human creator or person “by whom the arrangements necessary for the creation of the work are undertaken”.

However, there’s ongoing debate about how this practically applies to many forms of AI outputs, including software code, because the statutory language was drafted long before modern AI and doesn’t map cleanly to current AI models.

It’s also worth considering that ownership can depend on contractual terms, such as employment contracts or AI tool terms of service, which may assign rights to an employer or platform rather than the individual user.

AI-Generated Code May Contain Copyrighted Material

AI models are trained on a vast amount of data that may include copyrighted material. Therefore, there’s a risk that AI-generated code may closely resemble the copyrighted code from its training data.

If you add AI-generated code to an open-source project, you may unintentionally introduce a licensing conflict if the AI-generated patterns or structures of the code originate from software under incompatible licences. This could lead to the open-source project facing copyright infringement claims.

Callout

No AI-generated Code Policy for Open-Source Project Cloud Hypervisor

Cloud Hypervisor is an open-source software project that helps large computing systems run multiple programs safely and efficiently at the same time, which is a common requirement in cloud services (services provided over the internet rather than from a local computer). In 2025, the project’s maintainers implemented a no AI-generated code policy for contributions, out of concern that such code might unintentionally include material derived from other software with incompatible licences, creating legal risks for the project and its users.

In a post on GitHub, Cloud Hypervisor’s maintainers said: ‘Our policy is to decline any contributions known to contain contents generated or derived from using Large Language Models (LLMs). This includes ChatGPT, Gemini, Claude, Copilot and similar tools.’

De-Skilling and Overdependence on AI in Research Computing

AI tools can significantly enhance productivity in research computing, but excessive reliance on them introduces risks to research quality, integrity, and long-term capability.

Risks of De-Skilling

Over-reliance on AI for coding can prevent researchers from developing essential skills in research software development and data analysis. Without a solid understanding of the code you use, you can’t reliably verify whether your research results are correct, reducing confidence in the validity of any results you publish.

There are also long-term implications for the research community. If researchers become dependent on AI tools for software development tasks, institutions risk losing the collective ability to design, build, and maintain research software independently. This creates problem if tools become unavailable, restricted, or unsuitable for specific research needs.

Therefore, rather than skipping learning to code because AI can handle it, this is precisely the time to strengthen your research computing skills.

Preserving Critical Thinking in the Age of AI

A common bias among AI users is the tendency to over-value AI-generated outputs. Outputs from GPT systems often have an authoritative tone, which can make us inclined to accept the output without critically evaluating it.

However, maintaining human judgement is especially important in research, where novelty, insight, and deep understanding often matter more than speed.

Therefore, it’s important that we avoid uncritical trust in AI and instead treat AI outputs as suggestions rather than solutions. Also, remember that you as the researcher need to take responsibility for any AI-generated code you use.

Challenge

Personal Ethics and Security Policy

Bearing in mind the ethical, reliability and security considerations from this episode, write yourself some guidelines for how you will use AI tools responsibly to assist with coding.

e.g. Sally’s personal ethics and security policy for using AI tools for research coding is:

  • Make sure I understand any code generated by AI before using it for my research.
  • Never input sensitive, personal, or proprietary data into AI systems.
  • Maintain my critical thinking and decision making skills, never allow AI to do these things for me.
Key Points
  • AI-generated code is not fully reliable: it may contain subtle errors, outdated functions, or fabricated solutions (hallucinations) that compromise research validity and reproducibility.
  • Vibe coding (AI-assisted coding without planning) can produce messy, error-prone programs. Structured development and verification remain essential.
  • Using AI tools can create data privacy, confidentiality, and security risks, especially when submitting sensitive datasets or proprietary code to cloud-based AI services.
  • AI may suggest insecure or outdated coding practices. To mitigate the risk you could embed security expectations in prompts and review outputs critically.
  • Be aware of the evolving issues surrounding intellectual property of AI-generated code.
  • Over-reliance on AI can lead to de-skilling, reducing researchers’ coding proficiency, critical thinking, and long-term ability to maintain software.
  • Ethical AI use requires human oversight, responsible data practices, defined boundaries, transparency, and validation of AI-generated outputs.
  • It could be helpful for researchers to develop personal or team-level AI ethics and security policies.

References