Ethics, Reliability and Security Considerations

Last updated on 2026-02-06 | Edit this page

Overview

Questions

  • What are some risks of biased, inaccurate, or unreliable AI-generated outputs?
  • How can the use of AI tools compromise data privacy, security, or confidentiality in research and software development?
  • What intellectual property and authorship issues emerge when AI contributes to code or written work?
  • What are the long-term consequences of researchers relying on AI without developing core coding skills?
  • What best practices can ensure that AI is used responsibly, ethically, and transparently in research workflows?

Objectives

  • Describe common sources of bias, inaccuracy, and unreliability in AI-generated outputs.
  • Explain data privacy, confidentiality, and security risks associated with using AI tools in coding and research contexts.
  • Summarize intellectual property, authorship, and citation considerations related to AI-generated code and text.
  • Analyze the potential long-term consequences of researchers relying on AI tools without developing foundational coding skills.
  • Examine ethical challenges introduced by AI-assisted research, including accountability, transparency, and reproducibility.
  • Assess the appropriateness of AI tool usage in specific research or coding scenarios.
  • Apply best practices to mitigate ethical, security, and skills-related risks when using AI in research.
  • Develop personal or team-level guidelines for responsible and ethical AI use in coding and data analysis workflows.

Overview

Understanding the risks and implications of AI is critical to using AI tools for coding safely, effectively, and with confidence. In this episode, we’ll take a brief look at issues related to:

  • Errors, biases and security issues in AI-generated code
  • Intellectual property, authorship, and citation of AI-generated code
  • De-skilling and overdependence on AI in research computing
  • Best practices for responsible AI use in research

Errors, Biases and Security Issues in AI-Generated Code

Errors

For researchers, relying on AI-generated code carries significant risks. Incorrect code can lead to flawed results, which may compromise the validity of your research, damage your professional reputation, and even necessitate a paper retraction.

AI coding assistants can produce both random and systematic errors, threatening the reliability and reproducibility of your work. For instance, a study evaluating the code quality of AI-assisted generation tools found that ChatGPT generated correct code 65.2% of the time and GitHub Copilot generated correct code only 46.3% of the time. However, it’s important to note that this study was published in 2023, and given the rapid improvements to generative AI over the past few years, it may not be fair to suggest these figures are representative of outputs generated by the current models used by ChatGPT and GitHub Copilot.

Nonetheless, this study underscores the potential danger of depending solely on AI tools for critical research tasks without carefully reviewing the outputs.

There are a few different reasons for errors occurring in AI-generated code. These include:

  • One reason is that there are likely to be errors in the training data. Many large language models have been trained on vast amounts of publicly available code, some of which contains mistakes. As a result, AI-generated code can inherit these errors without any indication that they exist.

  • Another reason for error is that when the model lacks relevant training data or encounters an unfamiliar task, it may invent code or logic rather than responding with uncertainty. This can produce outputs that are plausible but incorrect, a phenomenon often called a hallucination.

  • AI-generated code may be outdated. An AI model is trained on vast amounts of publicly available code, including code written many years ago. Furthermore, the AI model can only draw on information up to the date that it was pre-trained, which may be a year or more in the past. This means that AI may not produce code that follows the most up to date standards. Sometimes this can be a problem. For example, the AI might suggest a function from an open-source library that hasn’t been well-maintained over the past few years. This could mean missing out on a better and more secure library that has been developed recently. Searching the documentation for the package online would make you aware of any potential problems.

  • Using AI to write your code without having a structured plan can lead to messy and confusing code that difficult to understand and maintain, increasing the risk of error.

Callout

Vibe Coding

Vibe Coding is a term used to describe AI-assisted coding without a structured plan, proper design, or architectural considerations. Decisions are made on the fly, often based on intuition or immediate needs rather than a thoughtful development strategy.

Andrej Karpathy, co-founder of OpenAI and one of Time Magazine’s 100 Most Influential People in AI in 2024, has said about vibe coding: “There’s a new kind of coding, I call ‘vibe coding’, where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It’s possible because the LLMs … are getting too good.

“When I get error messages I just copy [and] paste them in with no comment, usually that fixes it … I’m building a project or web app, but it’s not really coding – I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.”

This can be fantastic for developing a quite prototype or trying out an idea. However, coding in this way can also lead to some major problems:

  • Without planning the structure of your code at the start, programs are likely to become messy and confusing, and this can introduce mistakes into the code.
  • Outputs are likely to appear mostly correct and, while obvious errors are usually caught, the subtle mistakes are easy to miss.
  • This approach is likely to lead to problems being discovered only during the build or runtime phase instead of during design, which makes them more time-consuming and costly to fix.

Security Issues

Some of the errors in AI-generated code can pose security risks for your software.

For example, ChatGPT sometimes hallucinates non-existent coding libraries in its outputs. A study by the security company Vulcan identified a cyberattack technique where criminals could hijack these fake libraries by publishing a malicious package under the name of the non-existent library and hoping developers would install the infected library based on the AI tool’s recommendation.

This practice has become known as ‘slopsquatting’, a combination of ‘AI Slop’ and ‘typosquatting’ (the practice of registering domain names or software package names that are slightly misspelled versions of popular ones to trick users into visiting them or downloading malicious content).

A 2023 Stanford University study found that programmers who used AI assistants often produced less secure code but at the same time, felt more confident that it was secure - a risky combination!

Callout

Embed a ‘security conscience’ into the AI

A Security-Focused Guide for AI Code Assistant Instructions was written by the OpenSSF Best Practices and the AI/ML Working Groups. The guide suggests ways that you can improve the security of AI-generated code by deliberately embedding security expectations into the prompts. These might include:

  • Secure coding best practices that are relevant for your code (e.g. Input validation and output encoding, error handling and logging, secure defaults and configurations, testing for security)
  • Reminders of software supply chain security (i.e. security of suggested third-party libraries and dependencies)
  • Address relevant platform and runtime security considerations (e.g. operating system, deployment considerations, mobile app security)
  • Language-specific security considerations
  • Pointing the AI toward relevant security standards and frameworks

Note: Including security expectations in prompts requires knowledge of relevant software security practices, so is outside the scope of this novice course. However, it’s worth bearing in mind if you’re interested in developing research software.

Callout

Data Privacy and Confidentiality

It’s really important to be cautious that you don’t accidentally share confidential code, sensitive datasets or proprietary research methods with an AI tool. Depending on the settings of your AI tool, the information you enter may be reused to improve the AI model and/or could resurface in future outputs, creating risks around intellectual property leakage, confidentiality breaches, or non-compliance with data protection regulations.

Transparency, Explainability, and Bias

Many AI tools offer code suggestions without explaining the reasoning behind them, making it difficult to verify the proposed solutions. Not being able to fully verify code increases the risk of undetected errors influencing your experimental results.

This lack of transparency also has implications for research reproducibility. If the logic behind AI-generated code is unclear, other researchers may be unable to replicate your methods or results, even if the code appears to run correctly. There may be subtle errors or undocumented assumptions embedded in AI-generated solutions, which can lead to inconsistencies across experiments or datasets.

AI-generated code can contain undocumented assumptions that reflect biases in the model’s training data. These assumptions may lead to code or documentation that unintentionally favours certain demographic groups over others.

For example, if an AI tool is asked to generate code to validate a name on a user profile, it may produce code that only allows Latin letters and Western capitalization patterns, implicitly assuming names are formatted as “First Last”. These undocumented assumptions exclude valid names from many cultures (e.g., letters with accents, apostrophes, non-Latin scripts, or single-word names), reflecting biases in the model’s training data.

PYTHON 

import re

def is_valid_name(name: str) -> bool:
    """
    Returns True if the name is valid.
    A valid name contains only alphabetic characters and starts with a capital letter.
    """
    pattern = r"^[A-Z][a-z]+(?: [A-Z][a-z]+)*$"
    return bool(re.match(pattern, name))

Intellectual Property, Authorship, and Citation of AI-Generated Code

Intellectual Property and Ownership

Intellectual property rights for AI-generated code are currently evolving.

Currently in the UK, if a person creates some work using AI, the content is the human’s own intellectual creation and the copyright belongs to the human creator or person “by whom the arrangements necessary for the creation of the work are undertaken”.

However, there’s ongoing debate about how this practically applies to many forms of AI outputs, including software code, because the statutory language was drafted long before modern AI and doesn’t map cleanly to current AI models.

It’s also worth considering that ownership can depend on contractual terms, such as employment contracts or AI tool terms of service, which may assign rights to an employer or platform rather than the individual user.

AI-Generated Code in Open-Source Projects

AI models are trained on a vast amount of data that may include copyrighted material. Therefore, there’s a risk that AI-generated code may closely resemble the copyrighted code from its training data. If you add AI-generated code to an open-source project, you may unintentionally introduce a licensing conflict if the AI-generated patterns or structures of the code originate from software under incompatible licences. This could lead to the open-source project facing copyright infringement claims.

Callout

No AI-generated Code Policy for Open-Source Project Cloud Hypervisor

Cloud Hypervisor is an open-source software project that helps large computing systems run multiple programs safely and efficiently at the same time, which is a common requirement in cloud services (services provided over the internet rather than from a local computer). It is maintained by a community of organisations and developers and is made freely available under an open licence. In 2025, the project’s maintainers implemented a no AI-generated code policy for contributions, out of concern that such code might unintentionally include material derived from other software with incompatible licences, creating legal risks for the project and its users.

In a post on GitHub, Cloud Hypervisor’s maintainers said: ‘Our policy is to decline any contributions known to contain contents generated or derived from using Large Language Models (LLMs). This includes ChatGPT, Gemini, Claude, Copilot and similar tools.’

Authorship and Academic Credit

AI tools can influence research outputs, so to what extent should their contribution be acknowledged? AI systems can’t be authors but not disclosing their use can misrepresent the nature of the researchers’ work and raise academic integrity questions.

Most publishers agree that AI tools do not qualify for authorship and that human authors are fully accountable for the content they produce. Publishers are also broadly in agreement that AI use should be disclosed.

Callout

Publisher’s Stance on AI Use in Academic Research

Summarised from - Rana, N. K. (2025). Generative AI and academic research: A review of the policies from selected HEIs.

  • Cambridge University Press: AI tools cannot be credited as authors, and authors remain fully responsible for the accuracy, integrity, and originality of their work.

  • Nature Portfolio: AI tools are not permitted as authors, their use must be transparently disclosed, and AI-generated images are generally prohibited due to unresolved legal and ethical concerns. Nature requires disclosure of LLM use in the Methods section (or an equivalent section), rather than in acknowledgements or citations.

  • Elsevier: Elsevier allows AI-assisted tools for writing support, limited to improving clarity and readability. However, core scholarly activities, such as generating scientific insights, drawing conclusions, or making recommendations, must remain human-led. Elsevier requires authors to declare any AI tool usage and does not allow AI tools to be listed as authors.

Citation

The outputs of AI aren’t stable, they’re likely to vary depending on prompt wording and the AI model version among other factors. Therefore, they can’t be reliably cited in the same way as you would cite a research paper or software package.

Several universities and libraries recommend citing the tool used including the version and date, and describing how it was used. Many AI tools now allow chats to be shared through URLs, meaning that specific chats can be cited if that would be appropriate and helpful to readers of the research.

Duke University Libraries gives the following guidance on how to cite an AI Chat and AI Tool in several different referencing styles. For example, the APA style would be:

AI Chat

AI Company Name. (Year, Month Day). Title of chat [Description, such as Generative AI chat]. Tool Name/Model. URL of the chat.

Example: OpenAI. (2025, August 21). High school grammar concepts [Generative AI chat]. ChatGPT. https://chatgpt.com/share/68a77b60-0ee4-800c-9acc-cd3fd573c311

AI Tool

AI Company Name. (Year). Tool Name/Model [Description: e.g., Large language model]. URL of the tool

Example: OpenAI. (2025). ChatGPT [Large language model]. https://chatgpt.com/

De-Skilling and Overdependence on AI in Research Computing

AI tools can significantly enhance productivity in research computing, but excessive reliance on them introduces risks to research quality, integrity, and long-term capability.

Risks of De-Skilling

Over-reliance on AI for coding can prevent researchers from developing essential skills in research software development and data analysis. Without a solid understanding of the code you use, you can’t reliably verify whether your research results are correct, reducing confidence in the validity of any results you publish.

There are also long-term implications for the research community. If researchers become dependent on AI tools for software development tasks, institutions risk losing the collective ability to design, build, and maintain research software independently. This creates problem if tools become unavailable, restricted, or unsuitable for specific research needs.

Therefore, rather than skipping learning to code because AI can handle it, this is precisely the time to strengthen your research computing skills.

Preserving Critical Thinking in the Age of AI

A common bias among AI users is the tendency to over-value AI-generated outputs. Outputs from GPT systems often have an authoritative tone, which can make us inclined to accept the output without critically evaluating it.

However, maintaining human judgement is especially important in research, where novelty, insight, and deep understanding often matter more than speed.

Therefore, it’s important that we avoid uncritical trust in AI and instead treat AI outputs as suggestions rather than solutions. Also, remember that you as the researcher need to take responsibility for any AI-generated code you use.

Best Practices for Responsible AI Use in Research

Responsible use of AI for coding assistance requires a combination of ethical awareness, technical safeguards, and disciplined research practice. Here are some examples of ethical best practices, practices to support research reproducibility and scientific validity, and some security measures that you may decide to put in place when using AI to assist with research coding.

Ethical Best Practices

  • Maintain human oversight: Researchers must critically evaluate all AI-generated outputs, remaining alert to potential bias, errors, or inappropriate assumptions.
  • Test and validate rigorously: AI-generated code should be treated as untrusted by default. Apply thorough testing and validation to ensure correctness, reliability, and fitness for purpose.
  • Protect sensitive data: Avoid submitting proprietary code, confidential data, or sensitive research materials to online AI tools. If you use these materials for your research and have decided to use AI, you may want to investigate locally hosted or offline AI assistants to reduce data exposure risks.
  • Define clear usage guidelines: Establish and follow explicit policies for AI use in research computing. These may draw on recognised frameworks such as the ACM Code of Ethics or the European Commission’s Ethical Guidelines on AI

Practices Supporting Reproducibility and Scientific Validity

To maintain transparency, reproducibility, and scientific validity when using AI tools, researchers should:

  • Document AI involvement: Record when, how, and why AI-generated suggestions were used or modified.
  • Validate against known results: Test AI-generated code using benchmarks, reference datasets, or established methods before integration.
  • Combine AI with domain expertise: Use AI to support human judgement and subject-matter knowledge, rather than replacing them.

Security Measures

  • Code review: Review all AI-generated code, ideally using standard code review processes, to identify vulnerabilities, logic errors, or unsafe practices.
  • Secure development practices: If you’re working on a larger piece of research software, integrate security testing tools into development workflows and ensure researchers are trained in secure coding principles.
  • Protect access and data: Use access controls and encryption to safeguard codebases and datasets from unauthorised access by AI tools.
Challenge

Personal Ethics and Security Policy

Write a short personal policy outlining how you will use AI tools responsibly to assist with coding. Include at least three clear guidelines.

Your guidelines could include:

  • Always document when and how AI tools are used.
  • Make sure I understand any code generated by AI before using it for my research.
  • Never input sensitive, personal, or proprietary data into AI systems.
  • Take full responsibility for my research code, even when it is AI generated.
  • Maintain my critical thinking and decision making skills, never allow AI to do these things for me.
Key Points
  • AI-generated code is not fully reliable: it may contain subtle errors, outdated functions, or fabricated solutions (hallucinations) that compromise research validity and reproducibility.
  • Vibe coding (AI-assisted coding without planning) can produce messy, error-prone programs. Structured development and verification remain essential.
  • Using AI tools can create data privacy, confidentiality, and security risks, especially when submitting sensitive datasets or proprietary code to cloud-based AI services.
  • AI may suggest insecure or outdated coding practices. To mitigate the risk you could embed security expectations in prompts and review outputs critically.
  • Be aware of the evolving issues surrounding intellectual property, authorship, and citation of AI-generated code.
  • Over-reliance on AI can lead to de-skilling, reducing researchers’ coding proficiency, critical thinking, and long-term ability to maintain software.
  • Ethical AI use requires human oversight, responsible data practices, defined boundaries, transparency, and validation of AI-generated outputs.
  • It could be helpful for researchers to develop personal or team-level AI ethics and security policies.

References