Professional Judgment Should Not Be Delegated to AI
I have previously blogged about ethical issues relating to AI, including Responsible AI and Ethical AI in the Workplace and Guidelines for Ethical AI in Higher Education. In today’s blog I look at the complicated issue of bias and how it can compromise judgment.
As Kelly D. Mullins points out, “One of the most important concepts in responsible AI use is ‘human in the loop.’ AI can generate impressive output quickly, but it can also sound confident when it’s wrong, reflect bias, or miss nuance that only the human mind could catch. This is where professional judgment comes in. CPAs, for example, are trained to apply skepticism, use their experience, and understand context — qualities no algorithm can fully replicate.”
Mullins identifies three areas for evaluation in applying AI to data in an ethical and responsible manner. When reviewing AI output, ask yourself: Does this make sense given everything I know? Are there contradictions? What assumptions is the system making?
It helps to watch for a few human habits that AI can amplify:
- Automation bias. Trusting the machine’s answer even when your own reasoning or other evidence suggests caution.
- Overconfidence bias. Treating AI as infallible.
- Anchoring bias. Latching onto the first piece of information the tool provides and letting it color everything that follows.
What is AI bias?
Mark Jolley points out in his posting, The hidden AI bias in company financials, that “AI bias” is often used as a vague descriptor for any instance where an AI system produces an unfair or undesirable outcome. However, bias is not a single flaw but a series of vulnerabilities that can be introduced at multiple stages of an AI system’s lifecycle, from initial data collection to final deployment. Understanding this anatomy is crucial for moving the conversation from abstract concern to understanding and concrete action.” Jolley states that there are three levels of bias, depending on the source: data, model and human.
Data-level bias, the most pervasive source of bias in AI systems, originates not from malicious intent but from the data on which they are trained. AI models learn to make predictions and decisions by identifying patterns in vast datasets. If this training data is incomplete, unrepresentative, or reflects existing human prejudices, the AI will inevitably learn and reproduce those same biases. This fundamental vulnerability manifests in training data bias, historical bias, and selection bias.
Model-level bias can be introduced through the design of the algorithm. There are three main types:
- Algorithmic bias refers to instances where the rules of the model itself, often unintentionally, create discriminatory outcomes. This can happen through the way different inputs are weighted or how the system is optimized.
- Stereotyping bias is evident when a language translation model consistently associates certain activities or risks with gender, race or other descriptors.
- Exclusion bias happens when entire groups are left out of the data or the decision-making process.
Human-level bias can be introduced through human interaction with the system.
- Confirmation bias occurs when AI systems are designed or used in a way that reinforces a user’s pre-existing beliefs.
- Interaction bias occurs when a system learns prejudice directly from its users.
Amrita Choudhary states that the evolution of AI “has significantly influenced all aspects of human life, such as business, education, law, accounting and many more. AI technology has enabled individuals and organizations to handle huge chunks of data within the shortest time.”
Ethical Implications of AI in Accounting
Accounting firms that use AI are likely to experience bias and fairness concerns. AI has been designed to learn and interpret information based on the task at hand. This is achieved by using dataset algorithms, often based on expert knowledge or the organizations’ past reports and data to execute the desired functions. The information has human involvement, and there are possibilities that the data presented to the system might be subjective or biased. The consequences include distorted outputs, which might mislead the organization and result in undesired results. In response, firms should strive to ensure fairness in training datasets and the decision-making process to help eliminate bias.
Another ethical implication of AI use in accounting is uncertainties in data privacy and security. Accounting entails handling sensitive data that is often highly confidential. When using AI, accounting professionals must provide key datasets that might contain sensitive information. Most databases are prone to hacking and unauthorized access, or users may interfere with confidential financial information.
The Transparently Risk Engine is trained on decades of structured financial statements and disclosures from over 85,000 listed companies. It looks for subtle, data-driven patterns of accounting manipulation, for example accrual anomalies, irregular revenue recognition, and balance sheet distortions. These are signals rooted in the operational and reporting behaviors of firms, not people. Because of this, the nature of bias is different:
- Human data bias often reflects inequities in how individuals are represented in the data. This can lead to unfair treatment or flawed predictions when applied to real-world decisions about people.
- Company data bias, by contrast, emerges from outdated accounting standards, industry-specific quirks, uneven regulatory regimes, or over-representation of certain firm types. These factors shape what the model learns—and if left uncorrected, can lead to distorted risk assessments.
In short, the patterns a model learns from historical company data may not fully reflect today’s reporting realities. Data bias leads to incorrect results, renders data ineffective for its intended purpose, and contributes to systemic inequality.
Ethical Implications of AI in Finance
Data bias in finance refers to systematic errors or prejudices in financial datasets, which can lead to unfair or inaccurate outcomes. It can arise from various sources, such as historical, incomplete data collection, or algorithmic biases. Data bias in finance can have significant implications, including biased lending decisions, discriminatory pricing, or skewed investment strategies. Therefore, recognizing and mitigating data bias is essential to ensure fair and equitable outcomes in financial systems, requiring careful data collection, preprocessing, and algorithmic design.
One example of data bias is if in the financial analysis of a company, the model used might heavily prioritize the strong, clear signals it’s been trained to recognize (e.g., looking for signs of aggressive revenue recognition), yet the model might overlook new trends in revenue recognition such as those in ASC 606 on revenue recognition that changed the ways such amounts can be determined.
In this example, the confirmation bias isn’t about human prejudice, but about the model’s learned tendency to prioritize and interpret new data through the lens of its most strongly reinforced historical patterns, potentially leading it to miss evolving or less conventional forms of accounting manipulation.
Concluding Comments
There is much more to say about data bias because it provides the foundation for calculations and judgments that will be made by AI. The key point is that the basic ethical principles incorporated into professional codes of conduct, such as the AICPA Code of Conduct, must be adhered to in AI matters. This means that decisions are made objectively and with competence and using due care. The integrity of the data must be retained to enhance transparency in decision making.
Another important issue is that organizations must establish a strong set of internal controls, develop a culture of compliance, ad a system of governance to ensure the data inputted by AI is reliable. As the saying goes, “Garbage in, garbage out.”
Blog posted by Steven Mintz, PhD, professor emeritus at Cal Poly San Luis Obispo, on September 22, 2026. You can contact Dr. Mintz at: smintz@calpoly.edu. Visit Steve’s website to find out more about his activities.