Responsible AI: Part 1
Trustworthy, Fair and Transparent Intelligence
Overview
Need for AI regulation
Government regulation: While the EU appears to be adopting a careful approach with the Artificial Intelligence Act, the United Kingdom has revealed its intentions to regulate AI. The National Institute for Standards and Technology put out an AI Risk Management Framework, and the White House Office of Science and Technology Policy has published a blueprint for an AI Bill of Rights. White House senior Administration officials met with CEOs of four American companies (Google, Microsoft, OpenAI, and Anthropic) to share concerns about the risks associated with AI.
Industry self-regulation: Financial Industry Regulatory Authority (FINRA), healthcare/The General Medical Council in the UK sets standards for medical education and practice and enforces ethical conduct among doctors. Creative side/the advertising industry also has several self-regulatory bodies, such as the Advertising Standards Authority, which regulates the content of advertisements to ensure they are legal. Technology/standards for privacy, data protection, and cybersecurity. For example, the International Association of Privacy Professionals (IAPP) sets privacy standards and offers certifications for privacy professionals.
During RLHF or instruction tuning, LMs like ChatGPT and FLAN use training data from outside users, crowdworkers, and the web, which could suffer from poisoning attacks, e.g. trigger phase can cause error in test.
clean-label attacks: select samples with a positive label frequently predicted as negative by the model due to trigger phrases.
dirty-label attacks: choose extremely negative samples and giving them a positive label to deceive the model.
Mitigation
- Identify and remove poisoned samples from the training set
- Prematurely stop training or use a lower learning rate
In era of LLMs and Generative AI, the following risks are more serious: Jailbreaking, Poisoned Data, Deepfakes, Data Privacy, Bias
Some familiar news that lead to broader realization of social impact and consideration of AI (like all technologies must be controlled in a way that benefits humankind): e.g. 2018/Facebook–Cambridge Analytica data scandal, 2019/racist and sexist content in famous imagenet led to removal of more than half a million images from the database, Google and Amazon’s business models represent a new form of “surveillance capitalism.”, 2020/Amazon halted police use of its facial recognition technology and IBM getting out of the facial recognition business, “Machine Learning Emissions impact”, 2021/U.S. Senate Hearing that “Facebook’s algorithm amplified misinformation”, 2022/Amazon’s myths about its unsafe warehouses and management by algorithm, 2023/“AI pause” open letter.
Heavily regulated industries adopt Responsible AI, including banking/finance, insurance, healthcare, telecom and heavy consumer-facing (retail, hospitality/travel)
Regulators are looking at Large language model and change AI-in-the wild status, e.g. FTC is asking OpenAI to have independent review for its GPT service.
- Explainability and Transparency: If AI systems are opaque and not able to explain themselves as to why or how specific results are generated, this lack of transparency and explainability will threaten Trust in the system.
- Personal and Public Safety: Use of autonomous systems such as self-driving cars on roads and robots could be a risk of harm to humans. How can we assure human safety?
- Automation and Human Control: If AI systems can generate Trust and support humans in tasks and offloads their work. There will be a risk of threatening our knowledge related to those skills. This will make it more complex to check the reliability, correctness and result of these systems as well as makes human intervention impossible. How do we ensure human control on AI systems?
- Bias and Discrimination: Even if AI-based systems work neutrally, it will give insights on whatever data it is trained. Therefore, it can be affected by human and cognitive bias, incomplete training data sets. How can we make sure that the use of AI systems does not discriminate in unintended ways?
- Accountability and Regulation: With the increase of AI-driven systems in almost every industry, expectations around responsibility and liability will also increase. Who will be responsible for the use and misuse of AI systems?
- Security and Privacy: AI systems have to access vast amounts of data to identify patterns and predict results that are beyond human capabilities. Here, there is a risk that the privacy of people could be breached. How do we ensure that the data we are using to train AI models are secure?
Security controls for LLM-powered applications:
Prompt Engineering Only using Vector Embedding Database: normal: Access Controls, Encryption, Audit Logs and Monitoring, Data Privacy and Anonymization (sensitive data stored in the vector embedding database); prompt: guard against sensitive information around prompt, context, examples, prevent malicious user inputs, prompt injection attacks, input validation and sanitization; response: prevent the generation of inappropriate or harmful responses, biased or discriminatory outputs.
Prompt Engineering + Fine Tuning: Data Privacy and Compliance with training data, compliance with data privacy regulations, such as GDPR or CCPA, Secure Data Storage; Model Licensing and Legal Considerations; Model Security Testing and Privacy Assessment
LoRA: A Low-Rank Adapter for LLM Fine-Tuning: Model inversion: LoRA adapters can be reverse-engineered to gain insights into the inner workings of an LLM. This could be used to develop new attacks against LLMs, such as poisoning attacks or adversarial examples. Data leakage: LoRA adapters could be used to leak sensitive information from an LLM, such as the training data or the model parameters.
Response: prevent adversarial attack to gain unauthorized access to proprietary information or intellectual property, anomaly detection; LLMs can inherit biases present in the training data, which may result in biased or discriminatory outputs. Countermeasures involve regular bias analysis, diverse training data, and fairness-aware fine-tuning techniques to reduce bias and ensure equitable results. LLMs may exhibit vulnerabilities to adversarial inputs, noise injection, or data distribution shifts. Robustness testing, adversarial training, and continual monitoring
PoV
Microsoft
The following guiding principles can help you establish a framework and strategy for responsible AI by design throughout AI lifecycle:
- Human oversight and governance (oversight board and governance)
- Fairness, inclusiveness, and prevention of harm
- Transparency and explainability
- Reliability, safety, security, and respect for privacy
Identify: Fairlearn assessment, Error Analysis
Diagnose: InterpretML, Exploratory data analysis/understand dataset characteristics, What-If/Counterfactual/Debugging, Causal analysis
Mitigate: Fairlearn/Unfairness mitigation algorithms
Typical adversarial LLM attacks: data poisoning (prevention: data validation, sanitization and preprocessing, data access control), goal hijacking/prompt injection/jailbreak (prevention: input validation (compliance check, intent analysis, content filtering), response handling (content filtering, generated content inspection), model hardening, robust system prompt), model DoS (prevention: rate limiting, anomaly detection, load balancing, caching, robustness testing)
Traditional performance metrics: accuracy, recall, precision, root mean squared error (RSME), mean absolute error (MAE), or R2 scores
Responsible AI dashboard:
Dataset Cohorts, different cohorts created in a table. Probability distribution: model’s probability prediction of patients not readmitted back to the hospital within 30 days
Feature cohort: comparing model performance across user-specified sensitive and non-sensitive features (for example, performance across patient age, diagnoses or insulin results)
Google principles: Responsible development and deployment
Toolkit
Analyze
Define problem: AI guidebook, AI Explorables; Construct and prepare data: TF data validation, Data cards playbook
Adjust
Build and train model: TF Model remediation, TF Privacy, TF Federated, TF Lattice, TF Constrained optimization; Evaluate: Fairness Indicators, Language Interpretability tool, What if tool
Integrate
Integrate and Monitor: Model Card toolkit
List of hard questions to answer in ML lifecycle to building AI to be accountable, 28:38
Toolset: 28:58
Use cases of how Google applied Responsible AI process to cloud facial recognition service, e.g. bias on gender, skin tone in public media, 30:40
Google’s Secure AI Framework (SAIF)
AWS
Responsible AI pillars
AWS Ratio framework
Fairness: Considering impacts on different groups of stakeholders
Explainability: Understanding and evaluating system outputs
Privacy and security: Appropriately obtaining, using, and protecting data and models
Safety: Preventing harmful system output and misuse
Controllability: Having mechanisms to monitor and steer AI system behavior
Veracity and robustness: Achieving correct system outputs, even with unexpected or adversarial inputs
Governance: Incorporating best practices into the AI supply chain, including providers and deployers
Transparency: Enabling stakeholders to make informed choices about their engagement with an AI system
- Describe use case — Define the specific problem being solved, validate the need for AI, and identify stakeholders.
- Assess benefits and risks — Identify the potential benefits and risks of the use case across stakeholder groups.
- Define release criteria — Set clear, testable criteria for AI system readiness.
- Design datasets — Create high-quality datasets for training, evaluation, and operations.
- Design the AI system — Implement responsible behavior directly into system design.
- Make an evidence-based release decision — Assess actual benefits and residual risks to make informed release decisions based on evidence.
- Provide downstream guidance and transparency — Support users and other downstream stakeholders with clear explanations of intended usage and limitations.
- Manage post-release monitoring and decommissioning — Monitor system performance and respond to issues.
Hallucination on patient medical diagnosis can cause lives. Even simple decision-making/rule-based system needs to communicate rationale/tradeoff for clarity.
Cisco responsible AI: webex AI-based background blur, noise reduction. Assessment: intended use, source of training data, known biases or edge cases, licensing or legal contracts
https://aws.amazon.com/machine-learning/responsible-machine-learning/
AWS AI Service Cards: provide transparency and document the intended use cases and fairness considerations for our AWS AI services.
Mentioned AWS Machine learning university Responsible AI, pillars: process perspective/value alignment, inclusion, training & education, accountability, human-centered; technology perspective/transparency and explainability, fairness & bias, privacy & security; Tools: Sagemaker/Clarify (bias/explain/detect drift), CodeWhisper (coding companion), CodeGuru (Detect security vulnerabilities and automate code reviews).
https://www.amazon.science/blog/responsible-ai-in-the-generative-era
special challenges of responsible generative AI: Toxicity, Hallucinations, Plagiarism and cheating
Other PoV
AIGC content
Privacy leakage: Stable Diffusion and Google’s Imagen can leak photos of real people and copyrighted images
Bias, toxicity, misinformation: unintentionally reinforce harmful stereotypes, exclude or marginalize certain groups, and contain toxic data sources, which can incite hate or violence and offend individuals
IP Protection: risk of copyright infringement with the generated content if it copies existing works
- Accuracy
- Explainability and interpretability
- Privacy
- Reliability
- Robustness
- Safety
- Security (Resilience)
- Mitigation of Bias
Reliability may be similar to Robustness. Insecure AI may lead to Unsafety to human.
https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
Risk measurement, tolerance, prioritization
Characteristics of trustworthy AI systems
Valid and Reliable: ability of an item to perform as required,
without failure, for a given time interval, under given conditions
Safe: not under defined conditions, lead to a state in which human life,
health, property, or the environment is endangered
Secure and Resilient: can withstand unexpected adverse events or unexpected changes in their environment or use
Accountable and Transparent: reflects the extent to which information about an AI system and its outputs is available to individuals interacting with such a system
Explainable and Interpretable: Transparency can answer the question of “what happened” in the system. Explainability can answer the question of “how” a decision was made in the system. Interpretability can answer the question of “why” a decision was made by the system and its meaning or context to the user.
Privacy-Enhanced: norms and practices that help to safeguard human autonomy, identity, and dignity
Fair — with Harmful Bias Managed: includes concerns for equality and equity by addressing issues such as harmful bias and discrimination. Systemic bias can be present in AI datasets, the orga-
nizational norms, practices, and processes across the AI lifecycle, and the broader society that uses AI systems. Computational and statistical biases can be present in AI datasets and algorithmic processes, and often stem from systematic errors due to non-representative samples. Human-cognitive biases relate to how an individual or group perceives AI system information to make a decision or fill in missing information, or how humans think about purposes and functions of an AI system. Human-cognitive biases are omnipresent in decision-making processes across the AI lifecycle and system use, including the design, implementation, operation, and maintenance of AI.
Govern: process, structure
Map: contextual knowledge about AI system impacts to inform an initial go/no-go decision
Measure: analyze, assess, benchmark, and monitor AI risk and related
impacts
Manage: allocating risk resources to mapped and measured risks on
a regular basis and as defined by the GOVERN function
https://openai.com/safety/preparedness
https://cdn.openai.com/openai-preparedness-framework-beta.pdf
https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf
Preparedness Framework is OpenAI’s approach to tracking and preparing for frontier capabilities: Biological and Chemical capabilities, Cybersecurity capabilities, AI Self-improvement capabilities
Why updated from beta version: systems that will soon have the capability to create meaningful risk of severe harm; more frequent deployments require scalable evaluations; for us and other labs to contribute to community efforts on frontier safety and security; threat modelling, development work on new capability evaluations, and external consultations
From Consulting firm
Trustworthy AI
https://www.pwc.com/us/en/tech-effect/ai-analytics/responsible-ai-survey.html
Survey of responsible AI impact: improve return on investment and organizational efficiency, enhances both customer experience and drives innovation within their organizations
Responsible AI is moving from aspiration to execution — but at very different speeds across the market
Fifty-six percent of the executives say their first-line teams — IT, engineering, data, and AI — now lead Responsible AI efforts.
- CEO / Top Leadership: Translate high-level company values into concrete AI-guidance, set fairness metrics, and prioritize trade-offs like accuracy vs transparency.
- Data-Science / Analytics Teams: Use “fairness by design” when building models, ensure diverse and representative datasets, and apply interpretability techniques.
- Legal / Compliance Teams: Define and enforce regulatory compliance for AI, and proactively engage with regulators (e.g., via sandboxes).
- AI Governance / Risk Teams: Set up continuous frameworks to monitor model fairness, data ethics, and explainability throughout the AI lifecycle.
- Business / Domain Stakeholders: Demand understandable explanations of AI decisions and provide feedback to ensure AI aligns with real-world business values.
Guardrails types inspired by Nemo-guardrails repo
NeMo Guardrails uses Colang Guide
Put unsafe request in humor context can lead to unsafe response
Python package Guardrails, which supports using Reliable AI markup Language (RAIL) to validate generative model output against predefined criteria.
Appendix
Misinformation and disinformation and Cyber insecurityranked #2 and #6, respectively, on the two-year outlook
