Skip to main content
University of Oxford Department of Computer Science
  Contact Us     Search  
  1. Home
  2. Student Space
  3. Online Resources, Handbooks & Library
  4. Student projects - Cooperative Capabilities in Multi-Agent AI Systems: Formalisation, Training, and Evaluation

Cooperative Capabilities in Multi-Agent AI Systems: Formalisation, Training, and Evaluation

Supervisors

Imran Hashmi
Michael Wooldridge

Suitable for

MSc in Advanced Computer Science

Abstract

This thesis investigates the specification, emergence, and evaluation of cooperative capabilities in multi-agent AI systems. Cooperative capabilities are defined as functional properties that enable agents to collaborate effectively toward collective objectives. Two primary methodologies: (a) Game-theoretic modelling and (b) Multi-Agent Reinforcement Learning (MARL), will be explored to formalise, train, and evaluate cooperative behaviour. Through a case study involving resource allocation in multi-agent systems, this research demonstrates how agents can overcome challenges of cooperative AI. The main contribution of this thesis will be a comprehensive framework and actionable strategies to design and evaluate cooperative systems, enabling effective collaboration in dynamic and complex environments.

 

Goals and Objectives

  1. Formalisation of Cooperative Capabilities
    • Identify key aspects of cooperative capabilities: communication, coordination, negotiation, and collective planning.
    • Create a taxonomy categorising these capabilities.
    • Develop formal definitions of cooperative capabilities using game-theoretic models and MARL frameworks.
  2. Metrics for Rigorous Evaluation
    • Design robust, interpretable metrics for measuring cooperative capabilities.
    • Validate metrics using theoretical models and empirical results.
  3. Training and Emergence Studies
    • Investigate conditions under which cooperative capabilities emerge, considering the differences in training techniques between game theory-based modelling and MARL.
    • Perform empirical experiments in multi-agent settings to evaluate cooperative behaviours.
  4. Impact Analysis of Asymmetries
    • Asymmetric agent capabilities refer to differences in the abilities, resources, or knowledge that agents possess in a multi-agent system.
    • Model scenarios with asymmetric agent capabilities and evaluate their effects on collective outcomes.
    • Propose methods to mitigate risks from imbalances, such as reward redistribution or coalition-building mechanisms.
  5. Strategies for Fostering Cooperation
    • Suggest interventions to enhance cooperative outcomes using methods like fine-tuning policies or designing cooperative reward structures.
    • Adapt interventions for game-theoretic and MARL-based systems to test their effectiveness.

Tasks

  1. Literature Review and Taxonomy Development
    • Review existing research on cooperative AI in both game-theoretic and MARL paradigms.
    • Develop a comprehensive taxonomy of cooperative capabilities.
  1. Formal Specification (Game Theory)
    • Define cooperative behaviours using game-theoretic models such as Nash equilibrium, Pareto efficiency, and cooperative game solutions.
    • Specify desirable and undesirable behaviours while integrating ethical considerations.
  1. Learning-Based Modelling (MARL)
    • Train agents using MARL frameworks with cooperative policies, such as value decomposition networks or shared reward mechanisms.
    • Design reward structures and communication protocols to foster cooperation.
  1. Design of Evaluation Metrics
    • Create and validate task-specific metrics for both approaches (e.g., efficiency, fairness, and stability).
  1. Simulation and Empirical Studies
    • For Game Theory: Design controlled settings to explore theoretical cooperation strategies under bounded rationality and asymmetry in Agent-based simulation frameworks
    • For MARL: Simulate dynamic multi-agent environments with emergent behaviours using platforms like OpenAI Gym or Unity ML-Agents.

 

 

  1. Impact Analysis
    • Investigate the influence of asymmetries (e.g., mobility, resource access) on cooperation in game-theoretic and MARL systems.
    • Develop strategies to mitigate negative impacts in each framework.
  1. Intervention Design
    • Propose and implement interventions, such as dynamic coalitions or hybrid training strategies.
    • Test interventions across both approaches to assess their scalability and robustness.

 

Research Methodology

  • Game-Theoretic Modelling
  1. Theoretical Framework
    • Define agent interactions in an ABM using cooperative and non-cooperative game theory.
  2. Verification Tools
    • Use formal methods like probabilistic model checking (e.g., PRISM) to validate theoretical models.
  3. Simulation and Analysis
    • Create small-scale simulations to test theoretical predictions in controlled environments.
  • Multi-Agent Reinforcement Learning
  1. Model Training
    • Train agents in MARL environments
    • Implement cooperative reward structures and communication protocols.
  2. Simulation Environments
    • Use platforms like OpenAI Gym, Unity ML-Agents, or custom-built environments to train and test agents.
  3. Experimental Analysis
    • Test training strategies to evaluate emergent cooperative behaviours.

 

Case Study: Multi-Agent Resource Allocation Game

Objective

Evaluate cooperative capabilities in a resource-constrained environment using game-theoretic modelling or MARL.

Scenario

  • Environment: A grid world where agents compete for limited resources such as food, water, and energy.
  • Goal: Balance individual needs with collective objectives through cooperative mechanisms.

Tasks

  1. Baseline Experiment
    • Game Theory: Define the resource-sharing game and compute theoretical cooperative solutions (e.g., Pareto-optimal allocations).
    • MARL: Implement baseline training for agents to collect resources individually without cooperation.
  2. Cooperative Experiment
    • Game Theory: Introduce incentive schemes or binding agreements to encourage cooperation and measure changes in efficiency.
    • MARL: Enable agent communication and train cooperative policies to optimise resource allocation collectively.
  3. Asymmetry Experiment
    • Game Theory: Introduce capability asymmetries (e.g., restricted resource access) and analyse their impact on cooperation.
    • MARL: Simulate agents with heterogeneous abilities and explore strategies to mitigate disparities.
  4. Metric Validation
    • Apply proposed metrics to quantify cooperation levels and validate their effectiveness for both methods.
  5. Intervention Design
    • Develop strategies for improving cooperation, such as adaptive reward redistribution or prioritising weaker agents.
    • Test interventions using both game-theoretic and MARL setups.
  6. Reflection
    • Compare outcomes of the game-theoretic and MARL approaches.
    • Propose improvements and future directions for research.

 

Technical Requirements

Suitability: MSc

Prerequisites

  • Proficiency in Python and familiarity with either game theory, agent-based modelling, reinforcement learning.
  • Knowledge of MARL libraries (e.g., PyTorch, TensorFlow) or formal verification tools (e.g., PRISM).

Software and Tools

  • Game Theory: ABM framework (e.g., MESA, Agent.JL, BEAST)
  • Formal verification tools (PRISM, SPIN).
  • MARL: Simulation platforms (OpenAI Gym, Unity ML-Agents).

Hardware Requirements

  • Access to GPU-enabled machines for ABM simulations and MARL training.

Deliverables

  1. Formal Definitions and Taxonomy
    • Detailed formal definitions of cooperative capabilities, including a comprehensive taxonomy.
  2. Metrics and Evaluation Tools
    • Validated metrics and tools applicable to game-theoretic and MARL approaches.
  3. Empirical Results
    • Comparative analysis of cooperative behaviours under both paradigms.
  4. Strategies for Enhancing Cooperation
    • Interventions tailored for game-theoretic and MARL systems, with tested results and guidelines.

 

References

  1. Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., & Graepel, T. (2020), Open Problems in Cooperative AI. arXiv preprint arXiv:2012.08630.
  2. Conitzer, V., & Oesterheld, C. (2023) Foundations of Cooperative AI. Proceedings of the AAAI Conference on Artificial Intelligence, 37(13), 15359-15367.
  3. Barton, S. L., Waytowich, N. R., Zaroukian, E., & Asher, D. E. (2019), Measuring Collaborative Emergent Behavior in Multi-Agent Reinforcement Learning. In Human Systems Engineering and Design: Proceedings of the 1st International Conference on Human Systems Engineering and Design (IHSED2018) (pp. 422-427). Springer International Publishing.
  4. Shoham, Y., & Leyton-Brown, K. (2008), Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge University Press.
  5. An Introduction to MultiAgent Systems - Second Edition by Michael Wooldridge   Published May 2009  by John Wiley & Sons
  6. Multi-Agent Reinforcement Learning Foundations and Modern Approaches By Stefano V. Albrecht, Filippos Christianos, Lukas Schäfer · 2024
  7. Omicini, A., Petta, P., & Pitt, J. (Eds.). (2004) Engineering Societies in the Agents World IV. Springer.
  8. Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jaderberg, M., Czarnecki, W. M., ... & Silver, D. (2019).

Student Space

Back to Top