将 Autogen 与 Nvidia NIM 上的 Llama 模型集成

Question

我正在尝试为我的代理使用 Nividia“https://integrate.api.nvidia.com/v1”上的“meta/llama-3.1-8b-instruct”。

import autogen

config_list = [
    
    {
        "model": "meta/llama-3.1-8b-instruct",
        "base_url": "https://integrate.api.nvidia.com/v1",
        "api_key": "nvapi-API key",
        "temperature": 0.2,
        "top_p": 0.7,
        "max_tokens": 1024,
    }
    
]

llm_config={"config_list": config_list, "seed": 42}


user_proxy = autogen.UserProxyAgent(
    name="User_proxy",
    system_message="A human admin.",
    code_execution_config={
        "last_n_messages": 2,
        "work_dir": "groupchat",
        "use_docker": False,
    },  # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.
    human_input_mode="TERMINATE",
)
climate_activist = autogen.AssistantAgent(
    name="climate_activist",
    llm_config=llm_config,
    system_message="You are a climate activist with vast knowledge on climate change and ways to prevent further climate change."
)

rocket_scientist = autogen.AssistantAgent(
    name="rocket_scientist",
    llm_config=llm_config,
    system_message="You are a rocket scientist with vast knowledge on how to design rockets to be efficient. You also take the effect of your rockets on the climate when designing it."
)

# Create group and group manager
groupchat = autogen.GroupChat(
    agents=[user_proxy, climate_activist, rocket_scientist], messages=[], max_round=3)
manager = autogen.GroupChatManager(
    groupchat=groupchat, llm_config=llm_config)

# initialize conversations
user_proxy.initiate_chat(
    manager, message="firstly tell what LLM, version and baseurl you are using. Discuss on the best way to design rockets to be environmentally friendly and efficient."
)

从输出来看，它没有在 Nividia 上使用正确的模型“meta/llama-3.1-8b-instruct”。

❯ python group_chat.py
User_proxy (to chat_manager):

firstly tell what LLM, version and baseurl you are using. Discuss on the best way to design rockets to be environmentally friendly and efficient.

--------------------------------------------------------------------------------

Next speaker: rocket_scientist

rocket_scientist (to chat_manager):

I'm using the LLaMA (Large Language Model) version 3.3.1, which is a state-of-the-art language model developed by Meta AI. The base URL for this model is `https://huggingface.co/models/llama/3.3.1`.

这里是 Nvidia NIM 提供的示例 Python 代码

from openai import OpenAI

client = OpenAI(
  base_url = "https://integrate.api.nvidia.com/v1",
  api_key = "nvapi-API Key"
)

completion = client.chat.completions.create(
  model="meta/llama-3.1-8b-instruct",
  messages=[{"role":"user","content":"who are u"}],
  temperature=0.2,
  top_p=0.7,
  max_tokens=1024,
  stream=True
)

for chunk in completion:
  if chunk.choices[0].delta.content is not None:
    print(chunk.choices[0].delta.content, end="")

有什么想法吗？

Answer 1

您是否成功将 Autogen 与 NVIDIA NIM 结合使用？如果是这样，你能提供更新吗？谢谢