使用具有大型 YAML 变量的循环时，Ansible 性能缓慢

Question

开发者社区您好！

我一直致力于开发一些 Ansible playbook 来管理 Citrix NetScaler 配置，并希望获得有关以下内容的一些帮助。我在名为

nsapp_lb_server

的变量中定义了以下数据结构：

nsapp_lb_server:
    - name:                      "SRV-1"
      ipaddress:                 "10.102.102.1"
      comment:                   "Chewbacca"

    - name:                      "SRV-2"
      ipaddress:                 "10.102.102.2"
      comment:                   "C-3PO"

    - name:                      "SRV-3"
      ipaddress:                 "10.102.102.3"
      comment:                   "Obi-Wan Kenobi"
...

[+ another 1200 item...]

我有以下任务：

  - name: "Check variables (loop)"
    ansible.builtin.assert:
        that:
            - ( (item.name is defined) and (item.name | length > 0) )
            - ( (item.ipaddress is defined) and (item.ipaddress | ipaddr() == item.ipaddress) )
            - ( (item.comment | length > 0) if (item.comment is defined) else omit )
    loop: "{{ nsapp_lb_server }}"

我的问题是，当我在 nsapp_lb_server 变量中有

数千条记录

时，循环速度极其慢。任务在 30 分钟内完成，这是一个很长的时间......:-(

在互联网上进行一些挖掘后，似乎问题是由Ansible“循环”功能引起的，所以我想检查是否有任何其他方法可以使用而不是循环。

Ansible“循环”是否有任何替代方案可以提供相同的结果（循环变量的条目）？我正在考虑使用 json_query，但仍然不知道如何在这种特定情况下实现它。

我的环境：

$ ansible --version
ansible [core 2.12.6]
  config file = /home/ansible/.ansible.cfg
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ansible/ansible/lib/python3.9/site-packages/ansible
  ansible collection location = /home/ansible/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ansible/ansible/bin/ansible
  python version = 3.9.7 (default, Sep 21 2021, 00:13:39) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 3.0.3
  libyaml = True

有人可以指出我正确的方向吗？我已经在我的代码集上工作了很长时间，在用大数据测试代码后，由于运行时间的原因，代码似乎毫无用处。我还检查了分配给运行 Ansible 控制器的虚拟机的硬件资源，没有任何问题。

提前非常感谢！

Answer 1

作为数千个单独的任务运行此验证非常慢，因为它增加了大量的执行和回调开销。您可以在单个任务中完成此操作，但需要注意的是，追踪无效列表项将变得更加困难：

- hosts: localhost
  gather_facts: false
  vars:
    nsapp_lb_server: "{{ nsapp_lb_samples * 10000 }}"
    nsapp_lb_samples:
        - name:                      "SRV-1"
          ipaddress:                 "10.102.102.1"
          comment:                   "Chewbacca"
        - name:                      "SRV-2"
          ipaddress:                 "10.102.102.2"
          comment:                   "C-3PO"
        - name:                      "SRV-3"
          ipaddress:                 "10.102.102.3"
          comment:                   "Obi-Wan Kenobi"
  tasks:
    - assert:
        that:
          - nsapp_lb_server | rejectattr('name') | length == 0
          - (nsapp_lb_server | map(attribute='ipaddress') | map('ipaddr')) == (nsapp_lb_server | map(attribute='ipaddress'))
          - nsapp_lb_server | selectattr('comment', 'defined') | rejectattr('comment') | length == 0

我输入的 30,000 个测试条目的运行时间约为 5 秒。

为了更容易找到坏值而不使任务变得极其难看，您可以将其分成一系列任务：

- hosts: localhost
  gather_facts: false
  vars:
    nsapp_lb_server: "{{ nsapp_lb_samples * 10000 }}"
    nsapp_lb_samples:
        - name:                      "SRV-1"
          ipaddress:                 "10.102.102.1"
          comment:                   "Chewbacca"
        - name:                      "SRV-2"
          ipaddress:                 "10.102.102.2"
          comment:                   "C-3PO"
        - name:                      "SRV-3"
          ipaddress:                 "10.102.102.3"
          comment:                   "Obi-Wan Kenobi"
  tasks:
    - name: Check for missing names
      assert:
        that: nsapp_lb_server | rejectattr('name', 'defined') | length == 0
        fail_msg: "Bad entries: {{ nsapp_lb_server | rejectattr('name', 'defined') }}"

    - name: Check for bad names
      assert:
        that: nsapp_lb_server | rejectattr('name') | length == 0
        fail_msg: "Bad entries: {{ nsapp_lb_server | rejectattr('name') }}"

    - name: Check for missing IP addresses
      assert:
        that: nsapp_lb_server | rejectattr('ipaddress', 'defined') | length == 0
        fail_msg: "Bad entries: {{ nsapp_lb_server | rejectattr('ipaddress', 'defined') }}"

    - name: Check for bad IP addresses
      assert:
        that: (nsapp_lb_server | map(attribute='ipaddress') | map('ipaddr')) == (nsapp_lb_server | map(attribute='ipaddress'))
        fail_msg: "Suspicious values: {{ nsapp_lb_server | map(attribute='ipaddress') | map('ipaddr') | symmetric_difference(nsapp_lb_server | map(attribute='ipaddress')) }}"

    - name: Check for bad comments
      assert:
        that: nsapp_lb_server | selectattr('comment', 'defined') | rejectattr('comment') | length == 0
        fail_msg: "Bad entries: {{ nsapp_lb_server | selectattr('comment', 'defined') | rejectattr('comment') }}"

对于 30,000 个测试条目的相同列表，运行时间约为 12 秒。

Answer 2

我有严重的性能问题

with_items

，这些任务是整个 Runbook 中最昂贵的单个任务。

my-service : Set 'client_internal_ids' --- 62.13s
my-service : Set 'client_internal_ids' --- 61.73s

任务非常简单：

- name: "Set 'client_internal_ids'"
  set_fact:
    client_internal_ids: "{{ client_internal_ids|default({}) | combine ( { item.clientId: item.id } ) }}"
  no_log: true
  with_items:
    - "{{ all_clients_response.json }}"

这就是为什么我创建一个专用插件来使用简单的过滤器（在

filter_plugins

下）转换内联字典：

def items2dict(input: any, key: any, value: any = None):
    """Converts items to a single dict without using with_items|loop|combine"""
    target_dict = {}

    if isinstance(input, list):
        source = input
    elif isinstance(input, dict):
        source = input.items()
    else:
        source = input.__dict__.items()

    for item in source:
        if key in item:
            key_val = item[key]
        else:
            key_val = key

        if value in item:
            value_val = item[value]
        else:
            value_val = value

        target_dict[key_val] = value_val

    return target_dict

用途：

- name: "Set 'client_internal_ids'"
  set_fact:
    client_internal_ids: "{{ all_clients_response.json | items2dict('clientId','id') }}"
  no_log: true

其中

clientId

是项目键，

id

是项目值。

该任务不再显示在分析列表中。