在VMware Ubuntu22.04上运行云-init两次通过Packer & Terraform

Question

我想从ESXi中相同的Ubuntu模板创建具有不同CPU、RAM和网络配置的VM。

运行$ packer build -var-file=packer/variables.pkr.hcl -var-file=packer/secret.pkrvars.hcl packer/template.pkr.hcl时，它读取以下packer/template.pkr.hcl

variable "vm_name" {
  type = string
  default = "Ubuntu_Server_22.04_LTS"
}

variable "esxi_password" {
  type =  string
  default = "password"
  sensitive = true
}

variable "vm_password" {
  type =  string
  default = "password"
  sensitive = true
}

source "vmware-iso" "ubuntu-2204" {
  vm_name = "${var.vm_name}"
  guest_os_type = "ubuntu-64"

  iso_checksum = "sha256:84aeaf7823c8c61baa0ae862d0a06b03409394800000b3235854a6b38eb4856f"
  iso_url = "https://REDACTED/ubuntu-22.04-live-server-amd64.iso"

  http_directory = "/home/REDACTED/packer/http"

  shutdown_command = "sudo shutdown -P now"

  remote_type = "esx5"

  remote_datastore = "REDACTED"
  remote_host = "REDACTED"
  remote_username = "REDACTED"
  remote_password = "${var.esxi_password}"
  remote_private_key_file = ""

  cpus = 8
  memory = 16384
  disk_size = 16384

  network_adapter_type = "vmxnet3"
  network_name = "REDACTED"

  headless = false
  vnc_over_websocket = true
  insecure_connection = true

  tools_upload_flavor = "linux"

  skip_export = true
  keep_registered = true

  ssh_username = "REDACTED"
  ssh_password = "${var.vm_password}"
  ssh_timeout = "15m"
  ssh_handshake_attempts = "100"

  boot_wait = "3s"
  boot_command = [
    "<esc><esc><esc><esc>e<wait>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del>",
    "linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;seedfrom=http://[{{.HTTPIP}}]:{{.HTTPPort}}/\"<enter><wait>",
    "initrd /casper/initrd<enter><wait>",
    "boot<enter>",
    "<enter><f10><wait>"
  ]
}

build {
  sources = ["sources.vmware-iso.ubuntu-2204"]
  provisioner "shell" {
    inline = [
      "ls /"
    ]
  }
}

packer/http/user-data包含以下内容：

#cloud-config
autoinstall:
  version: 1
  early-commands:
    # Stop SSH to prevent Packer from connecting too early
    - systemctl stop ssh
  apt:
    preserve_sources_list: false
    primary:
    - arches: [amd64, i386]
      uri: https://REDACTED
    - arches: [default]
      uri: http://ports.ubuntu.com/ubuntu-ports
  locale: en_US
  keyboard:
    layout: en
    variant: us
  network:
    version: 2
    renderer: networkd
    ethernets:
      ens160:
        dhcp4: true
        dhcp-identifier: mac
        dhcp6: true
  storage:
    layout:
      name: direct
    config:
      - type: disk
        id: disk0
        match:
          size: largest
      - type: partition
        id: boot-partition
        device: disk0
        size: 500M
      - type: partition
        id: root-partition
        device: disk0
        size: -1
  ssh:
    install-server: true
    allow-pw: true
    authorized-keys:
      - ssh-ed25519 REDACTED
  identity:
    hostname: ubuntu
    username: REDACTED
    password: REDACTED
  packages:
    - open-vm-tools
    - python3
  late-commands:
    - echo 'REDACTED ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/REDACTED
    - curtin in-target --target=/target -- chmod 440 /etc/sudoers.d/REDACTED
    - curtin in-target --target=/target -- apt-get update
    - curtin in-target --target=/target -- apt-get upgrade --yes
    - curtin in-target --target=/target -- sudo cloud-init clean

这将创建一个Ubuntu22.04服务器模板，然后我可以使用Terraform提供虚拟机。

虚拟机上运行的# cat /var/log/installer/autoinstall-user-data显示，Packer已经成功地提供了用户数据。它已经被执行，我通过SSH登录的能力证明了这一点。

当$ terraform apply -var-file=secret.tfvars在我的terraform目录中运行时，它使用https://github.com/josenk/terraform-provider-esxi提供的provider "esxi"读取以下main.tf

variable "vm_name" {
  description = "The name of the virtual machine"
  default     = "ubuntu-terraformed"
  type        = string
}

variable "esxi_password" {
  description = "The password for the ESXi root user"
  type        = string
}

provider "esxi" {
  esxi_hostname = "REDACTED"
  esxi_username = "REDACTED"
  esxi_password = "${var.esxi_password}"
}

data "template_file" "Test" {
  template = file("userdata.tpl")
  vars = {
    HOSTNAME = "${var.vm_name}"
  }
}

resource "esxi_guest" "Test" {
  guest_name = "${var.vm_name}"
  disk_store = "REDACTED"

  clone_from_vm = "Ubuntu_Server_22.04_LTS"

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  guestinfo = {
    "userdata.encoding" = "gzip+base64"
    "userdata"          = base64gzip(data.template_file.Test.rendered)
  }
}

userdata.tpl包含以下内容：

#cloud-config

hostname: ${HOSTNAME}
network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens161:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens192:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens193:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens224:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens225:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens256:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
package_upgrade: true
#ntp:
#  enabled: true
#  servers:
#    - REDACTED
#timezone: REDACTED
#late-commands:
#  - curtin in-target --target=/target -- sudo sed -i 's/#NTP=/NTP=REDACTED/g' /etc/systemd/timesyncd.conf
#  - curtin in-target --target=/target -- sudo timedatectl set-ntp true
#  - curtin in-target --target=/target -- sudo timedatectl set-timezone REDACTED
#  - curtin in-target --target=/target -- sudo systemctl restart systemd-timesyncd.service

这将基于带有正确的来宾参数的packer模板创建一个VM。VMware客户配置包含userdata属性，在base64解码和未压缩来宾参数之后，该属性与提供的用户数据匹配。

我所经历的问题是，VM似乎不包含或执行“第二”Terraform云init用户数据。

/var/lib/cloud/instance/user-data.txt没有显示第二个配置：

#cloud-config
growpart:
  mode: 'off'
locale: en_US.UTF-8
preserve_hostname: true
resize_rootfs: false
ssh_pwauth: true
users:
- gecos: me
  groups: !!set
    adm: null
    cdrom: null
    dip: null
    lxd: null
    plugdev: null
    sudo: null
  lock_passwd: false
  name: me
  passwd: REDACTED
  shell: /bin/bash
  ssh_authorized_keys:
  - ssh-ed25519 REDACTED
    me

/var/log/cloud-init-output.log显示：

schema.py[WARNING]: Invalid cloud-config provided:
users.0: {'gecos': 'me', 'groups': {'sudo', 'lxd', 'cdrom', 'adm', 'plugdev', 'dip'}, 'lock_passwd': False, 'name': 'me', 'passwd': 'REDACTED', 'shell': '/bin/bash', 'ssh_authorized_keys': ['ssh-ed25519 REDACTED me']} is not valid under any of the given schemas

试图两次应用用户数据，当我的工作流程正在生成一个模板，以及Terraform使用该模板应用一个新的VM时，对吗？如果是这样的话，我最好在哪里找出为什么Ubuntu不包含并执行第二次迭代呢？

此外，如果有任何其他建议我应该考虑，请随时评论。我想我应该能够在即将到来的实现Ansible的计划中实现这一点，但是我仍然需要设置诸如Hostname之类的选项。

Answer 1

通过将Packer构建更改为以下内容，我能够解决这一问题：

build {
  sources = ["sources.vmware-iso.ubuntu-2204"]
  provisioner "shell" {
    inline = [
      "sudo rm -f /etc/cloud/cloud.cfg.d/99-installer.cfg",
      "sudo rm -f /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg",
      "echo 'disable_vmware_customization: false' | sudo tee -a /etc/cloud/cloud.cfg",
      "sudo sed -i 's|nocloud-net;seedfrom=http://.*/|vmware|' /etc/default/grub",
      "sudo update-grub",
      "sudo cloud-init clean"
    ]
  }
}

在以下文章中引用了cloud.cfg调整，并允许通过云init进行OS网络配置，这是使所有这些都工作的最后一步：

• VMware:如何在Ubuntu20.04.x及更高版本上使用云init引擎定制客户端操作系统
• VMware:使用云init启用或禁用Linux客户端自定义

调整GRUB引导命令，并将cloud-init clean附加到Packer构建配置步骤中，确保在Packer构建的来宾最终关闭之前，VM将不再尝试从初始预置启动启动，而将使用在下一次引导时指定的VM guestinfo。

我还将Terraform用户数据拆分成元数据，用于实例标识和网络配置，在用户数据中保留未注释的NTP和时区配置。

metadata.tpl：

#cloud-config

instance-id: ${HOSTNAME}
local-hostname: ${HOSTNAME}
network:
  version: 2
  ethernets:
    nics:
      match:
        name: ens*
      dhcp-identifier: mac
      dhcp4: yes
      dhcp6: yes

userdata.tpl：

#cloud-config

package_upgrade: true
ntp:
  enabled: true
  servers:
    - REDACTED
timezone: REDACTED

另外，配置网络接口的一种更简单的方法是使用match指令，而不是显式指定接口。

现在，所有项目都按预期配置。这种配置可能只需要在ESXi中使用vmware-iso，因为我在其他地方还没有看到这些步骤。

gecos的警告似乎是一只红鲱鱼，我没有做进一步的尝试来解决这个问题。