kubespray 2.24 추가

contrib/terraform/upcloud/README.md

@@ -0,0 +1,143 @@
+# Kubernetes on UpCloud with Terraform
+
+Provision a Kubernetes cluster on [UpCloud](https://upcloud.com/) using Terraform and Kubespray
+
+## Overview
+
+The setup looks like following
+
+```text
+   Kubernetes cluster
++--------------------------+
+|      +--------------+    |
+|      | +--------------+  |
+| -->  | |              |  |
+|      | | Master/etcd  |  |
+|      | | node(s)      |  |
+|      +-+              |  |
+|        +--------------+  |
+|              ^           |
+|              |           |
+|              v           |
+|      +--------------+    |
+|      | +--------------+  |
+| -->  | |              |  |
+|      | |    Worker    |  |
+|      | |    node(s)   |  |
+|      +-+              |  |
+|        +--------------+  |
++--------------------------+
+```
+
+The nodes uses a private network for node to node communication and a public interface for all external communication.
+
+## Requirements
+
+* Terraform 0.13.0 or newer
+
+## Quickstart
+
+NOTE: Assumes you are at the root of the kubespray repo.
+
+For authentication in your  cluster you can use the environment variables.
+
+```bash
+export TF_VAR_UPCLOUD_USERNAME=username
+export TF_VAR_UPCLOUD_PASSWORD=password
+```
+
+To allow API access to your UpCloud account, you need to allow API connections by visiting [Account-page](https://hub.upcloud.com/account) in your UpCloud Hub.
+
+Copy the cluster configuration file.
+
+```bash
+CLUSTER=my-upcloud-cluster
+cp -r inventory/sample inventory/$CLUSTER
+cp contrib/terraform/upcloud/cluster-settings.tfvars inventory/$CLUSTER/
+export ANSIBLE_CONFIG=ansible.cfg
+cd inventory/$CLUSTER
+```
+
+Edit  `cluster-settings.tfvars`  to match your requirement.
+
+Run Terraform to create the infrastructure.
+
+```bash
+terraform init ../../contrib/terraform/upcloud
+terraform apply --var-file cluster-settings.tfvars \
+    -state=tfstate-$CLUSTER.tfstate \
+     ../../contrib/terraform/upcloud/
+```
+
+You should now have a inventory file named `inventory.ini` that you can use with kubespray.
+You can use the inventory file with kubespray to set up a cluster.
+
+It is a good idea to check that you have basic SSH connectivity to the nodes. You can do that by:
+
+```bash
+ansible -i inventory.ini -m ping all
+```
+
+You can setup Kubernetes with kubespray using the generated inventory:
+
+```bash
+ansible-playbook -i inventory.ini ../../cluster.yml -b -v
+```
+
+## Teardown
+
+You can teardown your infrastructure using the following Terraform command:
+
+```bash
+terraform destroy --var-file cluster-settings.tfvars \
+      -state=tfstate-$CLUSTER.tfstate \
+      ../../contrib/terraform/upcloud/
+```
+
+## Variables
+
+* `prefix`: Prefix to add to all resources, if set to "" don't set any prefix
+* `template_name`: The name or UUID  of a base image
+* `username`: a user to access the nodes, defaults to "ubuntu"
+* `private_network_cidr`: CIDR to use for the private network, defaults to "172.16.0.0/24"
+* `ssh_public_keys`: List of public SSH keys to install on all machines
+* `zone`: The zone where to run the cluster
+* `machines`: Machines to provision. Key of this object will be used as the name of the machine
+  * `node_type`: The role of this node *(master|worker)*
+  * `plan`: Preconfigured cpu/mem plan to use (disables `cpu` and `mem` attributes below)
+  * `cpu`: number of cpu cores
+  * `mem`: memory size in MB
+  * `disk_size`: The size of the storage in GB
+  * `additional_disks`: Additional disks to attach to the node.
+    * `size`: The size of the additional disk in GB
+    * `tier`: The tier of disk to use (`maxiops` is the only one you can choose atm)
+* `firewall_enabled`: Enable firewall rules
+* `firewall_default_deny_in`: Set the firewall to deny inbound traffic by default. Automatically adds UpCloud DNS server and NTP port allowlisting.
+* `firewall_default_deny_out`: Set the firewall to deny outbound traffic by default.
+* `master_allowed_remote_ips`: List of IP ranges that should be allowed to access API of masters
+  * `start_address`: Start of address range to allow
+  * `end_address`: End of address range to allow
+* `k8s_allowed_remote_ips`: List of IP ranges that should be allowed SSH access to all nodes
+  * `start_address`: Start of address range to allow
+  * `end_address`: End of address range to allow
+* `master_allowed_ports`: List of port ranges that should be allowed to access the masters
+  * `protocol`: Protocol *(tcp|udp|icmp)*
+  * `port_range_min`: Start of port range to allow
+  * `port_range_max`: End of port range to allow
+  * `start_address`: Start of address range to allow
+  * `end_address`: End of address range to allow
+* `worker_allowed_ports`: List of port ranges that should be allowed to access the workers
+  * `protocol`: Protocol *(tcp|udp|icmp)*
+  * `port_range_min`: Start of port range to allow
+  * `port_range_max`: End of port range to allow
+  * `start_address`: Start of address range to allow
+  * `end_address`: End of address range to allow
+* `loadbalancer_enabled`: Enable managed load balancer
+* `loadbalancer_plan`: Plan to use for load balancer *(development|production-small)*
+* `loadbalancers`: Ports to load balance and which machines to forward to. Key of this object will be used as the name of the load balancer frontends/backends
+  * `port`: Port to load balance.
+  * `target_port`: Port to the backend servers.
+  * `backend_servers`: List of servers that traffic to the port should be forwarded to.
+* `server_groups`: Group servers together
+  * `servers`: The servers that should be included in the group.
+  * `anti_affinity_policy`: Defines if a server group is an anti-affinity group. Setting this to "strict" or yes" will result in all servers in the group being placed on separate compute hosts. The value can be "strict", "yes" or "no". "strict" refers to strict policy doesn't allow servers in the same server group to be on the same host. "yes" refers to best-effort policy and tries to put servers on different hosts, but this is not guaranteed.

contrib/terraform/upcloud/cluster-settings.tfvars

@@ -0,0 +1,148 @@
+# See: https://developers.upcloud.com/1.3/5-zones/
+zone     = "fi-hel1"
+username = "ubuntu"
+
+# Prefix to use for all resources to separate them from other resources
+prefix = "kubespray"
+
+inventory_file = "inventory.ini"
+
+#  Set the operating system using UUID or exact name
+template_name = "Ubuntu Server 20.04 LTS (Focal Fossa)"
+
+ssh_public_keys = [
+  # Put your public SSH key here
+  "ssh-rsa public key 1",
+  "ssh-rsa public key 2",
+]
+
+# check list of available plan https://developers.upcloud.com/1.3/7-plans/
+machines = {
+  "control-plane-0" : {
+    "node_type" : "master",
+    # plan to use instead of custom cpu/mem
+    "plan" : null,
+    #number of cpu cores
+    "cpu" : "2",
+    #memory size in MB
+    "mem" : "4096"
+    # The size of the storage in GB
+    "disk_size" : 250
+    "additional_disks" : {}
+  },
+  "worker-0" : {
+    "node_type" : "worker",
+    # plan to use instead of custom cpu/mem
+    "plan" : null,
+    #number of cpu cores
+    "cpu" : "2",
+    #memory size in MB
+    "mem" : "4096"
+    # The size of the storage in GB
+    "disk_size" : 250
+    "additional_disks" : {
+      # "some-disk-name-1": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # },
+      # "some-disk-name-2": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # }
+    }
+  },
+  "worker-1" : {
+    "node_type" : "worker",
+    # plan to use instead of custom cpu/mem
+    "plan" : null,
+    #number of cpu cores
+    "cpu" : "2",
+    #memory size in MB
+    "mem" : "4096"
+    # The size of the storage in GB
+    "disk_size" : 250
+    "additional_disks" : {
+      # "some-disk-name-1": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # },
+      # "some-disk-name-2": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # }
+    }
+  },
+  "worker-2" : {
+    "node_type" : "worker",
+    # plan to use instead of custom cpu/mem
+    "plan" : null,
+    #number of cpu cores
+    "cpu" : "2",
+    #memory size in MB
+    "mem" : "4096"
+    # The size of the storage in GB
+    "disk_size" : 250
+    "additional_disks" : {
+      # "some-disk-name-1": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # },
+      # "some-disk-name-2": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # }
+    }
+  }
+}
+
+firewall_enabled          = false
+firewall_default_deny_in  = false
+firewall_default_deny_out = false
+
+master_allowed_remote_ips = [
+  {
+    "start_address" : "0.0.0.0"
+    "end_address" : "255.255.255.255"
+  }
+]
+
+k8s_allowed_remote_ips = [
+  {
+    "start_address" : "0.0.0.0"
+    "end_address" : "255.255.255.255"
+  }
+]
+
+master_allowed_ports = []
+worker_allowed_ports = []
+
+loadbalancer_enabled = false
+loadbalancer_plan    = "development"
+loadbalancers = {
+  # "http" : {
+  #   "port" : 80,
+  #   "target_port" : 80,
+  #   "backend_servers" : [
+  #     "worker-0",
+  #     "worker-1",
+  #     "worker-2"
+  #   ]
+  # }
+}
+
+server_groups = {
+  # "control-plane" = {
+  #   servers = [
+  #     "control-plane-0"
+  #   ]
+  #   anti_affinity_policy = "strict"
+  # },
+  # "workers" = {
+  #   servers = [
+  #     "worker-0",
+  #     "worker-1",
+  #     "worker-2"
+  #   ]
+  #   anti_affinity_policy = "yes"
+  # }
+}

contrib/terraform/upcloud/main.tf

@@ -0,0 +1,73 @@
+
+terraform {
+  required_version = ">= 0.13.0"
+}
+provider "upcloud" {
+  # Your UpCloud credentials are read from  environment variables:
+  username = var.UPCLOUD_USERNAME
+  password = var.UPCLOUD_PASSWORD
+}
+
+module "kubernetes" {
+  source = "./modules/kubernetes-cluster"
+
+  prefix = var.prefix
+  zone   = var.zone
+
+  template_name = var.template_name
+  username      = var.username
+
+  private_network_cidr = var.private_network_cidr
+
+  machines = var.machines
+
+  ssh_public_keys = var.ssh_public_keys
+
+  firewall_enabled          = var.firewall_enabled
+  firewall_default_deny_in  = var.firewall_default_deny_in
+  firewall_default_deny_out = var.firewall_default_deny_out
+  master_allowed_remote_ips = var.master_allowed_remote_ips
+  k8s_allowed_remote_ips    = var.k8s_allowed_remote_ips
+  master_allowed_ports      = var.master_allowed_ports
+  worker_allowed_ports      = var.worker_allowed_ports
+
+  loadbalancer_enabled = var.loadbalancer_enabled
+  loadbalancer_plan    = var.loadbalancer_plan
+  loadbalancers        = var.loadbalancers
+
+  server_groups = var.server_groups
+}
+
+#
+# Generate ansible inventory
+#
+
+data "template_file" "inventory" {
+  template = file("${path.module}/templates/inventory.tpl")
+
+  vars = {
+    connection_strings_master = join("\n", formatlist("%s ansible_user=ubuntu ansible_host=%s ip=%s etcd_member_name=etcd%d",
+      keys(module.kubernetes.master_ip),
+      values(module.kubernetes.master_ip).*.public_ip,
+      values(module.kubernetes.master_ip).*.private_ip,
+    range(1, length(module.kubernetes.master_ip) + 1)))
+    connection_strings_worker = join("\n", formatlist("%s ansible_user=ubuntu ansible_host=%s ip=%s",
+      keys(module.kubernetes.worker_ip),
+      values(module.kubernetes.worker_ip).*.public_ip,
+    values(module.kubernetes.worker_ip).*.private_ip))
+    list_master = join("\n", formatlist("%s",
+    keys(module.kubernetes.master_ip)))
+    list_worker = join("\n", formatlist("%s",
+    keys(module.kubernetes.worker_ip)))
+  }
+}
+
+resource "null_resource" "inventories" {
+  provisioner "local-exec" {
+    command = "echo '${data.template_file.inventory.rendered}' > ${var.inventory_file}"
+  }
+
+  triggers = {
+    template = data.template_file.inventory.rendered
+  }
+}

contrib/terraform/upcloud/modules/kubernetes-cluster/main.tf

@@ -0,0 +1,558 @@
+locals {
+  # Create a list of all disks to create
+  disks = flatten([
+    for node_name, machine in var.machines : [
+      for disk_name, disk in machine.additional_disks : {
+        disk      = disk
+        disk_name = disk_name
+        node_name = node_name
+      }
+    ]
+  ])
+
+  lb_backend_servers = flatten([
+    for lb_name, loadbalancer in var.loadbalancers : [
+      for backend_server in loadbalancer.backend_servers : {
+        port        = loadbalancer.target_port
+        lb_name     = lb_name
+        server_name = backend_server
+      }
+    ]
+  ])
+
+  # If prefix is set, all resources will be prefixed with "${var.prefix}-"
+  # Else don't prefix with anything
+  resource-prefix = "%{if var.prefix != ""}${var.prefix}-%{endif}"
+}
+
+resource "upcloud_network" "private" {
+  name = "${local.resource-prefix}k8s-network"
+  zone = var.zone
+
+  ip_network {
+    address = var.private_network_cidr
+    dhcp    = true
+    family  = "IPv4"
+  }
+}
+
+resource "upcloud_storage" "additional_disks" {
+  for_each = {
+    for disk in local.disks : "${disk.node_name}_${disk.disk_name}" => disk.disk
+  }
+
+  size  = each.value.size
+  tier  = each.value.tier
+  title = "${local.resource-prefix}${each.key}"
+  zone  = var.zone
+}
+
+resource "upcloud_server" "master" {
+  for_each = {
+    for name, machine in var.machines :
+    name => machine
+    if machine.node_type == "master"
+  }
+
+  hostname = "${local.resource-prefix}${each.key}"
+  plan     = each.value.plan
+  cpu      = each.value.plan == null ? each.value.cpu : null
+  mem      = each.value.plan == null ? each.value.mem : null
+  zone     = var.zone
+
+  template {
+    storage = var.template_name
+    size    = each.value.disk_size
+  }
+
+  # Public network interface
+  network_interface {
+    type = "public"
+  }
+
+  # Private network interface
+  network_interface {
+    type    = "private"
+    network = upcloud_network.private.id
+  }
+
+  # Ignore volumes created by csi-driver
+  lifecycle {
+    ignore_changes = [storage_devices]
+  }
+
+  firewall = var.firewall_enabled
+
+  dynamic "storage_devices" {
+    for_each = {
+      for disk_key_name, disk in upcloud_storage.additional_disks :
+      disk_key_name => disk
+      # Only add the disk if it matches the node name in the start of its name
+      if length(regexall("^${each.key}_.+", disk_key_name)) > 0
+    }
+
+    content {
+      storage = storage_devices.value.id
+    }
+  }
+
+  # Include at least one public SSH key
+  login {
+    user            = var.username
+    keys            = var.ssh_public_keys
+    create_password = false
+  }
+}
+
+resource "upcloud_server" "worker" {
+  for_each = {
+    for name, machine in var.machines :
+    name => machine
+    if machine.node_type == "worker"
+  }
+
+  hostname = "${local.resource-prefix}${each.key}"
+  plan     = each.value.plan
+  cpu      = each.value.plan == null ? each.value.cpu : null
+  mem      = each.value.plan == null ? each.value.mem : null
+  zone     = var.zone
+
+  template {
+    storage = var.template_name
+    size    = each.value.disk_size
+  }
+
+  # Public network interface
+  network_interface {
+    type = "public"
+  }
+
+  # Private network interface
+  network_interface {
+    type    = "private"
+    network = upcloud_network.private.id
+  }
+
+  # Ignore volumes created by csi-driver
+  lifecycle {
+    ignore_changes = [storage_devices]
+  }
+
+  firewall = var.firewall_enabled
+
+  dynamic "storage_devices" {
+    for_each = {
+      for disk_key_name, disk in upcloud_storage.additional_disks :
+      disk_key_name => disk
+      # Only add the disk if it matches the node name in the start of its name
+      if length(regexall("^${each.key}_.+", disk_key_name)) > 0
+    }
+
+    content {
+      storage = storage_devices.value.id
+    }
+  }
+
+  # Include at least one public SSH key
+  login {
+    user            = var.username
+    keys            = var.ssh_public_keys
+    create_password = false
+  }
+}
+
+resource "upcloud_firewall_rules" "master" {
+  for_each  = upcloud_server.master
+  server_id = each.value.id
+
+  dynamic "firewall_rule" {
+    for_each = var.master_allowed_remote_ips
+
+    content {
+      action                 = "accept"
+      comment                = "Allow master API access from this network"
+      destination_port_end   = "6443"
+      destination_port_start = "6443"
+      direction              = "in"
+      family                 = "IPv4"
+      protocol               = "tcp"
+      source_address_end     = firewall_rule.value.end_address
+      source_address_start   = firewall_rule.value.start_address
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = length(var.master_allowed_remote_ips) > 0 ? [1] : []
+
+    content {
+      action                 = "drop"
+      comment                = "Deny master API access from other networks"
+      destination_port_end   = "6443"
+      destination_port_start = "6443"
+      direction              = "in"
+      family                 = "IPv4"
+      protocol               = "tcp"
+      source_address_end     = "255.255.255.255"
+      source_address_start   = "0.0.0.0"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.k8s_allowed_remote_ips
+
+    content {
+      action                 = "accept"
+      comment                = "Allow SSH from this network"
+      destination_port_end   = "22"
+      destination_port_start = "22"
+      direction              = "in"
+      family                 = "IPv4"
+      protocol               = "tcp"
+      source_address_end     = firewall_rule.value.end_address
+      source_address_start   = firewall_rule.value.start_address
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = length(var.k8s_allowed_remote_ips) > 0 ? [1] : []
+
+    content {
+      action                 = "drop"
+      comment                = "Deny SSH from other networks"
+      destination_port_end   = "22"
+      destination_port_start = "22"
+      direction              = "in"
+      family                 = "IPv4"
+      protocol               = "tcp"
+      source_address_end     = "255.255.255.255"
+      source_address_start   = "0.0.0.0"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.master_allowed_ports
+
+    content {
+      action                 = "accept"
+      comment                = "Allow access on this port"
+      destination_port_end   = firewall_rule.value.port_range_max
+      destination_port_start = firewall_rule.value.port_range_min
+      direction              = "in"
+      family                 = "IPv4"
+      protocol               = firewall_rule.value.protocol
+      source_address_end     = firewall_rule.value.end_address
+      source_address_start   = firewall_rule.value.start_address
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "UpCloud DNS"
+      source_port_end      = "53"
+      source_port_start    = "53"
+      direction            = "in"
+      family               = "IPv4"
+      protocol             = firewall_rule.value
+      source_address_end   = "94.237.40.9"
+      source_address_start = "94.237.40.9"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "UpCloud DNS"
+      source_port_end      = "53"
+      source_port_start    = "53"
+      direction            = "in"
+      family               = "IPv4"
+      protocol             = firewall_rule.value
+      source_address_end   = "94.237.127.9"
+      source_address_start = "94.237.127.9"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "UpCloud DNS"
+      source_port_end      = "53"
+      source_port_start    = "53"
+      direction            = "in"
+      family               = "IPv6"
+      protocol             = firewall_rule.value
+      source_address_end   = "2a04:3540:53::1"
+      source_address_start = "2a04:3540:53::1"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "UpCloud DNS"
+      source_port_end      = "53"
+      source_port_start    = "53"
+      direction            = "in"
+      family               = "IPv6"
+      protocol             = firewall_rule.value
+      source_address_end   = "2a04:3544:53::1"
+      source_address_start = "2a04:3544:53::1"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "NTP Port"
+      source_port_end      = "123"
+      source_port_start    = "123"
+      direction            = "in"
+      family               = "IPv4"
+      protocol             = firewall_rule.value
+      source_address_end   = "255.255.255.255"
+      source_address_start = "0.0.0.0"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["udp"] : []
+
+    content {
+      action            = "accept"
+      comment           = "NTP Port"
+      source_port_end   = "123"
+      source_port_start = "123"
+      direction         = "in"
+      family            = "IPv6"
+      protocol          = firewall_rule.value
+    }
+  }
+
+  firewall_rule {
+    action    = var.firewall_default_deny_in ? "drop" : "accept"
+    direction = "in"
+  }
+
+  firewall_rule {
+    action    = var.firewall_default_deny_out ? "drop" : "accept"
+    direction = "out"
+  }
+}
+
+resource "upcloud_firewall_rules" "k8s" {
+  for_each  = upcloud_server.worker
+  server_id = each.value.id
+
+  dynamic "firewall_rule" {
+    for_each = var.k8s_allowed_remote_ips
+
+    content {
+      action                 = "accept"
+      comment                = "Allow SSH from this network"
+      destination_port_end   = "22"
+      destination_port_start = "22"
+      direction              = "in"
+      family                 = "IPv4"
+      protocol               = "tcp"
+      source_address_end     = firewall_rule.value.end_address
+      source_address_start   = firewall_rule.value.start_address
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = length(var.k8s_allowed_remote_ips) > 0 ? [1] : []
+
+    content {
+      action                 = "drop"
+      comment                = "Deny SSH from other networks"
+      destination_port_end   = "22"
+      destination_port_start = "22"
+      direction              = "in"
+      family                 = "IPv4"
+      protocol               = "tcp"
+      source_address_end     = "255.255.255.255"
+      source_address_start   = "0.0.0.0"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.worker_allowed_ports
+
+    content {
+      action                 = "accept"
+      comment                = "Allow access on this port"
+      destination_port_end   = firewall_rule.value.port_range_max
+      destination_port_start = firewall_rule.value.port_range_min
+      direction              = "in"
+      family                 = "IPv4"
+      protocol               = firewall_rule.value.protocol
+      source_address_end     = firewall_rule.value.end_address
+      source_address_start   = firewall_rule.value.start_address
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "UpCloud DNS"
+      source_port_end      = "53"
+      source_port_start    = "53"
+      direction            = "in"
+      family               = "IPv4"
+      protocol             = firewall_rule.value
+      source_address_end   = "94.237.40.9"
+      source_address_start = "94.237.40.9"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "UpCloud DNS"
+      source_port_end      = "53"
+      source_port_start    = "53"
+      direction            = "in"
+      family               = "IPv4"
+      protocol             = firewall_rule.value
+      source_address_end   = "94.237.127.9"
+      source_address_start = "94.237.127.9"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "UpCloud DNS"
+      source_port_end      = "53"
+      source_port_start    = "53"
+      direction            = "in"
+      family               = "IPv6"
+      protocol             = firewall_rule.value
+      source_address_end   = "2a04:3540:53::1"
+      source_address_start = "2a04:3540:53::1"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "UpCloud DNS"
+      source_port_end      = "53"
+      source_port_start    = "53"
+      direction            = "in"
+      family               = "IPv6"
+      protocol             = firewall_rule.value
+      source_address_end   = "2a04:3544:53::1"
+      source_address_start = "2a04:3544:53::1"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["udp"] : []
+
+    content {
+      action               = "accept"
+      comment              = "NTP Port"
+      source_port_end      = "123"
+      source_port_start    = "123"
+      direction            = "in"
+      family               = "IPv4"
+      protocol             = firewall_rule.value
+      source_address_end   = "255.255.255.255"
+      source_address_start = "0.0.0.0"
+    }
+  }
+
+  dynamic "firewall_rule" {
+    for_each = var.firewall_default_deny_in ? ["udp"] : []
+
+    content {
+      action            = "accept"
+      comment           = "NTP Port"
+      source_port_end   = "123"
+      source_port_start = "123"
+      direction         = "in"
+      family            = "IPv6"
+      protocol          = firewall_rule.value
+    }
+  }
+
+  firewall_rule {
+    action    = var.firewall_default_deny_in ? "drop" : "accept"
+    direction = "in"
+  }
+
+  firewall_rule {
+    action    = var.firewall_default_deny_out ? "drop" : "accept"
+    direction = "out"
+  }
+}
+
+resource "upcloud_loadbalancer" "lb" {
+  count             = var.loadbalancer_enabled ? 1 : 0
+  configured_status = "started"
+  name              = "${local.resource-prefix}lb"
+  plan              = var.loadbalancer_plan
+  zone              = var.zone
+  network           = upcloud_network.private.id
+}
+
+resource "upcloud_loadbalancer_backend" "lb_backend" {
+  for_each = var.loadbalancer_enabled ? var.loadbalancers : {}
+
+  loadbalancer = upcloud_loadbalancer.lb[0].id
+  name         = "lb-backend-${each.key}"
+}
+
+resource "upcloud_loadbalancer_frontend" "lb_frontend" {
+  for_each = var.loadbalancer_enabled ? var.loadbalancers : {}
+
+  loadbalancer         = upcloud_loadbalancer.lb[0].id
+  name                 = "lb-frontend-${each.key}"
+  mode                 = "tcp"
+  port                 = each.value.port
+  default_backend_name = upcloud_loadbalancer_backend.lb_backend[each.key].name
+}
+
+resource "upcloud_loadbalancer_static_backend_member" "lb_backend_member" {
+  for_each = {
+    for be_server in local.lb_backend_servers :
+    "${be_server.server_name}-lb-backend-${be_server.lb_name}" => be_server
+    if var.loadbalancer_enabled
+  }
+
+  backend      = upcloud_loadbalancer_backend.lb_backend[each.value.lb_name].id
+  name         = "${local.resource-prefix}${each.key}"
+  ip           = merge(upcloud_server.master, upcloud_server.worker)[each.value.server_name].network_interface[1].ip_address
+  port         = each.value.port
+  weight       = 100
+  max_sessions = var.loadbalancer_plan == "production-small" ? 50000 : 1000
+  enabled      = true
+}
+
+resource "upcloud_server_group" "server_groups" {
+  for_each             = var.server_groups
+  title                = each.key
+  anti_affinity_policy = each.value.anti_affinity_policy
+  labels               = {}
+  members              = [for server in each.value.servers : merge(upcloud_server.master, upcloud_server.worker)[server].id]
+}

contrib/terraform/upcloud/modules/kubernetes-cluster/output.tf

@@ -0,0 +1,24 @@
+
+output "master_ip" {
+  value = {
+    for instance in upcloud_server.master :
+    instance.hostname => {
+      "public_ip" : instance.network_interface[0].ip_address
+      "private_ip" : instance.network_interface[1].ip_address
+    }
+  }
+}
+
+output "worker_ip" {
+  value = {
+    for instance in upcloud_server.worker :
+    instance.hostname => {
+      "public_ip" : instance.network_interface[0].ip_address
+      "private_ip" : instance.network_interface[1].ip_address
+    }
+  }
+}
+
+output "loadbalancer_domain" {
+  value = var.loadbalancer_enabled ? upcloud_loadbalancer.lb[0].dns_name : null
+}

contrib/terraform/upcloud/modules/kubernetes-cluster/variables.tf

@@ -0,0 +1,105 @@
+variable "prefix" {
+  type = string
+}
+
+variable "zone" {
+  type = string
+}
+
+variable "template_name" {}
+
+variable "username" {}
+
+variable "private_network_cidr" {}
+
+variable "machines" {
+  description = "Cluster machines"
+  type = map(object({
+    node_type = string
+    plan      = string
+    cpu       = string
+    mem       = string
+    disk_size = number
+    additional_disks = map(object({
+      size = number
+      tier = string
+    }))
+  }))
+}
+
+variable "ssh_public_keys" {
+  type = list(string)
+}
+
+variable "firewall_enabled" {
+  type = bool
+}
+
+variable "master_allowed_remote_ips" {
+  type = list(object({
+    start_address = string
+    end_address   = string
+  }))
+}
+
+variable "k8s_allowed_remote_ips" {
+  type = list(object({
+    start_address = string
+    end_address   = string
+  }))
+}
+
+variable "master_allowed_ports" {
+  type = list(object({
+    protocol       = string
+    port_range_min = number
+    port_range_max = number
+    start_address  = string
+    end_address    = string
+  }))
+}
+
+variable "worker_allowed_ports" {
+  type = list(object({
+    protocol       = string
+    port_range_min = number
+    port_range_max = number
+    start_address  = string
+    end_address    = string
+  }))
+}
+
+variable "firewall_default_deny_in" {
+  type = bool
+}
+
+variable "firewall_default_deny_out" {
+  type = bool
+}
+
+variable "loadbalancer_enabled" {
+  type = bool
+}
+
+variable "loadbalancer_plan" {
+  type = string
+}
+
+variable "loadbalancers" {
+  description = "Load balancers"
+
+  type = map(object({
+    port            = number
+    target_port     = number
+    backend_servers = list(string)
+  }))
+}
+
+variable "server_groups" {
+  description = "Server groups"
+
+  type = map(object({
+    anti_affinity_policy = string
+    servers              = list(string)
+  }))
+}

contrib/terraform/upcloud/modules/kubernetes-cluster/versions.tf

@@ -0,0 +1,10 @@
+
+terraform {
+  required_providers {
+    upcloud = {
+      source  = "UpCloudLtd/upcloud"
+      version = "~>2.12.0"
+    }
+  }
+  required_version = ">= 0.13"
+}

contrib/terraform/upcloud/output.tf

@@ -0,0 +1,12 @@
+
+output "master_ip" {
+  value = module.kubernetes.master_ip
+}
+
+output "worker_ip" {
+  value = module.kubernetes.worker_ip
+}
+
+output "loadbalancer_domain" {
+  value = module.kubernetes.loadbalancer_domain
+}

contrib/terraform/upcloud/sample-inventory/cluster.tfvars

@@ -0,0 +1,149 @@
+# See: https://developers.upcloud.com/1.3/5-zones/
+zone     = "fi-hel1"
+username = "ubuntu"
+
+# Prefix to use for all resources to separate them from other resources
+prefix = "kubespray"
+
+inventory_file = "inventory.ini"
+
+#  Set the operating system using UUID or exact name
+template_name = "Ubuntu Server 20.04 LTS (Focal Fossa)"
+
+ssh_public_keys = [
+  # Put your public SSH key here
+  "ssh-rsa I-did-not-read-the-docs",
+  "ssh-rsa I-did-not-read-the-docs 2",
+]
+
+# check list of available plan https://developers.upcloud.com/1.3/7-plans/
+machines = {
+  "control-plane-0" : {
+    "node_type" : "master",
+    # plan to use instead of custom cpu/mem
+    "plan" : null,
+    #number of cpu cores
+    "cpu" : "2",
+    #memory size in MB
+    "mem" : "4096"
+    # The size of the storage in GB
+    "disk_size" : 250
+    "additional_disks" : {}
+  },
+  "worker-0" : {
+    "node_type" : "worker",
+    # plan to use instead of custom cpu/mem
+    "plan" : null,
+    #number of cpu cores
+    "cpu" : "2",
+    #memory size in MB
+    "mem" : "4096"
+    # The size of the storage in GB
+    "disk_size" : 250
+    "additional_disks" : {
+      # "some-disk-name-1": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # },
+      # "some-disk-name-2": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # }
+    }
+  },
+  "worker-1" : {
+    "node_type" : "worker",
+    # plan to use instead of custom cpu/mem
+    "plan" : null,
+    #number of cpu cores
+    "cpu" : "2",
+    #memory size in MB
+    "mem" : "4096"
+    # The size of the storage in GB
+    "disk_size" : 250
+    "additional_disks" : {
+      # "some-disk-name-1": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # },
+      # "some-disk-name-2": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # }
+    }
+  },
+  "worker-2" : {
+    "node_type" : "worker",
+    # plan to use instead of custom cpu/mem
+    "plan" : null,
+    #number of cpu cores
+    "cpu" : "2",
+    #memory size in MB
+    "mem" : "4096"
+    # The size of the storage in GB
+    "disk_size" : 250
+    "additional_disks" : {
+      # "some-disk-name-1": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # },
+      # "some-disk-name-2": {
+      #   "size": 100,
+      #   "tier": "maxiops",
+      # }
+    }
+  }
+}
+
+firewall_enabled          = false
+firewall_default_deny_in  = false
+firewall_default_deny_out = false
+
+
+master_allowed_remote_ips = [
+  {
+    "start_address" : "0.0.0.0"
+    "end_address" : "255.255.255.255"
+  }
+]
+
+k8s_allowed_remote_ips = [
+  {
+    "start_address" : "0.0.0.0"
+    "end_address" : "255.255.255.255"
+  }
+]
+
+master_allowed_ports = []
+worker_allowed_ports = []
+
+loadbalancer_enabled = false
+loadbalancer_plan    = "development"
+loadbalancers = {
+  # "http" : {
+  #   "port" : 80,
+  #   "target_port" : 80,
+  #   "backend_servers" : [
+  #     "worker-0",
+  #     "worker-1",
+  #     "worker-2"
+  #   ]
+  # }
+}
+
+server_groups = {
+  # "control-plane" = {
+  #   servers = [
+  #     "control-plane-0"
+  #   ]
+  #   anti_affinity_policy = "strict"
+  # },
+  # "workers" = {
+  #   servers = [
+  #     "worker-0",
+  #     "worker-1",
+  #     "worker-2"
+  #   ]
+  #   anti_affinity_policy = "yes"
+  # }
+}

contrib/terraform/upcloud/sample-inventory/group_vars

@@ -0,0 +1 @@
+../../../../inventory/sample/group_vars/

contrib/terraform/upcloud/templates/inventory.tpl

@@ -0,0 +1,17 @@
+
+[all]
+${connection_strings_master}
+${connection_strings_worker}
+
+[kube_control_plane]
+${list_master}
+
+[etcd]
+${list_master}
+
+[kube_node]
+${list_worker}
+
+[k8s_cluster:children]
+kube_control_plane
+kube_node

contrib/terraform/upcloud/variables.tf

@@ -0,0 +1,144 @@
+variable "prefix" {
+  type    = string
+  default = "kubespray"
+
+  description = "Prefix that is used to distinguish these resources from others"
+}
+
+variable "zone" {
+  description = "The zone where to run the cluster"
+}
+
+variable "template_name" {
+  description = "Block describing the preconfigured operating system"
+}
+
+variable "username" {
+  description = "The username to use for the nodes"
+  default     = "ubuntu"
+}
+
+variable "private_network_cidr" {
+  description = "CIDR to use for the private network"
+  default     = "172.16.0.0/24"
+}
+
+variable "machines" {
+  description = "Cluster machines"
+
+  type = map(object({
+    node_type = string
+    plan      = string
+    cpu       = string
+    mem       = string
+    disk_size = number
+    additional_disks = map(object({
+      size = number
+      tier = string
+    }))
+  }))
+}
+
+variable "ssh_public_keys" {
+  description = "List of public SSH keys which are injected into the VMs."
+  type        = list(string)
+}
+
+variable "inventory_file" {
+  description = "Where to store the generated inventory file"
+}
+
+variable "UPCLOUD_USERNAME" {
+  description = "UpCloud username with API access"
+}
+
+variable "UPCLOUD_PASSWORD" {
+  description = "Password for UpCloud API user"
+}
+
+variable "firewall_enabled" {
+  description = "Enable firewall rules"
+  default     = false
+}
+
+variable "master_allowed_remote_ips" {
+  description = "List of IP start/end addresses allowed to access API of masters"
+  type = list(object({
+    start_address = string
+    end_address   = string
+  }))
+  default = []
+}
+
+variable "k8s_allowed_remote_ips" {
+  description = "List of IP start/end addresses allowed to SSH to hosts"
+  type = list(object({
+    start_address = string
+    end_address   = string
+  }))
+  default = []
+}
+
+variable "master_allowed_ports" {
+  description = "List of ports to allow on masters"
+  type = list(object({
+    protocol       = string
+    port_range_min = number
+    port_range_max = number
+    start_address  = string
+    end_address    = string
+  }))
+}
+
+variable "worker_allowed_ports" {
+  description = "List of ports to allow on workers"
+  type = list(object({
+    protocol       = string
+    port_range_min = number
+    port_range_max = number
+    start_address  = string
+    end_address    = string
+  }))
+}
+
+variable "firewall_default_deny_in" {
+  description = "Add firewall policies that deny all inbound traffic by default"
+  default     = false
+}
+
+variable "firewall_default_deny_out" {
+  description = "Add firewall policies that deny all outbound traffic by default"
+  default     = false
+}
+
+variable "loadbalancer_enabled" {
+  description = "Enable load balancer"
+  default     = false
+}
+
+variable "loadbalancer_plan" {
+  description = "Load balancer plan (development/production-small)"
+  default     = "development"
+}
+
+variable "loadbalancers" {
+  description = "Load balancers"
+
+  type = map(object({
+    port            = number
+    target_port     = number
+    backend_servers = list(string)
+  }))
+  default = {}
+}
+
+variable "server_groups" {
+  description = "Server groups"
+
+  type = map(object({
+    anti_affinity_policy = string
+    servers              = list(string)
+  }))
+
+  default = {}
+}

contrib/terraform/upcloud/versions.tf

@@ -0,0 +1,10 @@
+
+terraform {
+  required_providers {
+    upcloud = {
+      source  = "UpCloudLtd/upcloud"
+      version = "~>2.12.0"
+    }
+  }
+  required_version = ">= 0.13"
+}