sa_8001/offline_kubespray
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

kubespray 2.24 추가

Browse Source
This commit is contained in:
변정훈
2024-02-16 17:08:09 +09:00
parent 1fa9b0df4b
commit f69d904725
1423 changed files with 89069 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

128
contrib/terraform/vsphere/README.md Normal file
View File

@@ -0,0 +1,128 @@
# Kubernetes on vSphere with Terraform
Provision a Kubernetes cluster on [vSphere](https://www.vmware.com/products/vsphere.html) using Terraform and Kubespray.
## Overview
The setup looks like following.
```text
Kubernetes cluster
+-----------------------+
| +--------------+ |
| | +--------------+ |
| | | | |
| | | Master/etcd | |
| | | node(s) | |
| +-+ | |
| +--------------+ |
| ^ |
| | |
| v |
| +--------------+ |
| | +--------------+ |
| | | | |
| | | Worker | |
| | | node(s) | |
| +-+ | |
| +--------------+ |
+-----------------------+
```
## Warning
This setup assumes that the DHCP is disabled in the vSphere cluster and IP addresses have to be provided in the configuration file.
## Requirements
* Terraform 0.13.0 or newer (0.12 also works if you modify the provider block to include version and remove all `versions.tf` files)
## Quickstart
NOTE: *Assumes you are at the root of the kubespray repo*
Copy the sample inventory for your cluster and copy the default terraform variables.
```bash
CLUSTER=my-vsphere-cluster
cp -r inventory/sample inventory/$CLUSTER
cp contrib/terraform/vsphere/default.tfvars inventory/$CLUSTER/
cd inventory/$CLUSTER
```
Edit `default.tfvars` to match your setup. You MUST set values specific for you network and vSphere cluster.
```bash
# Ensure $EDITOR points to your favorite editor, e.g., vim, emacs, VS Code, etc.
$EDITOR default.tfvars
```
For authentication in your vSphere cluster you can use the environment variables.
```bash
export TF_VAR_vsphere_user=username
export TF_VAR_vsphere_password=password
```
Run Terraform to create the infrastructure.
```bash
terraform init ../../contrib/terraform/vsphere
terraform apply \
-var-file default.tfvars \
-state=tfstate-$CLUSTER.tfstate \
../../contrib/terraform/vsphere
```
You should now have a inventory file named `inventory.ini` that you can use with kubespray.
You can now copy your inventory file and use it with kubespray to set up a cluster.
You can type `terraform output` to find out the IP addresses of the nodes.
It is a good idea to check that you have basic SSH connectivity to the nodes. You can do that by:
```bash
ansible -i inventory.ini -m ping all
```
Example to use this with the default sample inventory:
```bash
ansible-playbook -i inventory.ini ../../cluster.yml -b -v
```
## Variables
### Required
* `machines`: Machines to provision. Key of this object will be used as the name of the machine
* `node_type`: The role of this node *(master|worker)*
* `ip`: The IP address of the machine
* `netmask`: The netmask to use (to be used on the right hand side in CIDR notation, e.g., `24`)
* `network`: The name of the network to attach the machines to
* `gateway`: The IP address of the network gateway
* `vsphere_datacenter`: The identifier of vSphere data center
* `vsphere_compute_cluster`: The identifier of vSphere compute cluster
* `vsphere_datastore`: The identifier of vSphere data store
* `vsphere_server`: This is the vCenter server name or address for vSphere API operations.
* `ssh_public_keys`: List of public SSH keys to install on all machines
* `template_name`: The name of a base image (the OVF template be defined in vSphere beforehand)
### Optional
* `folder`: Name of the folder to put all machines in (default: `""`)
* `prefix`: Prefix to use for all resources, required to be unique for all clusters in the same project (default: `"k8s"`)
* `inventory_file`: Name of the generated inventory file for Kubespray to use in the Ansible step (default: `inventory.ini`)
* `dns_primary`: The IP address of primary DNS server (default: `8.8.4.4`)
* `dns_secondary`: The IP address of secondary DNS server (default: `8.8.8.8`)
* `firmware`: Firmware to use (default: `bios`)
* `hardware_version`: The version of the hardware (default: `15`)
* `master_cores`: The number of CPU cores for the master nodes (default: 4)
* `master_memory`: The amount of RAM for the master nodes in MB (default: 4096)
* `master_disk_size`: The amount of disk space for the master nodes in GB (default: 20)
* `worker_cores`: The number of CPU cores for the worker nodes (default: 16)
* `worker_memory`: The amount of RAM for the worker nodes in MB (default: 8192)
* `worker_disk_size`: The amount of disk space for the worker nodes in GB (default: 100)
* `vapp`: Boolean to set the template type to vapp. (Default: false)
* `interface_name`: Name of the interface to configure. (Default: ens192)
An example variables file can be found `default.tfvars`

38
contrib/terraform/vsphere/default.tfvars Normal file
View File

@@ -0,0 +1,38 @@
prefix = "k8s"
inventory_file = "inventory.ini"
network = "VM Network"
machines = {
"master-0" : {
"node_type" : "master",
"ip" : "i-did-not-read-the-docs", # e.g. 192.168.0.10
"netmask" : "24"
},
"worker-0" : {
"node_type" : "worker",
"ip" : "i-did-not-read-the-docs", # e.g. 192.168.0.20
"netmask" : "24"
},
"worker-1" : {
"node_type" : "worker",
"ip" : "i-did-not-read-the-docs", # e.g. 192.168.0.21
"netmask" : "24"
}
}
gateway = "i-did-not-read-the-docs" # e.g. 192.168.0.1
ssh_public_keys = [
# Put your public SSH key here
"ssh-rsa I-did-not-read-the-docs",
"ssh-rsa I-did-not-read-the-docs 2",
]
vsphere_datacenter = "i-did-not-read-the-docs"
vsphere_compute_cluster = "i-did-not-read-the-docs" # e.g. Cluster
vsphere_datastore = "i-did-not-read-the-docs" # e.g. ssd-000000
vsphere_server = "i-did-not-read-the-docs" # e.g. vsphere.server.com
template_name = "i-did-not-read-the-docs" # e.g. ubuntu-bionic-18.04-cloudimg

100
contrib/terraform/vsphere/main.tf Normal file
View File

@@ -0,0 +1,100 @@
provider "vsphere" {
# Username and password set through env vars VSPHERE_USER and VSPHERE_PASSWORD
user = var.vsphere_user
password = var.vsphere_password
vsphere_server = var.vsphere_server
# If you have a self-signed cert
allow_unverified_ssl = true
}
data "vsphere_datacenter" "dc" {
name = var.vsphere_datacenter
}
data "vsphere_datastore" "datastore" {
name = var.vsphere_datastore
datacenter_id = data.vsphere_datacenter.dc.id
}
data "vsphere_network" "network" {
name = var.network
datacenter_id = data.vsphere_datacenter.dc.id
}
data "vsphere_virtual_machine" "template" {
name = var.template_name
datacenter_id = data.vsphere_datacenter.dc.id
}
data "vsphere_compute_cluster" "compute_cluster" {
name = var.vsphere_compute_cluster
datacenter_id = data.vsphere_datacenter.dc.id
}
resource "vsphere_resource_pool" "pool" {
name = "${var.prefix}-cluster-pool"
parent_resource_pool_id = data.vsphere_compute_cluster.compute_cluster.resource_pool_id
}
module "kubernetes" {
source = "./modules/kubernetes-cluster"
prefix = var.prefix
machines = var.machines
## Master ##
master_cores = var.master_cores
master_memory = var.master_memory
master_disk_size = var.master_disk_size
## Worker ##
worker_cores = var.worker_cores
worker_memory = var.worker_memory
worker_disk_size = var.worker_disk_size
## Global ##
gateway = var.gateway
dns_primary = var.dns_primary
dns_secondary = var.dns_secondary
pool_id = vsphere_resource_pool.pool.id
datastore_id = data.vsphere_datastore.datastore.id
folder = var.folder
guest_id = data.vsphere_virtual_machine.template.guest_id
scsi_type = data.vsphere_virtual_machine.template.scsi_type
network_id = data.vsphere_network.network.id
adapter_type = data.vsphere_virtual_machine.template.network_interface_types[0]
interface_name = var.interface_name
firmware = var.firmware
hardware_version = var.hardware_version
disk_thin_provisioned = data.vsphere_virtual_machine.template.disks.0.thin_provisioned
template_id = data.vsphere_virtual_machine.template.id
vapp = var.vapp
ssh_public_keys = var.ssh_public_keys
}
#
# Generate ansible inventory
#
resource "local_file" "inventory" {
content = templatefile("${path.module}/templates/inventory.tpl", {
connection_strings_master = join("\n", formatlist("%s ansible_user=ubuntu ansible_host=%s etcd_member_name=etcd%d",
keys(module.kubernetes.master_ip),
values(module.kubernetes.master_ip),
range(1, length(module.kubernetes.master_ip) + 1))),
connection_strings_worker = join("\n", formatlist("%s ansible_user=ubuntu ansible_host=%s",
keys(module.kubernetes.worker_ip),
values(module.kubernetes.worker_ip))),
list_master = join("\n", formatlist("%s", keys(module.kubernetes.master_ip))),
list_worker = join("\n", formatlist("%s", keys(module.kubernetes.worker_ip)))
})
filename = var.inventory_file
}

149
contrib/terraform/vsphere/modules/kubernetes-cluster/main.tf Normal file
View File

@@ -0,0 +1,149 @@
resource "vsphere_virtual_machine" "worker" {
for_each = {
for name, machine in var.machines :
name => machine
if machine.node_type == "worker"
}
name = "${var.prefix}-${each.key}"
resource_pool_id = var.pool_id
datastore_id = var.datastore_id
num_cpus = var.worker_cores
memory = var.worker_memory
memory_reservation = var.worker_memory
guest_id = var.guest_id
enable_disk_uuid = "true" # needed for CSI provider
scsi_type = var.scsi_type
folder = var.folder
firmware = var.firmware
hardware_version = var.hardware_version
wait_for_guest_net_routable = false
wait_for_guest_net_timeout = 0
network_interface {
network_id = var.network_id
adapter_type = var.adapter_type
}
disk {
label = "disk0"
size = var.worker_disk_size
thin_provisioned = var.disk_thin_provisioned
}
lifecycle {
ignore_changes = [disk]
}
clone {
template_uuid = var.template_id
}
cdrom {
client_device = true
}
dynamic "vapp" {
for_each = var.vapp ? [1] : []
content {
properties = {
"user-data" = base64encode(templatefile("${path.module}/templates/vapp-cloud-init.tpl", { ssh_public_keys = var.ssh_public_keys }))
}
}
}
extra_config = {
"isolation.tools.copy.disable" = "FALSE"
"isolation.tools.paste.disable" = "FALSE"
"isolation.tools.setGUIOptions.enable" = "TRUE"
"guestinfo.userdata" = base64encode(templatefile("${path.module}/templates/cloud-init.tpl", { ssh_public_keys = var.ssh_public_keys }))
"guestinfo.userdata.encoding" = "base64"
"guestinfo.metadata" = base64encode(templatefile("${path.module}/templates/metadata.tpl", { hostname = "${var.prefix}-${each.key}",
interface_name = var.interface_name
ip = each.value.ip,
netmask = each.value.netmask,
gw = var.gateway,
dns = var.dns_primary,
ssh_public_keys = var.ssh_public_keys }))
"guestinfo.metadata.encoding" = "base64"
}
}
resource "vsphere_virtual_machine" "master" {
for_each = {
for name, machine in var.machines :
name => machine
if machine.node_type == "master"
}
name = "${var.prefix}-${each.key}"
resource_pool_id = var.pool_id
datastore_id = var.datastore_id
num_cpus = var.master_cores
memory = var.master_memory
memory_reservation = var.master_memory
guest_id = var.guest_id
enable_disk_uuid = "true" # needed for CSI provider
scsi_type = var.scsi_type
folder = var.folder
firmware = var.firmware
hardware_version = var.hardware_version
wait_for_guest_net_routable = false
wait_for_guest_net_timeout = 0
network_interface {
network_id = var.network_id
adapter_type = var.adapter_type
}
disk {
label = "disk0"
size = var.master_disk_size
thin_provisioned = var.disk_thin_provisioned
}
lifecycle {
ignore_changes = [disk]
}
clone {
template_uuid = var.template_id
}
cdrom {
client_device = true
}
dynamic "vapp" {
for_each = var.vapp ? [1] : []
content {
properties = {
"user-data" = base64encode(templatefile("${path.module}/templates/vapp-cloud-init.tpl", { ssh_public_keys = var.ssh_public_keys }))
}
}
}
extra_config = {
"isolation.tools.copy.disable" = "FALSE"
"isolation.tools.paste.disable" = "FALSE"
"isolation.tools.setGUIOptions.enable" = "TRUE"
"guestinfo.userdata" = base64encode(templatefile("${path.module}/templates/cloud-init.tpl", { ssh_public_keys = var.ssh_public_keys }))
"guestinfo.userdata.encoding" = "base64"
"guestinfo.metadata" = base64encode(templatefile("${path.module}/templates/metadata.tpl", { hostname = "${var.prefix}-${each.key}",
interface_name = var.interface_name
ip = each.value.ip,
netmask = each.value.netmask,
gw = var.gateway,
dns = var.dns_primary,
ssh_public_keys = var.ssh_public_keys }))
"guestinfo.metadata.encoding" = "base64"
}
}

15
contrib/terraform/vsphere/modules/kubernetes-cluster/output.tf Normal file
View File

@@ -0,0 +1,15 @@
output "master_ip" {
value = {
for name, machine in var.machines :
"${var.prefix}-${name}" => machine.ip
if machine.node_type == "master"
}
}
output "worker_ip" {
value = {
for name, machine in var.machines :
"${var.prefix}-${name}" => machine.ip
if machine.node_type == "worker"
}
}

6
contrib/terraform/vsphere/modules/kubernetes-cluster/templates/cloud-init.tpl Normal file
View File

@@ -0,0 +1,6 @@
#cloud-config
ssh_authorized_keys:
%{ for ssh_public_key in ssh_public_keys ~}
- ${ssh_public_key}
%{ endfor ~}

14
contrib/terraform/vsphere/modules/kubernetes-cluster/templates/metadata.tpl Normal file
View File

@@ -0,0 +1,14 @@
instance-id: ${hostname}
local-hostname: ${hostname}
network:
version: 2
ethernets:
${interface_name}:
match:
name: ${interface_name}
dhcp4: false
addresses:
- ${ip}/${netmask}
gateway4: ${gw}
nameservers:
addresses: [${dns}]

24
contrib/terraform/vsphere/modules/kubernetes-cluster/templates/vapp-cloud-init.tpl Normal file
View File

@@ -0,0 +1,24 @@
#cloud-config
ssh_authorized_keys:
%{ for ssh_public_key in ssh_public_keys ~}
- ${ssh_public_key}
%{ endfor ~}
write_files:
- path: /etc/netplan/10-user-network.yaml
content: |.
network:
version: 2
ethernets:
${interface_name}:
dhcp4: false #true to use dhcp
addresses:
- ${ip}/${netmask}
gateway4: ${gw} # Set gw here
nameservers:
addresses:
- ${dns} # Set DNS ip address here
runcmd:
- netplan apply

43
contrib/terraform/vsphere/modules/kubernetes-cluster/variables.tf Normal file
View File

@@ -0,0 +1,43 @@
## Global ##
variable "prefix" {}
variable "machines" {
description = "Cluster machines"
type = map(object({
node_type = string
ip = string
netmask = string
}))
}
variable "gateway" {}
variable "dns_primary" {}
variable "dns_secondary" {}
variable "pool_id" {}
variable "datastore_id" {}
variable "guest_id" {}
variable "scsi_type" {}
variable "network_id" {}
variable "interface_name" {}
variable "adapter_type" {}
variable "disk_thin_provisioned" {}
variable "template_id" {}
variable "vapp" {
type = bool
}
variable "firmware" {}
variable "folder" {}
variable "ssh_public_keys" {
type = list(string)
}
variable "hardware_version" {}
## Master ##
variable "master_cores" {}
variable "master_memory" {}
variable "master_disk_size" {}
## Worker ##
variable "worker_cores" {}
variable "worker_memory" {}
variable "worker_disk_size" {}

9
contrib/terraform/vsphere/modules/kubernetes-cluster/versions.tf Normal file
View File

@@ -0,0 +1,9 @@
terraform {
required_providers {
vsphere = {
source = "hashicorp/vsphere"
version = ">= 1.24.3"
}
}
required_version = ">= 0.13"
}

31
contrib/terraform/vsphere/output.tf Normal file
View File

@@ -0,0 +1,31 @@
output "master_ip_addresses" {
value = module.kubernetes.master_ip
}
output "worker_ip_addresses" {
value = module.kubernetes.worker_ip
}
output "vsphere_datacenter" {
value = var.vsphere_datacenter
}
output "vsphere_server" {
value = var.vsphere_server
}
output "vsphere_datastore" {
value = var.vsphere_datastore
}
output "vsphere_network" {
value = var.network
}
output "vsphere_folder" {
value = var.folder
}
output "vsphere_pool" {
value = "${terraform.workspace}-cluster-pool"
}

33
contrib/terraform/vsphere/sample-inventory/cluster.tfvars Normal file
View File

@@ -0,0 +1,33 @@
prefix = "default"
inventory_file = "inventory.ini"
machines = {
"master-0" : {
"node_type" : "master",
"ip" : "i-did-not-read-the-docs" # e.g. 192.168.0.2/24
},
"worker-0" : {
"node_type" : "worker",
"ip" : "i-did-not-read-the-docs" # e.g. 192.168.0.2/24
},
"worker-1" : {
"node_type" : "worker",
"ip" : "i-did-not-read-the-docs" # e.g. 192.168.0.2/24
}
}
gateway = "i-did-not-read-the-docs" # e.g. 192.168.0.2
ssh_public_keys = [
# Put your public SSH key here
"ssh-rsa I-did-not-read-the-docs",
"ssh-rsa I-did-not-read-the-docs 2",
]
vsphere_datacenter = "i-did-not-read-the-docs"
vsphere_compute_cluster = "i-did-not-read-the-docs" # e.g. Cluster
vsphere_datastore = "i-did-not-read-the-docs" # e.g. ssd-000000
vsphere_server = "i-did-not-read-the-docs" # e.g. vsphere.server.com
template_name = "i-did-not-read-the-docs" # e.g. ubuntu-bionic-18.04-cloudimg

1
contrib/terraform/vsphere/sample-inventory/group_vars Symbolic link
View File

@@ -0,0 +1 @@
../../../../inventory/sample/group_vars

17
contrib/terraform/vsphere/templates/inventory.tpl Normal file
View File

@@ -0,0 +1,17 @@
[all]
${connection_strings_master}
${connection_strings_worker}
[kube_control_plane]
${list_master}
[etcd]
${list_master}
[kube_node]
${list_worker}
[k8s_cluster:children]
kube_control_plane
kube_node

101
contrib/terraform/vsphere/variables.tf Normal file
View File

@@ -0,0 +1,101 @@
## Global ##
# Required variables
variable "machines" {
description = "Cluster machines"
type = map(object({
node_type = string
ip = string
netmask = string
}))
}
variable "network" {}
variable "gateway" {}
variable "vsphere_datacenter" {}
variable "vsphere_compute_cluster" {}
variable "vsphere_datastore" {}
variable "vsphere_user" {}
variable "vsphere_password" {
sensitive = true
}
variable "vsphere_server" {}
variable "ssh_public_keys" {
description = "List of public SSH keys which are injected into the VMs."
type = list(string)
}
variable "template_name" {}
# Optional variables (ones where reasonable defaults exist)
variable "vapp" {
default = false
}
variable "interface_name" {
default = "ens192"
}
variable "folder" {
default = ""
}
variable "prefix" {
default = "k8s"
}
variable "inventory_file" {
default = "inventory.ini"
}
variable "dns_primary" {
default = "8.8.4.4"
}
variable "dns_secondary" {
default = "8.8.8.8"
}
variable "firmware" {
default = "bios"
}
variable "hardware_version" {
default = "15"
}
## Master ##
variable "master_cores" {
default = 4
}
variable "master_memory" {
default = 4096
}
variable "master_disk_size" {
default = "20"
}
## Worker ##
variable "worker_cores" {
default = 16
}
variable "worker_memory" {
default = 8192
}
variable "worker_disk_size" {
default = "100"
}

9
contrib/terraform/vsphere/versions.tf Normal file
View File

@@ -0,0 +1,9 @@
terraform {
required_providers {
vsphere = {
source = "hashicorp/vsphere"
version = ">= 1.24.3"
}
}
required_version = ">= 0.13"
}