k8s1.28 基于docker容器安装

2023-06-02

字数统计: 2k字 | 阅读时长≈ 10分

k8s 1.28版本集群部署环境准备

1.1.1主机操作系统说明

序号	操作系统及版本	备注
1	Centos/7.9

1.1.2 主机硬件配置说明

需求	CPU	内存	硬盘	角色	主机名
值	4C	8G	100GB	master	k8s-master
值	4C	8G	100GB	worker	k8s-worker01
值	4C	8G	100GB	worker	k8s-worker02

1.1.3 虚拟机网络设置

# vim /etc/sysconfig/network-scripts/ifcfg-ens33 （虚拟机设置NAT模式）
# cat /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="c8838cc3-83d3-45fb-8ddd-863e5f6c7c2c"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="192.168.254.130"
PREFIX="24"
GATEWAY="192.168.254.2"
DNS1="119.29.29.29"

systemctl restart network

1.1.4 主机配置

master节点
hostnamectl set-hostname k8s-master
worker01节点
hostnamectl set-hostname k8s-worker01
worker02节点
hostnamectl set-hostname k8s-worker02

1.15主机名与IP地址解析

#cat /etc/hosts
192.168.254.130  k8s-master
192.168.254.131  k8s-worker01
192.168.254.132  k8s-worker02

1.16 防火墙配置

关闭防火墙
# systemctl stop firewalld
设置防火墙为开机禁用状态
# systemctl disable firewalld
# 查看
firewall-cmd --state

1.1.7 关闭SELINUX

sed

1.18 时间同步配置

yum install -y ntp
#
crontab -l
0 */1 * * * /usr/sbin/ntpdate timel.aliyun.com

1.19 升级内核

查看内核uname -r

#方法一安装elrepo yum源仓库（网络可能无法连通）

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
#安装kernel-ml 版本，ml为长期稳定版，lt为长期维护版本
yum --enablerepo=elrepo-kernel install kernel-ml

方法二用ip直接下载4.19

cd k8s-toolawget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm

## 从master节点同步到其它节点
for i in k8s-master  k8s-worker01 k8s-worker02;do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/k8s-tool ; done

###所有节点安装内核：
cd /k8s-tool && yum localinstall -y kernel-ml*
### 所有节点更改内核启动顺序：

grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

###检查默认内核是不是4.19
grubby --default-kernel


## vim /etc/modules-load.d/ipvs.conf 
# 加入以下内容
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip

##然后执行即可（报错不用管)
systemctl enable --now systemd-modules-load.service

##所有节点开启一些k8s集群中必须的内核参数，所有节点配置k8s内核：内核转发以及网络

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
net.ipv4.conf.all.route_localnet = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
vm.swappiness=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
## 让内核生效 
sysctl --system

##所有节点安装ipvsadm：yum install ipvsadm ipset sysstat conntrack libseccomp -y#所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可：

cat /etc/sysconfig/modules/ipvs.modules << EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

1
2
3

# 授权运行检查是否加载
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep --color=auto -e ip_vs -e nf_conntrack

1.20关闭SWAP分区

修改后要重启系统

# 永久关闭swap分区
# cat /etc/fstab
# /dev/mapper/centos-swap swap                    swap    defaults        0 0
注释上一行

1.21docker准备

使用阿里源镜像站

1	wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

1.211安装docker

1	yum -y install docker-ce

1.212 启动docker服务

1
2
3

systemctl enable --now docker

1.213 修改cgroup方式

# /etc/docker/daemon.json 添加以下内容
#cat /etc/docker/daemon.json
{
    "exec-opts":["native.cgroupdriver=systemd"]
}
systemctl restart docker

1.23cri-dockerd 安装

1.231goang 环境安装

#获取golang安装包
wget https://go.dev/dl/go1.21.9.linux-amd64.tar.gz
$ rm -rf /usr/local/go && tar -C /usr/local -xzf go1.21.9.linux-amd64.tar.gz
# 环境变量cat /etc/profile
$ export GROOT=/usr/local/go
$ export GOPATH= $HOME/go
$ export PATH=$PATH:$GROOT/bin:$GOPATH/bin
$ source /etc/profile

mkdir -p go/bin ~ go/pkg ~ go/src

$ go env -w GO111MODULE=on
$ go env -w GOPROXY=

1.232 cri-dockerd 安装

https://github.com/Mirantis/cri-dockerd

1. 克隆cri-dockerd
git clone https://github.com/Mirantis/cri-dockerd.git
2. cd cri-dockerd
3. 安装gcc （或者直接go build -o ../bin）本例按最新官方make方式
yum install gcc
  make cri-dockerd
3.     # Run these commands as root

cd cri-dockerd

install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
install packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable --now cri-docker.socket

2.1 kubernetes 1.29.0 集群部署

	kubeadm	kubelet	kubectl
版本	1.28.2	1.28.2	1.28.2
安装位置	集群所有主机	集群所有主机	集群所有主机
作用	初始化集群	用于接受api-server指令，对pod生命周期进行管理	集群应用命令管理工具

2.1.1 阿里云YUM源

# 
cat <<EOF > /etc/yum.repos.d/k8s.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2.12 安装所有节点主机

yum -y install kubeadm kubelet kubectl

2.13 配置kubelet

为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容

#vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
# 设置kubelet 开启自启动
systemctl enable kubelet

2.14 镜像准备

# kubeadm config images list --kubernetes-version=1.28.2
registry.k8s.io/kube-apiserver:v1.28.2
registry.k8s.io/kube-controller-manager:v1.28.2
registry.k8s.io/kube-scheduler:v1.28.2
registry.k8s.io/kube-proxy:v1.28.2
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1

方法1自己先下载下来

# cat image_download.sh
#!/bin/bash
images_list='
registry.k8s.io/kube-apiserver:v1.28.2
registry.k8s.io/kube-controller-manager:v1.28.2
registry.k8s.io/kube-scheduler:v1.28.2
registry.k8s.io/kube-proxy:v1.28.2
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1'

for i in $images_list
do
        docker pull $i
done

docker save -o k8s-1-28-2.tar $images_list

方法二用阿里镜像

#!/bin/bash
repo=registry.aliyuncs.com/google_containers

for name in `kubeadm config images list --kubernetes-version v1.28.2`; do

    src_name=${name#registry.k8s.io/}
    src_name=${src_name#coredns/}

    docker pull $repo/$src_name

    docker tag $repo/$src_name $name
    docker rmi $repo/$src_name
done

2.15 集群初始化

# master节点
kubeadm init \
    --pod-network-cidr=10.10.0.0/16 \
    --apiserver-advertise-address=192.168.254.130 \
    --kubernetes-version=v1.28.2\
    --cri-socket unix:///var/run/cri-dockerd.sock
    
# 如果清楚 kubeadm reset --cri-socket unix:///var/run/cri-dockerd.sock

返回结果第三步 join 更改cri-dockerd

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.254.130:6443 --token 8ox269.62z253zoolshcb33 \
        --discovery-token-ca-cert-hash sha256:9ad78e38c954e950652ef3a4c301d2899d8872a0909018e733396dffcbfbd344

2.16 Calico网络插件安装：

#应用资源清单文件
wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml
kubectl apply -f tigera-operator.yaml
# 通过自定义资源方式安装
wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml


# 修改文件13行，修改为 kubeadm init ----pod-network-cidr 对应的IP地址段

  ipPools:
 12     - blockSize: 26
 13       cidr: 10.10.0.0/16
 14       encapsulation: VXLANCrossSubnet
 15       natOutgoing: Enabled
 16       nodeSelector: all()
 17 
 kubectl apply -f custom-resources.yaml
 #watch kubectl get pods -n calico-system
 # 删除master上的 taint
 kubectl taint nodes --all node-role.kubernetes.io/control-plane-
kubectl taint nodes --all node-role.kubernetes.io/master-

2.17安装客户端calicoctl

1
2
3

curl -L https://github.com/projectcalico/calico/releases/download/v3.26.1/calicoctl-linux-amd64 -o calicoctl
mv calicoctl /usr/local/bin
chmod +x /usr/local/bin/calicoctl

2.18 集群工作节点添加（–cri-socket）

1
2

kubeadm join 192.168.254.130:6443 --token 8ox269.62z253zoolshcb33 \
        --discovery-token-ca-cert-hash sha256:9ad78e38c954e950652ef3a4c301d2899d8872a0909018e733396dffcbfbd344 --cri-socket unix:///var/run/cri-dockerd.sock

kubectl get svc -n kube-system

dig -t a www.baidu.com @10.96.0.10

本文作者： 东方觉主
本文链接： http://www.charon193.com/2023/06/02/k8s1.28/
版权声明： 本博客所有文章除特别声明外，均采用 MIT 许可协议。转载请注明出处！