K8S-Help

K8S开启IPVS

[root@master1 ~]# cat /etc/modules-load.d/k8s.conf
#!/bin/bash
modprobe br_netfilter
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

设置SC为默认存储

kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

K8S二进制安装遇到的网络问题

在签发apiserver证书时需要加上K8S集群SVC地址

K8S使用NFS作为SC时遇到的错误

必须要在所有节点安装nfs-client工具(nfs-utils)
不推荐集群使用NFS当作存储,测试即可

K8S安装度量组件(环境是K8S二进制集群)

报错

metrics x509: cannot validate certificate for 192.168.64.103 because it doesn’t cont

解决方法

在metrics-server的yaml添加如下参数

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      hostNetwork: true
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname,InternalDNS
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls              # 添加这段

报错

[root@master ~]# kubectl top node
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
同时apiserver报错
Oct 26 17:21:01 master kube-apiserver[18236]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Oct 26 17:21:01 master kube-apiserver[18236]: I1026 17:21:01.182587 18236 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Oct 26 17:21:05 master kube-apiserver[18236]: E1026 17:21:05.182475 18236 available_controller.go:524] v1beta1.metrics.k8s.io failed with: failing or missing response from https://172.12.118.65:443/apis/metrics.k8s.io/v1beta1: Get …

解决方法

[root@master ~]# cat /lib/systemd/system/apiserver.service
[Unit]
Description=apiServer
[Service]
Type=simple
ExecStart=kube-apiserver \
  --advertise-address=192.168.64.101 \
  --allow-privileged=true \
  --authorization-mode=Node,RBAC \
  --client-ca-file=/etc/kubernetes/ca.crt \
  --enable-admission-plugins=NodeRestriction \
  --enable-bootstrap-token-auth=true \
  --etcd-cafile=/etc/etcd/etcd-ca.crt \
  --etcd-certfile=/etc/etcd/apiserver-etcd-client.crt \
  --etcd-keyfile=/etc/etcd/apiserver-etcd-client.key \
  --etcd-servers=https://master:2379 \
  --kubelet-client-certificate=/etc/kubernetes/apiserver-kubelet-client.crt \
  --kubelet-client-key=/etc/kubernetes/apiserver-kubelet-client.key \
  --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
  --secure-port=6443 \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --service-account-key-file=/etc/kubernetes/sa.pub \
  --service-account-signing-key-file=/etc/kubernetes/sa.key \
  --service-cluster-ip-range=172.12.0.0/16 \
  --tls-cert-file=/etc/kubernetes/apiserver.crt \
  --tls-private-key-file=/etc/kubernetes/apiserver.key \
  --proxy-client-cert-file=/etc/kubernetes/front-proxy-client.crt \
  --proxy-client-key-file=/etc/kubernetes/front-proxy-client.key \
  --requestheader-allowed-names=front-proxy-client \
  --requestheader-client-ca-file=/etc/kubernetes/front-proxy-ca.crt \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --token-auth-file=/etc/kubernetes/token.csv \
  --enable-bootstrap-token-auth=true \
  --service-node-port-range=80-60000 \
  --enable-aggregator-routing=true \   # 添加此选项
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota
[Install]
WantedBy=multi-user.target

效果

[root@xiaowangc ~]# kubectl top node
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node01   72m          1%     1007Mi          13%
node02   82m          2%     1112Mi          14%
node03   73m          1%     1052Mi          13%
[root@xiaowangc ~]# kubectl top pod
NAME                                      CPU(cores)   MEMORY(bytes)
nfs-client-provisioner-5684cb8fbd-cl9zp   2m           15Mi
nfs-client-provisioner-5684cb8fbd-t72xg   1m           13Mi
[root@xiaowangc ~]#

重启控制器

kubectl rollout restart deploy/prom-deployment

设置K8S ROLES属性

[root@master1 ~]# kubectl get node
NAME                      STATUS   ROLES           AGE    VERSION
master1.xiaowangc.local   Ready    control-plane   10h    v1.25.4
master2.xiaowangc.local   Ready    control-plane   9h     v1.25.4
master3.xiaowangc.local   Ready    control-plane   9h     v1.25.4
node1.xiaowangc.local     Ready    nodes           154m   v1.25.4
node2.xiaowangc.local     Ready    nodes           146m   v1.25.4
node3.xiaowangc.local     Ready    nodes           138m   v1.25.4
node4.xiaowangc.local     Ready    nodes           123m   v1.25.4
[root@master1 ~]# kubectl get node --show-labels
NAME                      STATUS   ROLES           AGE    VERSION   LABELS
master1.xiaowangc.local   Ready    control-plane   10h    v1.25.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master1.xiaowangc.local,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
master2.xiaowangc.local   Ready    control-plane   9h     v1.25.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master2.xiaowangc.local,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
master3.xiaowangc.local   Ready    control-plane   9h     v1.25.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master3.xiaowangc.local,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
node1.xiaowangc.local     Ready    nodes           155m   v1.25.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node1.xiaowangc.local,kubernetes.io/os=linux,node-role.kubernetes.io/nodes=
node2.xiaowangc.local     Ready    nodes           147m   v1.25.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node2.xiaowangc.local,kubernetes.io/os=linux,node-role.kubernetes.io/nodes=
node3.xiaowangc.local     Ready    nodes           139m   v1.25.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node3.xiaowangc.local,kubernetes.io/os=linux,node-role.kubernetes.io/nodes=
node4.xiaowangc.local     Ready    nodes           125m   v1.25.4   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node4.xiaowangc.local,kubernetes.io/os=linux,node-role.kubernetes.io/nodes=

[root@master1 ~]# kubectl label nodes 节点名字 node-role.kubernetes.io/ROLES属性名称=或-

Kubernetes的Service无法ping通

初步分析，在使用iptables转发时也是无法使用ping命令,因为ping基于ICMP ,ICMP不能在TCP/UDP上运行，因此没有TCP/UDP端口的概念。因此，无法在配置为侦听和trap的服务的端口上使用ping
iptable默认策略为拒绝任何icmp端口，除非你手动打开让其支持icmp协议才可以ping通。

集群如果使用的是iptables转发，那么可以尝试设置允许ICMP协议
集群如果使用的是IPVS转发，可能被iptables的规则所作用，可以尝试使用以下命令对其进行清空

iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat

同时也可以考虑让iptables转发IPVS所有流量

iptables -I INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD -j ACCEPT

集群是使用IPVS转发,在ping Service名称的时候时候无法解析,查看相关的Pod日志并无报错,同时宿主机报错IPVS：rr：TCP x.x.x.x:x - no destination available
使用iptables清空防火墙规则即可

iptables -F

kubernetes查看代理模式

curl 127.0.0.1:10249/proxyMode
 
[root@k8s-master01 ~]# curl 127.0.0.1:10249/proxyMode
iptables