deploy: same deployment as for csi-driver-host-path
Using a script makes it possible to configure images without manually
editing .yaml files. This is necessary for running in a CI and more
user-friendly.
The RBAC file doesn't need to be copied into the repo when using that
script.
The script itself is a simplified version of the corresponding script
in the hostpath repo:
a72d13d1dd/deploy/util/deploy-hostpath.sh
This commit is contained in:
Patrick Ohly
parent
4fa924a251
commit
25dab4b691
66
README.md
66
README.md
|
|
@ -65,6 +65,7 @@ nfstestvol
|
||||||
$ csc node get-id --endpoint tcp://127.0.0.1:10000
|
$ csc node get-id --endpoint tcp://127.0.0.1:10000
|
||||||
CSINode
|
CSINode
|
||||||
```
|
```
|
||||||
|
|
||||||
## Running Kubernetes End To End tests on an NFS Driver
|
## Running Kubernetes End To End tests on an NFS Driver
|
||||||
|
|
||||||
First, stand up a local cluster `ALLOW_PRIVILEGED=1 hack/local-up-cluster.sh` (from your Kubernetes repo)
|
First, stand up a local cluster `ALLOW_PRIVILEGED=1 hack/local-up-cluster.sh` (from your Kubernetes repo)
|
||||||
|
|
@ -74,18 +75,67 @@ For Fedora/RHEL clusters, the following might be required:
|
||||||
sudo chown -R $USER:$USER /var/lib/kubelet
|
sudo chown -R $USER:$USER /var/lib/kubelet
|
||||||
sudo chcon -R -t svirt_sandbox_file_t /var/lib/kubelet
|
sudo chcon -R -t svirt_sandbox_file_t /var/lib/kubelet
|
||||||
```
|
```
|
||||||
If you are plannig to test using your own private image, you could either install your nfs driver using your own set of YAML files, or edit the existing YAML files to use that private image.
|
|
||||||
|
|
||||||
When using the [existing set of YAML files](https://github.com/kubernetes-csi/csi-driver-nfs/tree/master/deploy/kubernetes), you would edit the [csi-attacher-nfsplugin.yaml](https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/deploy/kubernetes/csi-attacher-nfsplugin.yaml#L46) and [csi-nodeplugin-nfsplugin.yaml](https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/deploy/kubernetes/csi-nodeplugin-nfsplugin.yaml#L45) files to include your private image instead of the default one. After editing these files, skip to step 3 of the following steps.
|
For all of the following commands, set the `KUBECONFIG` env variables as instructed by `local-up-cluster.sh` or as needed for some other cluster.
|
||||||
|
|
||||||
If you already have a driver installed, skip to step 4 of the following steps.
|
`deploy/kubernetes/deploy.sh` will deploy the nfs driver using an
|
||||||
|
image from quay.io which (at the time of writing this) isn't available
|
||||||
|
yet.
|
||||||
|
|
||||||
1) Build the nfs driver by running `make`
|
It is possible to use a locally built image without any registry:
|
||||||
2) Create NFS Driver Image, where the image tag would be whatever that is required by your YAML deployment files `docker build -t quay.io/k8scsi/nfsplugin:v1.0.0 .`
|
``` sh
|
||||||
3) Install the Driver: `kubectl create -f deploy/kubernetes`
|
$ make container
|
||||||
4) Build E2E test binary: `make build-tests`
|
...
|
||||||
5) Run E2E Tests using the following command: `./bin/tests --ginkgo.v --ginkgo.progress --kubeconfig=/var/run/kubernetes/admin.kubeconfig`
|
Successfully tagged nfsplugin:latest
|
||||||
|
|
||||||
|
$ NFSPLUGIN_REGISTRY=none NFSPLUGIN_TAG=latest deploy/kubernetes/deploy.sh
|
||||||
|
applying RBAC rules
|
||||||
|
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-attacher/v1.0.1/deploy/kubernetes/rbac.yaml
|
||||||
|
serviceaccount/csi-attacher created
|
||||||
|
clusterrole.rbac.authorization.k8s.io/external-attacher-runner created
|
||||||
|
clusterrolebinding.rbac.authorization.k8s.io/csi-attacher-role created
|
||||||
|
role.rbac.authorization.k8s.io/external-attacher-cfg created
|
||||||
|
rolebinding.rbac.authorization.k8s.io/csi-attacher-role-cfg created
|
||||||
|
deploying nfs plugin components
|
||||||
|
deploy/kubernetes/csi-attacher-nfsplugin.yaml
|
||||||
|
using image: quay.io/k8scsi/csi-attacher:v1.0.1
|
||||||
|
using image: nfsplugin:latest
|
||||||
|
service/csi-attacher-nfsplugin created
|
||||||
|
statefulset.apps/csi-attacher-nfsplugin created
|
||||||
|
deploy/kubernetes/csi-nodeplugin-nfsplugin.yaml
|
||||||
|
using image: quay.io/k8scsi/csi-node-driver-registrar:v1.0.2
|
||||||
|
using image: nfsplugin:latest
|
||||||
|
daemonset.apps/csi-nodeplugin-nfsplugin created
|
||||||
|
deploy/kubernetes/csi-nodeplugin-rbac.yaml
|
||||||
|
serviceaccount/csi-nodeplugin created
|
||||||
|
clusterrole.rbac.authorization.k8s.io/csi-nodeplugin created
|
||||||
|
clusterrolebinding.rbac.authorization.k8s.io/csi-nodeplugin created
|
||||||
|
10:53:11 waiting for nfs deployment to complete, attempt #0
|
||||||
|
10:53:21 waiting for nfs deployment to complete, attempt #1
|
||||||
|
```
|
||||||
|
|
||||||
|
Other clusters may need a registry to pull from:
|
||||||
|
``` sh
|
||||||
|
$ make push REGISTRY_NAME=my-registry:5000
|
||||||
|
...
|
||||||
|
$ NFSPLUGIN_REGISTRY=my-registry:5000 NFSPLUGIN_TAG=latest deploy/kubernetes/deploy.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
Once you have the driver installed, tests can be run with:
|
||||||
|
``` sh
|
||||||
|
$ make build-tests
|
||||||
|
mkdir -p bin
|
||||||
|
CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=4fa924a251193c9eef937042112462433089d658 -extldflags "-static"' -o ./bin/tests ./cmd/tests
|
||||||
|
$ ./bin/tests --ginkgo.v --ginkgo.progress
|
||||||
|
Jun 7 10:57:39.667: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used.
|
||||||
|
Running Suite: CSI Suite
|
||||||
|
========================
|
||||||
|
Random Seed: 1559897859 - Will randomize all specs
|
||||||
|
Will run 103 of 103 specs
|
||||||
|
...
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
## Community, discussion, contribution, and support
|
## Community, discussion, contribution, and support
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,37 +0,0 @@
|
||||||
# This YAML file contains RBAC API objects that are necessary to run external
|
|
||||||
# CSI attacher for nfs flex adapter
|
|
||||||
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
|
||||||
name: csi-attacher
|
|
||||||
|
|
||||||
---
|
|
||||||
kind: ClusterRole
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
metadata:
|
|
||||||
name: external-attacher-runner
|
|
||||||
rules:
|
|
||||||
- apiGroups: [""]
|
|
||||||
resources: ["persistentvolumes"]
|
|
||||||
verbs: ["get", "list", "watch", "update"]
|
|
||||||
- apiGroups: [""]
|
|
||||||
resources: ["nodes"]
|
|
||||||
verbs: ["get", "list", "watch"]
|
|
||||||
- apiGroups: ["storage.k8s.io"]
|
|
||||||
resources: ["volumeattachments"]
|
|
||||||
verbs: ["get", "list", "watch", "update"]
|
|
||||||
|
|
||||||
---
|
|
||||||
kind: ClusterRoleBinding
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
metadata:
|
|
||||||
name: csi-attacher-role
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: csi-attacher
|
|
||||||
namespace: default
|
|
||||||
roleRef:
|
|
||||||
kind: ClusterRole
|
|
||||||
name: external-attacher-runner
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
|
|
@ -0,0 +1,154 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# This script captures the steps required to successfully
|
||||||
|
# deploy the hostpath plugin driver. This should be considered
|
||||||
|
# authoritative and all updates for this process should be
|
||||||
|
# done here and referenced elsewhere.
|
||||||
|
|
||||||
|
# The script assumes that kubectl is available on the OS path
|
||||||
|
# where it is executed.
|
||||||
|
|
||||||
|
set -e
|
||||||
|
set -o pipefail
|
||||||
|
|
||||||
|
BASE_DIR=$(dirname "$0")
|
||||||
|
|
||||||
|
# If set, the following env variables override image registry and/or tag for each of the images.
|
||||||
|
# They are named after the image name, with hyphen replaced by underscore and in upper case.
|
||||||
|
#
|
||||||
|
# - CSI_ATTACHER_REGISTRY
|
||||||
|
# - CSI_ATTACHER_TAG
|
||||||
|
# - CSI_NODE_DRIVER_REGISTRAR_REGISTRY
|
||||||
|
# - CSI_NODE_DRIVER_REGISTRAR_TAG
|
||||||
|
# - NFSPLUGIN_REGISTRY
|
||||||
|
# - NFSPLUGIN_TAG
|
||||||
|
#
|
||||||
|
# Alternatively, it is possible to override all registries or tags with:
|
||||||
|
# - IMAGE_REGISTRY
|
||||||
|
# - IMAGE_TAG
|
||||||
|
# These are used as fallback when the more specific variables are unset or empty.
|
||||||
|
#
|
||||||
|
# Beware that the .yaml files do not have "imagePullPolicy: Always". That means that
|
||||||
|
# also the "canary" images will only be pulled once. This is good for testing
|
||||||
|
# (starting a pod multiple times will always run with the same canary image), but
|
||||||
|
# implies that refreshing that image has to be done manually.
|
||||||
|
#
|
||||||
|
# As a special case, 'none' as registry removes the registry name.
|
||||||
|
|
||||||
|
# The default is to use the RBAC rules that match the image that is
|
||||||
|
# being used, also in the case that the image gets overridden. This
|
||||||
|
# way if there are breaking changes in the RBAC rules, the deployment
|
||||||
|
# will continue to work.
|
||||||
|
#
|
||||||
|
# However, such breaking changes should be rare and only occur when updating
|
||||||
|
# to a new major version of a sidecar. Nonetheless, to allow testing the scenario
|
||||||
|
# where the image gets overridden but not the RBAC rules, updating the RBAC
|
||||||
|
# rules can be disabled.
|
||||||
|
: ${UPDATE_RBAC_RULES:=true}
|
||||||
|
function rbac_version () {
|
||||||
|
yaml="$1"
|
||||||
|
image="$2"
|
||||||
|
update_rbac="$3"
|
||||||
|
|
||||||
|
# get version from `image: quay.io/k8scsi/csi-attacher:v1.0.1`, ignoring comments
|
||||||
|
version="$(sed -e 's/ *#.*$//' "$yaml" | grep "image:.*$image" | sed -e 's/ *#.*//' -e 's/.*://')"
|
||||||
|
|
||||||
|
if $update_rbac; then
|
||||||
|
# apply overrides
|
||||||
|
varname=$(echo $image | tr - _ | tr a-z A-Z)
|
||||||
|
eval version=\${${varname}_TAG:-\${IMAGE_TAG:-\$version}}
|
||||||
|
fi
|
||||||
|
|
||||||
|
# When using canary images, we have to assume that the
|
||||||
|
# canary images were built from the corresponding branch.
|
||||||
|
case "$version" in canary) version=master;;
|
||||||
|
*-canary) version="$(echo "$version" | sed -e 's/\(.*\)-canary/release-\1/')";;
|
||||||
|
esac
|
||||||
|
|
||||||
|
echo "$version"
|
||||||
|
}
|
||||||
|
|
||||||
|
# In addition, the RBAC rules can be overridden separately.
|
||||||
|
CSI_ATTACHER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/csi-attacher-nfsplugin.yaml" csi-attacher false)/deploy/kubernetes/rbac.yaml"
|
||||||
|
: ${CSI_ATTACHER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/csi-attacher-nfsplugin.yaml" csi-attacher "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}
|
||||||
|
|
||||||
|
INSTALL_CRD=${INSTALL_CRD:-"false"}
|
||||||
|
|
||||||
|
# Some images are not affected by *_REGISTRY/*_TAG and IMAGE_* variables.
|
||||||
|
# The default is to update unless explicitly excluded.
|
||||||
|
update_image () {
|
||||||
|
case "$1" in socat) return 1;; esac
|
||||||
|
}
|
||||||
|
|
||||||
|
run () {
|
||||||
|
echo "$@" >&2
|
||||||
|
"$@"
|
||||||
|
}
|
||||||
|
|
||||||
|
# rbac rules
|
||||||
|
echo "applying RBAC rules"
|
||||||
|
for component in CSI_ATTACHER; do
|
||||||
|
eval current="\${${component}_RBAC}"
|
||||||
|
eval original="\${${component}_RBAC_YAML}"
|
||||||
|
if [ "$current" != "$original" ]; then
|
||||||
|
echo "Using non-default RBAC rules for $component. Changes from $original to $current are:"
|
||||||
|
diff -c <(wget --quiet -O - "$original") <(if [[ "$current" =~ ^http ]]; then wget --quiet -O - "$current"; else cat "$current"; fi) || true
|
||||||
|
fi
|
||||||
|
run kubectl apply -f "${current}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# deploy nfs plugin and registrar sidecar
|
||||||
|
echo "deploying nfs plugin components"
|
||||||
|
for i in $(ls ${BASE_DIR}/*.yaml | sort); do
|
||||||
|
echo " $i"
|
||||||
|
modified="$(cat "$i" | while IFS= read -r line; do
|
||||||
|
nocomments="$(echo "$line" | sed -e 's/ *#.*$//')"
|
||||||
|
if echo "$nocomments" | grep -q '^[[:space:]]*image:[[:space:]]*'; then
|
||||||
|
# Split 'image: quay.io/k8scsi/csi-attacher:v1.0.1'
|
||||||
|
# into image (quay.io/k8scsi/csi-attacher:v1.0.1),
|
||||||
|
# registry (quay.io/k8scsi),
|
||||||
|
# name (csi-attacher),
|
||||||
|
# tag (v1.0.1).
|
||||||
|
image=$(echo "$nocomments" | sed -e 's;.*image:[[:space:]]*;;')
|
||||||
|
registry=$(echo "$image" | sed -e 's;\(.*\)/.*;\1;')
|
||||||
|
name=$(echo "$image" | sed -e 's;.*/\([^:]*\).*;\1;')
|
||||||
|
tag=$(echo "$image" | sed -e 's;.*:;;')
|
||||||
|
|
||||||
|
# Variables are with underscores and upper case.
|
||||||
|
varname=$(echo $name | tr - _ | tr a-z A-Z)
|
||||||
|
|
||||||
|
# Now replace registry and/or tag, if set as env variables.
|
||||||
|
# If not set, the replacement is the same as the original value.
|
||||||
|
# Only do this for the images which are meant to be configurable.
|
||||||
|
if update_image "$name"; then
|
||||||
|
prefix=$(eval echo \${${varname}_REGISTRY:-${IMAGE_REGISTRY:-${registry}}}/ | sed -e 's;none/;;')
|
||||||
|
suffix=$(eval echo :\${${varname}_TAG:-${IMAGE_TAG:-${tag}}})
|
||||||
|
line="$(echo "$nocomments" | sed -e "s;$image;${prefix}${name}${suffix};")"
|
||||||
|
fi
|
||||||
|
echo " using $line" >&2
|
||||||
|
fi
|
||||||
|
echo "$line"
|
||||||
|
done)"
|
||||||
|
if ! echo "$modified" | kubectl apply -f -; then
|
||||||
|
echo "modified version of $i:"
|
||||||
|
echo "$modified"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# Wait until all pods are running. We have to make some assumptions
|
||||||
|
# about the deployment here, otherwise we wouldn't know what to wait
|
||||||
|
# for: the expectation is that we run attacher and nfs plugin in the default namespace.
|
||||||
|
cnt=0
|
||||||
|
while [ $(kubectl get pods 2>/dev/null | grep '^csi-.*nfsplugin.* Running ' | wc -l) -lt 2 ]; do
|
||||||
|
if [ $cnt -gt 30 ]; then
|
||||||
|
echo "Running pods:"
|
||||||
|
kubectl describe pods
|
||||||
|
|
||||||
|
echo >&2 "ERROR: nfs deployment not ready after over 5min"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo $(date +%H:%M:%S) "waiting for nfs deployment to complete, attempt #$cnt"
|
||||||
|
cnt=$(($cnt + 1))
|
||||||
|
sleep 10
|
||||||
|
done
|
||||||
Loading…
Reference in New Issue