ehlxr/docker_practice
1
0
Fork 0
mirror of https://github.com/yeasy/docker_practice.git synced 2026-03-10 20:04:36 +00:00
Code Issues Projects Releases Wiki Activity

Release v1.5.0: Restructure chapters and update for Docker v30.x

Browse Source
This commit is contained in:
Baohua Yang
2026-02-04 22:12:38 -08:00
parent b4b0d4160a
commit fdb879dcf2
304 changed files with 1314 additions and 364 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
11_orchestration/README.md Normal file
View File

@@ -0,0 +1,8 @@
# 容器编排
本章将介绍容器编排相关的技术与工具。
* [Etcd 项目](etcd/README.md)
* [Kubernetes - 开源容器编排引擎](kubernetes/README.md)
* [部署 Kubernetes](setup/README.md)
* [Kubernetes 命令行 kubectl](kubectl/README.md)

3
11_orchestration/etcd/README.md Normal file
View File

@@ -0,0 +1,3 @@
# etcd
`etcd` 是 `CoreOS` 团队发起的一个管理配置信息和服务发现（`Service Discovery`）的项目，在这一章里面，我们将基于 `etcd 3.x` 版本介绍该项目的目标，安装和使用，以及实现的技术。

BIN
11_orchestration/etcd/_images/etcd_logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

129
11_orchestration/etcd/cluster.md Normal file
View File

@@ -0,0 +1,129 @@
# etcd 集群
下面我们使用 [Docker Compose](../compose/) 模拟启动一个 3 节点的 `etcd` 集群。
编辑 `docker-compose.yml` 文件
```yaml
version: "3.6"
services:
node1:
image: quay.io/coreos/etcd:v3.4.0
volumes:
- node1-data:/etcd-data
expose:
- 2379
- 2380
networks:
cluster_net:
ipv4_address: 172.16.238.100
environment:
- ETCDCTL_API=3
command:
- /usr/local/bin/etcd
- --data-dir=/etcd-data
- --name
- node1
- --initial-advertise-peer-urls
- http://172.16.238.100:2380
- --listen-peer-urls
- http://0.0.0.0:2380
- --advertise-client-urls
- http://172.16.238.100:2379
- --listen-client-urls
- http://0.0.0.0:2379
- --initial-cluster
- node1=http://172.16.238.100:2380,node2=http://172.16.238.101:2380,node3=http://172.16.238.102:2380
- --initial-cluster-state
- new
- --initial-cluster-token
- docker-etcd
node2:
image: quay.io/coreos/etcd:v3.4.0
volumes:
- node2-data:/etcd-data
networks:
cluster_net:
ipv4_address: 172.16.238.101
environment:
- ETCDCTL_API=3
expose:
- 2379
- 2380
command:
- /usr/local/bin/etcd
- --data-dir=/etcd-data
- --name
- node2
- --initial-advertise-peer-urls
- http://172.16.238.101:2380
- --listen-peer-urls
- http://0.0.0.0:2380
- --advertise-client-urls
- http://172.16.238.101:2379
- --listen-client-urls
- http://0.0.0.0:2379
- --initial-cluster
- node1=http://172.16.238.100:2380,node2=http://172.16.238.101:2380,node3=http://172.16.238.102:2380
- --initial-cluster-state
- new
- --initial-cluster-token
- docker-etcd
node3:
image: quay.io/coreos/etcd:v3.4.0
volumes:
- node3-data:/etcd-data
networks:
cluster_net:
ipv4_address: 172.16.238.102
environment:
- ETCDCTL_API=3
expose:
- 2379
- 2380
command:
- /usr/local/bin/etcd
- --data-dir=/etcd-data
- --name
- node3
- --initial-advertise-peer-urls
- http://172.16.238.102:2380
- --listen-peer-urls
- http://0.0.0.0:2380
- --advertise-client-urls
- http://172.16.238.102:2379
- --listen-client-urls
- http://0.0.0.0:2379
- --initial-cluster
- node1=http://172.16.238.100:2380,node2=http://172.16.238.101:2380,node3=http://172.16.238.102:2380
- --initial-cluster-state
- new
- --initial-cluster-token
- docker-etcd
volumes:
node1-data:
node2-data:
node3-data:
networks:
cluster_net:
driver: bridge
ipam:
driver: default
config:
-
subnet: 172.16.238.0/24
```
使用 `docker compose up` 启动集群之后使用 `docker exec` 命令登录到任一节点测试 `etcd` 集群。
```bash
/ # etcdctl member list
daf3fd52e3583ff, started, node3, http://172.16.238.102:2380, http://172.16.238.102:2379
422a74f03b622fef, started, node1, http://172.16.238.100:2380, http://172.16.238.100:2379
ed635d2a2dbef43d, started, node2, http://172.16.238.101:2380, http://172.16.238.101:2379
```

112
11_orchestration/etcd/demo/cluster/docker-compose.yml Normal file
View File

@@ -0,0 +1,112 @@
version: "3.6"
services:
node1:
image: quay.io/coreos/etcd:v3.4.0
volumes:
- node1-data:/etcd-data
expose:
- 2379
- 2380
networks:
cluster_net:
ipv4_address: 172.16.238.100
environment:
- ETCDCTL_API=3
command:
- /usr/local/bin/etcd
- --data-dir=/etcd-data
- --name
- node1
- --initial-advertise-peer-urls
- http://172.16.238.100:2380
- --listen-peer-urls
- http://0.0.0.0:2380
- --advertise-client-urls
- http://172.16.238.100:2379
- --listen-client-urls
- http://0.0.0.0:2379
- --initial-cluster
- node1=http://172.16.238.100:2380,node2=http://172.16.238.101:2380,node3=http://172.16.238.102:2380
- --initial-cluster-state
- new
- --initial-cluster-token
- docker-etcd
node2:
image: quay.io/coreos/etcd:v3.4.0
volumes:
- node2-data:/etcd-data
networks:
cluster_net:
ipv4_address: 172.16.238.101
environment:
- ETCDCTL_API=3
expose:
- 2379
- 2380
command:
- /usr/local/bin/etcd
- --data-dir=/etcd-data
- --name
- node2
- --initial-advertise-peer-urls
- http://172.16.238.101:2380
- --listen-peer-urls
- http://0.0.0.0:2380
- --advertise-client-urls
- http://172.16.238.101:2379
- --listen-client-urls
- http://0.0.0.0:2379
- --initial-cluster
- node1=http://172.16.238.100:2380,node2=http://172.16.238.101:2380,node3=http://172.16.238.102:2380
- --initial-cluster-state
- new
- --initial-cluster-token
- docker-etcd
node3:
image: quay.io/coreos/etcd:v3.4.0
volumes:
- node3-data:/etcd-data
networks:
cluster_net:
ipv4_address: 172.16.238.102
environment:
- ETCDCTL_API=3
expose:
- 2379
- 2380
command:
- /usr/local/bin/etcd
- --data-dir=/etcd-data
- --name
- node3
- --initial-advertise-peer-urls
- http://172.16.238.102:2380
- --listen-peer-urls
- http://0.0.0.0:2380
- --advertise-client-urls
- http://172.16.238.102:2379
- --listen-client-urls
- http://0.0.0.0:2379
- --initial-cluster
- node1=http://172.16.238.100:2380,node2=http://172.16.238.101:2380,node3=http://172.16.238.102:2380
- --initial-cluster-state
- new
- --initial-cluster-token
- docker-etcd
volumes:
node1-data:
node2-data:
node3-data:
networks:
cluster_net:
driver: bridge
ipam:
driver: default
config:
-
subnet: 172.16.238.0/24

150
11_orchestration/etcd/etcdctl.md Normal file
View File

@@ -0,0 +1,150 @@
# 使用 etcdctl
`etcdctl` 是一个命令行客户端，它能提供一些简洁的命令，供用户直接跟 `etcd` 服务打交道，而无需基于 `HTTP API` 方式。这在某些情况下将很方便，例如用户对服务进行测试或者手动修改数据库内容。我们也推荐在刚接触 `etcd` 时通过 `etcdctl` 命令来熟悉相关的操作，这些操作跟 `HTTP API` 实际上是对应的。
`etcd` 项目二进制发行包中已经包含了 `etcdctl` 工具，没有的话，可以从 [github.com/etcd-io/etcd/releases](https://github.com/etcd-io/etcd/releases) 下载。
`etcdctl` 支持如下的命令，大体上分为数据库操作和非数据库操作两类，后面将分别进行解释。
```
NAME:
etcdctl - A simple command line client for etcd3.
USAGE:
etcdctl
VERSION:
3.4.0
API VERSION:
3.4
COMMANDS:
get Gets the key or a range of keys
put Puts the given key into the store
del Removes the specified key or range of keys [key, range_end)
txn Txn processes all the requests in one transaction
compaction Compacts the event history in etcd
alarm disarm Disarms all alarms
alarm list Lists all alarms
defrag Defragments the storage of the etcd members with given endpoints
endpoint health Checks the healthiness of endpoints specified in `--endpoints` flag
endpoint status Prints out the status of endpoints specified in `--endpoints` flag
watch Watches events stream on keys or prefixes
version Prints the version of etcdctl
lease grant Creates leases
lease revoke Revokes leases
lease timetolive Get lease information
lease keep-alive Keeps leases alive (renew)
member add Adds a member into the cluster
member remove Removes a member from the cluster
member update Updates a member in the cluster
member list Lists all members in the cluster
snapshot save Stores an etcd node backend snapshot to a given file
snapshot restore Restores an etcd member snapshot to an etcd directory
snapshot status Gets backend snapshot status of a given file
make-mirror Makes a mirror at the destination etcd cluster
migrate Migrates keys in a v2 store to a mvcc store
lock Acquires a named lock
elect Observes and participates in leader election
auth enable Enables authentication
auth disable Disables authentication
user add Adds a new user
user delete Deletes a user
user get Gets detailed information of a user
user list Lists all users
user passwd Changes password of user
user grant-role Grants a role to a user
user revoke-role Revokes a role from a user
role add Adds a new role
role delete Deletes a role
role get Gets detailed information of a role
role list Lists all roles
role grant-permission Grants a key to a role
role revoke-permission Revokes a key from a role
check perf Check the performance of the etcd cluster
help Help about any command
OPTIONS:
--cacert="" verify certificates of TLS-enabled secure servers using this CA bundle
--cert="" identify secure client using this TLS certificate file
--command-timeout=5s timeout for short running command (excluding dial timeout)
--debug[=false] enable client-side debug logging
--dial-timeout=2s dial timeout for client connections
--endpoints=[127.0.0.1:2379] gRPC endpoints
--hex[=false] print byte strings as hex encoded strings
--insecure-skip-tls-verify[=false] skip server certificate verification
--insecure-transport[=true] disable transport security for client connections
--key="" identify secure client using this TLS key file
--user="" username[:password] for authentication (prompt if password is not supplied)
-w, --write-out="simple" set the output format (fields, json, protobuf, simple, table)
```
## 数据库操作
数据库操作围绕对键值和目录的 CRUD （符合 REST 风格的一套操作：Create）完整生命周期的管理。
etcd 在键的组织上采用了层次化的空间结构（类似于文件系统中目录的概念），用户指定的键可以为单独的名字，如 `testkey`，此时实际上放在根目录 `/` 下面，也可以为指定目录结构，如 `cluster1/node2/testkey`，则将创建相应的目录结构。
>注：CRUD 即 Create, Read, Update, Delete，是符合 REST 风格的一套 API 操作。
### put
```bash
$ etcdctl put /testdir/testkey "Hello world"
OK
```
### get
获取指定键的值。例如
```bash
$ etcdctl put testkey hello
OK
$ etcdctl get testkey
testkey
hello
```
支持的选项为
`--sort` 对结果进行排序
`--consistent` 将请求发给主节点，保证获取内容的一致性
### del
删除某个键值。例如
```bash
$ etcdctl del testkey
1
```
## 非数据库操作
### watch
监测一个键值的变化，一旦键值发生更新，就会输出最新的值。
例如，用户更新 `testkey` 键值为 `Hello world`。
```bash
$ etcdctl watch testkey
PUT
testkey
2
```
### member
通过 `list`、`add`、`update`、`remove` 命令列出、添加、更新、删除 etcd 实例到 etcd 集群中。
例如本地启动一个 `etcd` 服务实例后，可以用如下命令进行查看。
```bash
$ etcdctl member list
422a74f03b622fef, started, node1, http://172.16.238.100:2380, http://172.16.238.100:23
```

99
11_orchestration/etcd/install.md Normal file
View File

@@ -0,0 +1,99 @@
# 安装
`etcd` 基于 `Go` 语言实现，因此，用户可以从 [项目主页](https://github.com/etcd-io/etcd) 下载源代码自行编译，也可以下载编译好的二进制文件，甚至直接使用制作好的 `Docker` 镜像文件来体验。
>注意：本章节内容基于 etcd `3.4.x` 版本
## 二进制文件方式下载
编译好的二进制文件都在 [github.com/etcd-io/etcd/releases](https://github.com/etcd-io/etcd/releases/) 页面，用户可以选择需要的版本，或通过下载工具下载。
例如，使用 `curl` 工具下载压缩包，并解压。
```bash
$ curl -L https://github.com/etcd-io/etcd/releases/download/v3.4.0/etcd-v3.4.0-linux-amd64.tar.gz -o etcd-v3.4.0-linux-amd64.tar.gz
# 国内用户可以使用以下方式加快下载
$ curl -L https://download.fastgit.org/etcd-io/etcd/releases/download/v3.4.0/etcd-v3.4.0-linux-amd64.tar.gz -o etcd-v3.4.0-linux-amd64.tar.gz
$ tar xzvf etcd-v3.4.0-linux-amd64.tar.gz
$ cd etcd-v3.4.0-linux-amd64
```
解压后，可以看到文件包括
```bash
$ ls
Documentation README-etcdctl.md README.md READMEv2-etcdctl.md etcd etcdctl
```
其中 `etcd` 是服务主文件，`etcdctl` 是提供给用户的命令客户端，其他文件是支持文档。
下面将 `etcd` `etcdctl` 文件放到系统可执行目录（例如 `/usr/local/bin/`）。
```bash
$ sudo cp etcd* /usr/local/bin/
```
默认 `2379` 端口处理客户端的请求，`2380` 端口用于集群各成员间的通信。启动 `etcd` 显示类似如下的信息：
```bash
$ etcd
...
2017-12-03 11:18:34.411579 I | embed: listening for peers on http://localhost:2380
2017-12-03 11:18:34.411938 I | embed: listening for client requests on localhost:2379
```
此时，可以使用 `etcdctl` 命令进行测试，设置和获取键值 `testkey: "hello world"`，检查 `etcd` 服务是否启动成功：
```bash
$ ETCDCTL_API=3 etcdctl member list
8e9e05c52164694d, started, default, http://localhost:2380, http://localhost:2379
$ ETCDCTL_API=3 etcdctl put testkey "hello world"
OK
$ etcdctl get testkey
testkey
hello world
```
说明 etcd 服务已经成功启动了。
## Docker 镜像方式运行
镜像名称为 `quay.io/coreos/etcd`，可以通过下面的命令启动 `etcd` 服务监听到 `2379` 和 `2380` 端口。
```bash
$ docker run \
-p 2379:2379 \
-p 2380:2380 \
--mount type=bind,source=/tmp/etcd-data.tmp,destination=/etcd-data \
--name etcd-gcr-v3.4.0 \
quay.io/coreos/etcd:v3.4.0 \
/usr/local/bin/etcd \
--name s1 \
--data-dir /etcd-data \
--listen-client-urls http://0.0.0.0:2379 \
--advertise-client-urls http://0.0.0.0:2379 \
--listen-peer-urls http://0.0.0.0:2380 \
--initial-advertise-peer-urls http://0.0.0.0:2380 \
--initial-cluster s1=http://0.0.0.0:2380 \
--initial-cluster-token tkn \
--initial-cluster-state new \
--log-level info \
--logger zap \
--log-outputs stderr
```
打开新的终端按照上一步的方法测试 `etcd` 是否成功启动。
## macOS 中运行
```bash
$ brew install etcd
$ etcd
$ etcdctl member list
```

22
11_orchestration/etcd/intro.md Normal file
View File

@@ -0,0 +1,22 @@
# 简介
![](../_images/etcd_logo.png)
`etcd` 是 `CoreOS` 团队于 2013 年 6 月发起的开源项目，它的目标是构建一个高可用的分布式键值（`key-value`）数据库，基于 `Go` 语言实现。我们知道，在分布式系统中，各种服务的配置信息的管理分享，服务的发现是一个很基本同时也是很重要的问题。`CoreOS` 项目就希望基于 `etcd` 来解决这一问题。
`etcd` 目前在 [github.com/etcd-io/etcd](https://github.com/etcd-io/etcd) 进行维护。
受到 [Apache ZooKeeper](https://zookeeper.apache.org/) 项目和 [doozer](https://github.com/ha/doozerd) 项目的启发，`etcd` 在设计的时候重点考虑了下面四个要素：
* 简单：具有定义良好、面向用户的 `API` ([gRPC](https://github.com/grpc/grpc))
* 安全：支持 `HTTPS` 方式的访问
* 快速：支持并发 `10 k/s` 的写操作
* 可靠：支持分布式结构，基于 `Raft` 的一致性算法
_Apache ZooKeeper 是一套知名的分布式系统中进行同步和一致性管理的工具。_
_doozer 是一个一致性分布式数据库。_
[_Raft_](https://raft.github.io/) _是一套通过选举主节点来实现分布式系统一致性的算法，相比于大名鼎鼎的 Paxos 算法，它的过程更容易被人理解，由 Stanford 大学的 Diego Ongaro 和 John Ousterhout 提出。更多细节可以参考_ [_raftconsensus.github.io_](http://raftconsensus.github.io)_。_
一般情况下，用户使用 `etcd` 可以在多个节点上启动多个实例，并添加它们为一个集群。同一个集群中的 `etcd` 实例将会保持彼此信息的一致性。

81
11_orchestration/kubectl/README.md Normal file
View File

@@ -0,0 +1,81 @@
# kubectl 使用
[kubectl](https://github.com/kubernetes/kubernetes) 是 Kubernetes 自带的客户端，可以用它来直接操作 Kubernetes。
使用格式有两种：
```bash
kubectl [flags]
kubectl [command]
```
## get
显示一个或多个资源
## describe
显示资源详情
## create
从文件或标准输入创建资源
## update
从文件或标准输入更新资源
## delete
通过文件名、标准输入、资源名或者 label selector 删除资源
## log
输出 pod 中一个容器的日志
## rolling-update
对指定的 replication controller 执行滚动升级
## exec
在容器内部执行命令
## port-forward
将本地端口转发到Pod
## proxy
为 Kubernetes API server 启动代理服务器
## run
在集群中使用指定镜像启动容器
## expose
将 replication controller service 或 pod 暴露为新的 kubernetes service
## label
更新资源的 label
## config
修改 kubernetes 配置文件
## cluster-info
显示集群信息
## api-versions
以 "组/版本" 的格式输出服务端支持的 API 版本
## version
输出服务端和客户端的版本信息
## help
显示各个命令的帮助信息

13
11_orchestration/kubernetes/README.md Normal file
View File

@@ -0,0 +1,13 @@
# Kubernetes
`Kubernetes` 是 Google 团队发起并维护的基于 Docker 的开源容器集群管理系统，它不仅支持常见的云平台，而且支持内部数据中心。
建于 Docker 之上的 `Kubernetes` 可以构建一个容器的调度服务，其目的是让用户透过 `Kubernetes` 集群来进行云端容器集群的管理，而无需用户进行复杂的设置工作。系统会自动选取合适的工作节点来执行具体的容器集群调度处理工作。其核心概念是 `Container Pod`。一个 `Pod` 由一组工作于同一物理工作节点的容器构成。这些组容器拥有相同的网络命名空间、IP以及存储配额，也可以根据实际情况对每一个 `Pod` 进行端口映射。此外，`Kubernetes` 工作节点会由主系统进行管理，节点包含了能够运行 Docker 容器所用到的服务。
本章将分为 5 节介绍 `Kubernetes`，包括
* 项目简介
* 快速入门
* 基本概念
* 实践例子
* 架构分析等高级话题

BIN
11_orchestration/kubernetes/_images/k8s-singlenode-docker.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 31 KiB

BIN
11_orchestration/kubernetes/_images/k8s_architecture.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 217 KiB

BIN
11_orchestration/kubernetes/_images/kube-proxy.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 67 KiB

BIN
11_orchestration/kubernetes/_images/kubernetes_design.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 153 KiB

BIN
11_orchestration/kubernetes/_images/kubernetes_logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

374
11_orchestration/kubernetes/_images/kubernetes_logo.svg Normal file
View File

@@ -0,0 +1,374 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
version="1.1"
width="562.78589"
height="134.62993"
id="svg2"
xml:space="preserve"><metadata
id="metadata8"><rdf:RDF><cc:Work
rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" /><dc:title></dc:title></cc:Work></rdf:RDF></metadata><defs
id="defs6" /><g
transform="matrix(1.25,0,0,-1.25,-169.2546,1414.8444)"
id="g10"><g
transform="matrix(0,0.18092,-0.18275,0,0,0)"
id="g12"><path
d="m 6196.6587,-1043.6173 -94.2902,-195.4939 -211.9113,-48.3046 -169.9617,135.2607 -0.025,216.9692 169.9297,135.2974 211.9254,-48.257 94.3336,-195.4718 z"
id="path14"
style="fill:none;stroke:#ffffff;stroke-width:118.52590179;stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 190.7198,1121.0876 35.725,-17.0586 8.8274,-38.3391 -24.7181,-30.7489 -39.6505,0 -24.7249,30.7434 8.8192,38.3412 35.7219,17.0665 z"
id="path16"
style="fill:#336ee5;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0,0.18092,-0.18275,0,0,0)"
id="g18"><path
d="m 6196.6587,-1043.6173 -94.2888,-195.4939 -211.9141,-48.3046 -169.9603,135.2593 -0.025,216.9723 169.9297,135.2942 211.9237,-48.2572 94.3353,-195.4701 z"
id="path20"
style="fill:none;stroke:#336ee5;stroke-width:74.74790192;stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><g
transform="scale(0.18275,0.18275)"
id="g22"><path
d="m 1013.0746,6022.3961 c 73.5242,16.6963 146.8298,-29.4129 163.7263,-102.9881 16.9013,-73.5693 -29.0033,-146.7459 -102.5258,-163.4409 -73.5273,-16.6903 -146.8343,29.4189 -163.7325,102.9867 -16.8967,73.5769 29.0047,146.7505 102.532,163.4423 z"
id="path24"
style="fill:none;stroke:#ffffff;stroke-width:30.78089905;stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 188.4186,1104.8074 2.7468,-0.01 0.1749,-20.1918 -4.0657,-0.038 1.144,20.236 z"
id="path26"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="scale(0.17178,0.18275)"
id="g28"><path
d="m 1096.8024,6045.6095 15.9899,-0.036 1.0191,-110.4894 -23.6699,-0.2094 6.6609,110.7345 z"
id="path30"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 193.0309,1104.8074 -2.7474,-0.01 -0.1703,-20.1918 4.0654,-0.037 -1.1477,20.2354 z"
id="path32"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="scale(0.17178,0.18275)"
id="g34"><path
d="m 1123.6518,6045.6098 -15.993,-0.036 -0.991,-110.4894 23.6681,-0.2029 -6.6841,110.7283 z"
id="path36"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 192.8625,1112.4315 c 0,-1.3119 -0.9576,-2.3758 -2.1382,-2.3758 -1.1806,0 -2.1379,1.0639 -2.1379,2.3752 0,1.3119 0.9564,2.3754 2.1379,2.3763 1.1806,0 2.1382,-1.0636 2.1382,-2.3757"
id="path38"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.16447,2e-5,-2e-5,0.18275,0,0)"
id="g40"><path
d="m 1173.5053,6087.183 c -8e-4,-7.1804 -5.8238,-12.9997 -13.0019,-12.9988 -7.1785,8e-4 -12.998,5.8229 -12.9969,12.9971 0,7.1819 5.817,13.0011 13.0006,13.0031 7.1781,-6e-4 12.9994,-5.8229 12.9982,-13.0014 z"
id="path42"
style="fill:none;stroke:#ffffff;stroke-width:0.2744;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 188.5873,1112.4629 c 5e-4,-0.1876 -0.009,-0.458 -0.003,-0.6389 0.0289,-0.7566 0.1945,-1.3368 0.294,-2.0344 0.1797,-1.4922 0.3311,-2.7289 0.2381,-3.8781 -0.0851,-0.5757 -0.4184,-0.8028 -0.6959,-1.0689 l 3.3721,-2.2315 -0.5069,9.8354 -2.6988,0.016 z"
id="path44"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0,-0.17178,0.18275,0,0,0)"
id="g46"><path
d="m -6476.0579,1031.9675 c 1.0925,0 2.6666,-0.048 3.7194,-0.014 4.4045,0.1568 7.7839,1.0641 11.8431,1.6087 8.6865,0.9819 15.8862,1.8102 22.5791,1.3028 3.3483,-0.4652 4.6701,-2.2896 6.2212,-3.8095 l 12.9884,18.4555 -57.257,-2.7751 -0.094,-14.7685 z"
id="path48"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 192.8625,1112.4629 c -0.001,-0.1876 0.008,-0.458 0.002,-0.6389 -0.0289,-0.7566 -0.1942,-1.3368 -0.2934,-2.0344 -0.1797,-1.4922 -0.3311,-2.7289 -0.2384,-3.8781 0.0847,-0.5757 0.4184,-0.8028 0.6959,-1.0689 l -3.3724,-2.2315 0.5074,9.8354 2.6989,0.016 z"
id="path50"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0,-0.17178,0.18275,0,0,0)"
id="g52"><path
d="m -6476.0579,1055.3604 c 1.0925,0 2.6666,0.046 3.7194,0.011 4.4045,-0.1534 7.7839,-1.0624 11.8431,-1.6022 8.6865,-0.9867 15.8862,-1.815 22.5791,-1.3076 3.3483,0.4669 4.6701,2.291 6.2212,3.8081 l 12.9884,-18.4541 -57.257,2.7765 -0.094,14.7685 z"
id="path54"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><g
transform="scale(0.18275,0.18275)"
id="g56"><path
d="m 1073.7275,5865.2637 -30.1062,-14.4303 -30.1,14.438 -7.4344,32.4422 20.8395,26.0065 33.4099,0 20.8321,-26.0175 -7.4409,-32.4374 z"
id="path58"
style="fill:none;stroke:#ffffff;stroke-width:30.34600067;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 166.9153,1092.2468 1.7178,2.1436 15.8952,-12.4532 -2.5049,-3.2023 -15.1081,13.5119 z"
id="path60"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.1071,0.1343,-0.14288,0.11394,0,0)"
id="g62"><path
d="m 5577.0313,3012.37 15.9908,-0.036 1.0134,-110.4917 -23.6665,-0.2083 6.6623,110.7357 z"
id="path64"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 169.7905,1095.853 -1.7074,-2.1523 15.6805,-12.723 2.5636,3.1561 -16.5367,11.7192 z"
id="path66"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.1071,0.1343,-0.14288,0.11394,0,0)"
id="g68"><path
d="m 5603.8799,3012.3729 -15.9928,-0.039 -0.9944,-110.4931 23.6693,-0.2001 -6.6821,110.7321 z"
id="path70"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 163.7252,1100.4746 c 1.0258,-0.8183 1.2608,-2.2297 0.5247,-3.1529 -0.7368,-0.9233 -2.1651,-1.0089 -3.191,-0.1908 -1.0256,0.8181 -1.2606,2.2292 -0.5238,3.1524 0.7358,0.9235 2.1642,1.0086 3.1901,0.1913"
id="path72"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.10253,0.1286,-0.14289,0.11392,0,0)"
id="g74"><path
d="m 5852.363,3053.3992 c 0,-7.181 -5.8216,-13.0011 -13.0009,-13.0005 -7.1815,0 -13.0025,5.8227 -13.0013,13.0025 0,7.1796 5.8198,12.9977 13.0013,12.9949 7.1799,0 12.998,-5.8198 13.0009,-12.9969 z"
id="path76"
style="fill:none;stroke:#ffffff;stroke-width:0.2744;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 161.0351,1097.1516 c 0.1474,-0.1171 0.3518,-0.2931 0.4986,-0.4005 0.6089,-0.4496 1.1656,-0.6818 1.7734,-1.0387 1.2781,-0.7894 2.3397,-1.4428 3.1807,-2.232 0.3969,-0.4249 0.3662,-0.8268 0.4008,-1.2104 l 3.8472,1.2453 -8.0047,5.7356 -1.696,-2.0993 z"
id="path78"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.1343,-0.1071,0.11394,0.14288,0,0)"
id="g80"><path
d="m -3249.2299,5243.3232 c 1.0919,-9e-4 2.6612,-0.054 3.7205,-0.014 4.403,0.1539 7.7794,1.0602 11.8417,1.6056 8.6817,0.9844 15.8837,1.8121 22.5768,1.3042 3.3486,-0.4641 4.6681,-2.2882 6.2175,-3.8109 l 12.9912,18.4518 -57.2536,-2.7726 -0.094,-14.7639 z"
id="path82"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 163.6999,1100.4936 c 0.1472,-0.1176 0.3643,-0.278 0.5012,-0.3965 0.574,-0.4944 0.9247,-0.9851 1.4077,-1.4979 1.0548,-1.0709 1.9275,-1.9601 2.8842,-2.6039 0.5026,-0.2928 0.8876,-0.1737 1.2691,-0.1221 l -0.3586,-4.0275 -7.3732,6.5273 1.6696,2.1206 z"
id="path84"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.1343,-0.1071,0.11394,0.14288,0,0)"
id="g86"><path
d="m -3249.2341,5266.7113 c 1.0978,-3e-4 2.6671,0.053 3.7204,0.013 4.4068,-0.1565 7.7837,-1.0596 11.8432,-1.6053 8.6876,-0.983 15.8876,-1.8099 22.5782,-1.2999 3.3503,0.462 4.6704,2.2847 6.2195,3.8072 l 12.9884,-18.4521 -57.2517,2.768 -0.098,14.7688 z"
id="path88"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 163.3717,1067.5885 -0.6052,2.6796 19.6464,4.6636 0.9419,-3.9555 -19.9831,-3.3877 z"
id="path90"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.03823,-0.16747,0.17816,0.04067,0,0)"
id="g92"><path
d="m -5847.3595,2171.5736 -15.992,0.034 -1.017,110.4899 23.669,0.2087 -6.66,-110.7329 z"
id="path94"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 162.3444,1072.0845 0.6182,-2.6767 19.7238,4.3271 -0.8685,3.9719 -19.4735,-5.6223 z"
id="path96"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.03823,-0.16747,0.17816,0.04067,0,0)"
id="g98"><path
d="m -5874.2073,2171.5679 15.9931,0.04 0.9924,110.4916 -23.6673,0.203 6.6818,-110.7349 z"
id="path100"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 154.9497,1070.2236 c 1.2787,0.2928 2.529,-0.4039 2.7921,-1.5551 0.2625,-1.1509 -0.561,-2.3213 -1.8405,-2.6127 -1.2785,-0.2928 -2.5285,0.4039 -2.7919,1.5551 -0.2625,1.1509 0.5613,2.3207 1.8403,2.6127"
id="path102"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.03662,-0.16034,0.17816,0.04069,0,0)"
id="g104"><path
d="m -6133.9467,2130.5761 c 0,7.1782 5.8161,12.9985 12.9991,12.9988 7.1756,-3e-4 12.9985,-5.8192 12.9951,-13.0005 0.01,-7.1753 -5.8189,-12.9966 -12.9988,-12.9988 -7.1773,5e-4 -12.9963,5.8218 -12.9954,13.0005 z"
id="path106"
style="fill:none;stroke:#ffffff;stroke-width:0.2744;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 155.8706,1066.049 c 0.1829,0.042 0.4488,0.093 0.6234,0.14 0.7316,0.1962 1.2606,0.4867 1.9182,0.7387 1.4148,0.508 2.5869,0.9301 3.7281,1.0951 0.58,0.046 0.8745,-0.2282 1.1968,-0.4405 l 1.4247,3.7842 -9.4759,-2.6816 0.5847,-2.6359 z"
id="path108"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.16747,0.03823,-0.04067,0.17816,0,0)"
id="g110"><path
d="m 2265.6285,5497.4356 c 1.0922,0 2.6666,-0.045 3.7177,-0.01 4.4093,0.1534 7.7862,1.0596 11.8448,1.603 8.6851,0.9845 15.8865,1.8117 22.5771,1.3023 3.3483,-0.4592 4.6676,-2.2825 6.2201,-3.8072 l 12.9895,18.4535 -57.2534,-2.7672 -0.096,-14.7744 z"
id="path112"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 154.9191,1070.2165 c 0.1834,0.042 0.4447,0.1106 0.6224,0.1451 0.7447,0.1404 1.3471,0.1078 2.0489,0.1661 1.4947,0.1571 2.7338,0.2849 3.8339,0.6313 0.5428,0.2106 0.6894,0.5859 0.8867,0.9165 l 2.9262,-2.791 -9.701,-1.696 -0.6171,2.628 z"
id="path114"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.16747,0.03823,-0.04067,0.17816,0,0)"
id="g116"><path
d="m 2265.6266,5520.8273 c 1.0955,0 2.6674,0.048 3.7193,0.017 4.4096,-0.1585 7.7859,-1.0659 11.8448,-1.6093 8.6865,-0.9822 15.8843,-1.809 22.5766,-1.3005 3.3536,0.4626 4.6684,2.287 6.2195,3.8092 l 12.9917,-18.4527 -57.2533,2.7683 -0.099,14.7682 z"
id="path118"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 180.4031,1049.4196 -2.473,1.1973 8.6032,18.2685 3.6802,-1.73 -9.8104,-17.7358 z"
id="path120"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.15477,-0.07453,0.07929,0.16465,0,0)"
id="g122"><path
d="m -1704.3131,5602.1797 -15.9959,0.035 -1.0171,110.4913 23.6718,0.2081 -6.6588,-110.7343 z"
id="path124"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 176.2469,1051.4203 2.4784,-1.1858 8.9141,18.1179 -3.6474,1.7978 -7.7451,-18.7299 z"
id="path126"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.15477,-0.07453,0.07929,0.16465,0,0)"
id="g128"><path
d="m -1731.1657,5602.1774 15.9936,0.038 0.9918,110.4882 -23.669,0.2044 6.6836,-110.7309 z"
id="path130"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 173.0911,1044.4779 c 0.5695,1.1824 1.8927,1.7255 2.9568,1.2132 1.0642,-0.5122 1.4653,-1.8858 0.8964,-3.0676 -0.5687,-1.1826 -1.8933,-1.7252 -2.9569,-1.2132 -1.0644,0.5122 -1.4655,1.8856 -0.8963,3.0676"
id="path132"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.14819,-0.07134,0.07926,0.16466,0,0)"
id="g134"><path
d="m -1806.2371,5560.6799 c 3e-4,7.1804 5.821,12.9988 13.0023,13.0011 7.1781,0 12.9974,-5.8221 12.9957,-12.9997 0.01,-7.1801 -5.821,-12.9991 -12.9974,-12.9994 -7.1822,-5e-4 -13.0014,5.819 -13.0006,12.998 z"
id="path136"
style="fill:none;stroke:#ffffff;stroke-width:0.2744;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 176.9292,1042.5954 c 0.0817,0.1695 0.207,0.4085 0.2792,0.5749 0.3033,0.6934 0.4057,1.2883 0.6183,1.9596 0.4853,1.4227 0.8858,2.6025 1.468,3.5974 0.3266,0.4819 0.7249,0.542 1.0908,0.6614 l -2.0698,3.4733 -3.8107,-9.0808 2.4242,-1.1858 z"
id="path138"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.07453,0.15477,-0.16465,0.07929,0,0)"
id="g140"><path
d="m 5915.209,1602.9547 c 1.0953,0 2.6649,-0.05 3.7202,-0.012 4.4056,0.1519 7.7811,1.0601 11.8409,1.605 8.6862,0.9844 15.8865,1.8127 22.5763,1.3028 3.3508,-0.4666 4.6717,-2.2873 6.222,-3.8084 l 12.9909,18.4516 -57.2547,-2.7709 -0.096,-14.7685 z"
id="path142"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 173.0784,1044.4501 c 0.0813,0.169 0.1902,0.4165 0.2744,0.5766 0.354,0.6698 0.7554,1.12 1.1471,1.7059 0.8099,1.266 1.482,2.3145 1.8975,3.3905 0.1735,0.5553 -0.0286,0.904 -0.1644,1.2646 l 4.0071,0.5473 -4.7237,-8.6411 -2.438,1.1562 z"
id="path144"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.07453,0.15477,-0.16465,0.07929,0,0)"
id="g146"><path
d="m 5915.2107,1626.3439 c 1.0917,0 2.6638,0.052 3.7171,0.016 4.4073,-0.1553 7.7822,-1.0655 11.8432,-1.6064 8.6876,-0.9845 15.8853,-1.8102 22.5771,-1.3051 3.3499,0.4646 4.6689,2.2873 6.22,3.8141 l 12.9887,-18.4575 -57.2528,2.7737 -0.093,14.7653 z"
id="path148"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 205.2142,1051.4548 -2.4778,-1.1863 -8.9186,18.1171 3.6467,1.798 7.7497,-18.7288 z"
id="path150"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.15477,0.07454,-0.0793,0.16465,0,0)"
id="g152"><path
d="m 3732.232,4696.5288 -15.992,0.035 -1.0145,110.4942 23.669,0.2069 -6.6625,-110.7357 z"
id="path154"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 201.0589,1049.4533 2.4729,1.1988 -8.6082,18.2661 -3.6797,-1.7314 9.815,-17.7335 z"
id="path156"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.15477,0.07454,-0.0793,0.16465,0,0)"
id="g158"><path
d="m 3705.3822,4696.5274 15.9982,0.038 0.9873,110.4936 -23.6687,0.201 6.6832,-110.7329 z"
id="path160"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 204.5188,1042.6575 c -0.5697,1.1821 -0.1686,2.5563 0.8952,3.0685 1.0636,0.5122 2.3879,-0.03 2.9574,-1.2124 0.5689,-1.182 0.1684,-2.5554 -0.8952,-3.0682 -1.0644,-0.5125 -2.3876,0.031 -2.9574,1.2121"
id="path162"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.14817,0.07138,-0.07932,0.16464,0,0)"
id="g164"><path
d="m 3871.7606,4654.3567 c -9e-4,7.1818 5.8221,13.0025 13.0017,13.0022 7.1767,0 12.9999,-5.8209 12.9991,-13.002 0,-7.1784 -5.8212,-12.9977 -13,-12.9996 -7.183,0 -12.9977,5.8215 -13.0008,12.9994 z"
id="path166"
style="fill:none;stroke:#ffffff;stroke-width:0.2744;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 208.3836,1044.4844 c -0.0811,0.1698 -0.1896,0.4167 -0.2747,0.5775 -0.354,0.6698 -0.7554,1.1199 -1.1472,1.7056 -0.8095,1.2665 -1.4819,2.3142 -1.8978,3.3908 -0.1732,0.5553 0.029,0.9037 0.1636,1.2639 l -4.0059,0.5477 4.7233,-8.6411 2.4387,1.1556 z"
id="path168"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.07454,-0.15477,0.16465,0.0793,0,0)"
id="g170"><path
d="m -4951.7391,3507.378 c -1.097,5e-4 -2.6659,0.055 -3.7224,0.017 -4.4059,-0.1556 -7.7822,-1.065 -11.844,-1.6073 -8.6868,-0.9827 -15.8817,-1.8093 -22.5771,-1.3039 -3.35,0.4646 -4.665,2.2876 -6.2153,3.8089 l -12.992,-18.4507 57.2536,2.7686 0.097,14.7677 z"
id="path172"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 204.5333,1042.6297 c -0.0825,0.1693 -0.2072,0.4091 -0.2801,0.5746 -0.3024,0.6945 -0.405,1.2895 -0.6179,1.9602 -0.4859,1.423 -0.8861,2.6022 -1.4684,3.5983 -0.3265,0.4816 -0.7248,0.5408 -1.091,0.6599 l 2.0701,3.4738 3.8112,-9.0802 -2.4239,-1.1866 z"
id="path174"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.07454,-0.15477,0.16465,0.0793,0,0)"
id="g176"><path
d="m -4951.7379,3483.9904 c -1.0953,0 -2.6683,-0.049 -3.7196,-0.014 -4.4074,0.1565 -7.7871,1.0633 -11.8438,1.6036 -8.687,0.987 -15.8847,1.8128 -22.5799,1.3073 -3.3508,-0.4663 -4.6681,-2.2904 -6.2186,-3.8114 l -12.989,18.4538 57.2508,-2.7706 0.1001,-14.7688 z"
id="path178"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 219.1056,1072.1339 -0.6173,-2.6771 -19.7252,4.3226 0.868,3.9727 19.4745,-5.6182 z"
id="path180"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.03822,0.16747,-0.17816,0.04066,0,0)"
id="g182"><path
d="m 6368.633,136.4414 -15.9914,0.0349 -1.0165,110.4948 23.6699,0.206 -6.662,-110.7357 z"
id="path184"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 218.0798,1067.6375 0.6046,2.6794 -19.6481,4.659 -0.9405,-3.9555 19.984,-3.3829 z"
id="path186"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.03822,0.16747,-0.17816,0.04066,0,0)"
id="g188"><path
d="m 6341.7858,136.44 15.9894,0.0363 0.993,110.4933 -23.6673,0.2019 6.6849,-110.7315 z"
id="path190"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 225.5496,1066.1054 c -1.279,0.2923 -2.103,1.4613 -1.8405,2.6127 0.2631,1.1517 1.5126,1.8479 2.7913,1.5562 1.2793,-0.2914 2.1036,-1.4609 1.8411,-2.6127 -0.2631,-1.1514 -1.5123,-1.8482 -2.7919,-1.5562"
id="path192"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.03657,0.16035,-0.17817,0.04063,0,0)"
id="g194"><path
d="m 6624.6812,93.8699 c 0,7.1821 5.8189,12.9977 13.0011,12.9988 7.1796,-0.003 12.9974,-5.8215 12.9971,-12.9986 0,-7.1795 -5.8161,-13.0019 -12.9985,-13.0013 -7.1813,0.0022 -13.0012,5.8223 -12.9997,13.0011 z"
id="path196"
style="fill:none;stroke:#ffffff;stroke-width:0.2744;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 226.531,1070.2664 c -0.1826,0.042 -0.4439,0.1105 -0.6225,0.1443 -0.7435,0.1411 -1.3462,0.1085 -2.048,0.1669 -1.4947,0.1562 -2.7346,0.2844 -3.8345,0.6305 -0.5422,0.2103 -0.6891,0.5862 -0.8869,0.9164 l -2.9251,-2.7918 9.7005,-1.6937 0.6165,2.6274 z"
id="path198"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.16748,-0.03822,0.04066,0.17817,0,0)"
id="g200"><path
d="m -100.5092,5985.5941 c -1.0916,-2e-4 -2.6645,0.05 -3.7196,0.012 -4.4028,-0.1522 -7.7808,-1.0598 -11.8414,-1.6029 -8.684,-0.9862 -15.8879,-1.8139 -22.5803,-1.3054 -3.3477,0.4654 -4.6655,2.2895 -6.2161,3.8103 l -12.9872,-18.4544 57.2485,2.7726 0.0961,14.7679 z"
id="path202"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 225.5803,1066.0992 c -0.1834,0.042 -0.4485,0.093 -0.624,0.14 -0.731,0.1953 -1.26,0.4859 -1.9176,0.7379 -1.4148,0.5074 -2.5869,0.93 -3.7281,1.0953 -0.5794,0.046 -0.8754,-0.2288 -1.1965,-0.4405 l -1.4247,3.7839 9.4748,-2.6821 -0.5839,-2.6345 z"
id="path204"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.16748,-0.03822,0.04066,0.17817,0,0)"
id="g206"><path
d="m -100.5077,5962.2049 c -1.0954,6e-4 -2.6652,-0.052 -3.7225,-0.013 -4.4022,0.1536 -7.7803,1.0581 -11.8358,1.6027 -8.6899,0.9836 -15.8896,1.8116 -22.5808,1.3036 -3.3505,-0.4626 -4.6715,-2.2858 -6.22,-3.8066 l -12.9867,18.4527 57.2477,-2.7737 0.0981,-14.766 z"
id="path208"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 211.6429,1095.893 1.7084,-2.1515 -15.6779,-12.7267 -2.5639,3.1547 16.5334,11.7235 z"
id="path210"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.10711,-0.1343,0.14287,0.11395,0,0)"
id="g212"><path
d="m -4219.3791,4644.5956 15.993,-0.032 1.012,-110.4928 -23.6625,-0.208 6.6575,110.7329 z"
id="path214"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 214.52,1092.2873 -1.7192,2.1436 -15.8913,-12.4566 2.5053,-3.2014 15.1052,13.5144 z"
id="path216"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.10711,-0.1343,0.14287,0.11395,0,0)"
id="g218"><path
d="m -4192.5257,4644.6018 -15.9959,-0.041 -0.9899,-110.4863 23.6639,-0.2055 -6.6781,110.7329 z"
id="path220"
style="fill:none;stroke:#ffffff;stroke-width:0.41159999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 220.3747,1097.1729 c -1.0256,-0.8181 -2.4542,-0.7331 -3.1901,0.1902 -0.7364,0.9229 -0.5017,2.3349 0.5233,3.1529 1.0261,0.8181 2.4542,0.7331 3.1909,-0.1899 0.7359,-0.923 0.5018,-2.3352 -0.5241,-3.1532"
id="path222"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.10257,-0.12857,0.14286,0.11397,0,0)"
id="g224"><path
d="m -4379.2058,4686.834 c -3e-4,-7.1787 -5.8215,-12.9999 -13,-12.9971 -7.179,-6e-4 -13,5.8218 -13.0017,12.998 0,7.1813 5.8204,13.0011 13.0009,13.0005 7.1778,-5e-4 13.0022,-5.8192 13.0008,-13.0014 z"
id="path226"
style="fill:none;stroke:#ffffff;stroke-width:0.2744;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 217.7337,1100.5347 c -0.1471,-0.1173 -0.364,-0.2786 -0.5012,-0.3962 -0.5737,-0.4938 -0.9241,-0.9859 -1.4068,-1.4976 -1.0554,-1.0718 -1.9273,-1.9613 -2.8843,-2.6053 -0.5026,-0.2925 -0.8872,-0.1735 -1.2691,-0.1216 l 0.3586,-4.028 7.3724,6.5284 -1.6696,2.1203 z"
id="path228"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.1343,0.10711,-0.11395,0.14287,0,0)"
id="g230"><path
d="m 4985.5978,3965.3263 c -1.097,8e-4 -2.6683,0.052 -3.7208,0.016 -4.4036,-0.1551 -7.7842,-1.0647 -11.8383,-1.6067 -8.6933,-0.9845 -15.8919,-1.8139 -22.5811,-1.3048 -3.3517,0.4632 -4.6689,2.2839 -6.2198,3.8098 l -12.9886,-18.4561 57.2496,2.7757 0.099,14.7665 z"
id="path232"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 220.3994,1097.193 c -0.1474,-0.1171 -0.3532,-0.2926 -0.4984,-0.4006 -0.6088,-0.4498 -1.1664,-0.6814 -1.7733,-1.0391 -1.2784,-0.7898 -2.34,-1.4429 -3.1811,-2.2323 -0.3968,-0.4247 -0.3662,-0.8266 -0.4008,-1.2101 l -3.8472,1.2452 8.0045,5.7362 1.6963,-2.0993 z"
id="path234"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><g
transform="matrix(0.1343,0.10711,-0.11395,0.14287,0,0)"
id="g236"><path
d="m 4985.5986,3941.9357 c -1.0956,0 -2.6665,-0.048 -3.7216,-0.013 -4.4042,0.1528 -7.7836,1.0656 -11.8429,1.6047 -8.6839,0.9848 -15.887,1.8114 -22.5788,1.3028 -3.3499,-0.4629 -4.6672,-2.2867 -6.2169,-3.8072 l -12.9903,18.4524 57.2507,-2.7706 0.1,-14.7687 z"
id="path238"
style="fill:none;stroke:#ffffff;stroke-width:0.26840001;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /></g><path
d="m 187.367,1086.9849 c -0.0426,-1.0035 -0.868,-1.8049 -1.8822,-1.8049 -0.4156,0 -0.7988,0.1338 -1.1101,0.3606 l -0.489,-0.2324 1.0012,-2.0696 4.3855,2.1152 -0.9958,2.0693 -0.9096,-0.4382 z"
id="path240"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 180.2086,1080.3039 c 0.7586,-0.6588 0.8697,-1.8043 0.2379,-2.5971 -0.2591,-0.3243 -0.603,-0.5409 -0.9746,-0.6429 l -0.1227,-0.5276 2.2419,-0.5076 1.0811,4.7477 -2.2393,0.5119 -0.2243,-0.9844 z"
id="path242"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 180.9609,1070.5462 c 0.9874,0.182 1.9525,-0.4448 2.1782,-1.4335 0.0927,-0.4045 0.0476,-0.8087 -0.1052,-1.1628 l 0.3365,-0.4243 1.7952,1.4357 -3.039,3.805 -1.7958,-1.4315 0.6301,-0.7886 z"
id="path244"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 189.059,1065.0538 c 0.4731,0.8861 1.5653,1.2495 2.4786,0.8095 0.3739,-0.1797 0.6613,-0.4665 0.8438,-0.8064 l 0.5415,0 -0.004,2.2986 -4.8699,0 0,-2.2975 1.0097,6e-4 z"
id="path246"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 198.3934,1067.993 c -0.3971,0.9233 -8e-4,2.003 0.9131,2.4426 0.3739,0.1809 0.777,0.2262 1.1562,0.1562 l 0.3391,0.4227 -1.7989,1.4309 -3.0339,-3.8095 1.7954,-1.4326 0.629,0.7897 z"
id="path248"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 201.9577,1077.1194 c -0.9688,0.2648 -1.5661,1.2481 -1.3408,2.2366 0.0922,0.4045 0.3079,0.7489 0.5987,1.0023 l -0.1182,0.5284 -2.2399,-0.5148 1.0862,-4.7472 2.2394,0.5103 -0.2254,0.9844 z"
id="path250"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 197.0442,1085.5809 c -0.811,-0.5922 -1.9525,-0.4462 -2.5844,0.3469 -0.2588,0.324 -0.394,0.7073 -0.4104,1.0928 l -0.4868,0.2369 -0.9935,-2.0726 4.388,-2.1104 0.9975,2.0695 -0.9104,0.4369 z"
id="path252"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 281.6892,1072.3652 c 0.7671,0.8949 1.5449,1.8431 2.3332,2.8442 0.7883,1.0015 1.5554,2.035 2.3012,3.1003 0.7458,1.0653 1.4595,2.0988 2.1413,3.1003 0.682,1.0014 1.2784,1.9496 1.7898,2.8445 l 9.4603,0 c -0.767,-0.9802 -1.6407,-2.0455 -2.6206,-3.196 -0.9802,-1.1506 -1.9922,-2.3228 -3.0365,-3.5159 -1.044,-1.1931 -2.088,-2.3436 -3.132,-3.4517 -1.044,-1.1078 -1.9922,-2.1305 -2.8445,-3.0682 1.0655,-1.0228 2.2161,-2.2479 3.4517,-3.6754 1.2359,-1.4276 2.4503,-2.9084 3.6437,-4.4425 1.1931,-1.5341 2.2903,-3.0682 3.2918,-4.6023 1.0015,-1.5341 1.8003,-2.8979 2.397,-4.091 l -9.0127,0 c -0.5114,1.0653 -1.1827,2.2692 -2.0138,3.6116 -0.8308,1.3423 -1.7149,2.6632 -2.6526,3.9632 -0.9375,1.2997 -1.8856,2.5353 -2.8443,3.7074 -0.959,1.1718 -1.8431,2.1413 -2.653,2.9083 l 0,-14.1905 -7.9262,0 0,43.0189 7.9262,1.3422 0,-26.2074 z"
id="path254"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 329.694,1055.2343 c -1.4491,-0.3836 -3.2069,-0.7671 -5.2736,-1.1506 -2.0668,-0.3835 -4.3146,-0.5752 -6.7436,-0.5752 -2.4719,0 -4.4958,0.3408 -6.0727,1.0225 -1.5766,0.682 -2.823,1.6515 -3.7392,2.9086 -0.9161,1.2572 -1.5449,2.738 -1.8859,4.4425 -0.3407,1.7045 -0.5113,3.5796 -0.5113,5.625 l 0,16.7474 7.8624,0 0,-15.7246 c 0,-2.77 0.3302,-4.826 0.9907,-6.1685 0.6605,-1.3422 1.9497,-2.0134 3.8673,-2.0134 1.1931,0 2.3862,0.1066 3.5796,0.3194 l 0,23.5871 7.9263,0 0,-29.0202 z"
id="path256"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 355.7095,1069.4248 c 0,2.8125 -0.4581,4.9646 -1.3743,6.4559 -0.9161,1.4916 -2.3119,2.2374 -4.1867,2.2374 -0.8524,0 -1.694,-0.1174 -2.5248,-0.3518 -0.8312,-0.2341 -1.5449,-0.5219 -2.1416,-0.8629 l 0,-16.2997 c 0.5967,-0.1279 1.2251,-0.2132 1.8859,-0.2557 0.6604,-0.043 1.2251,-0.064 1.6937,-0.064 2.0454,0 3.6649,0.6925 4.858,2.0775 1.1931,1.385 1.7898,3.7395 1.7898,7.0634 z m 7.9903,-0.1919 c 0,-2.3862 -0.3198,-4.5491 -0.959,-6.488 -0.6392,-1.9389 -1.5766,-3.59 -2.8125,-4.9538 -1.2359,-1.3637 -2.7485,-2.4182 -4.5383,-3.164 -1.7898,-0.7458 -3.8353,-1.1186 -6.1364,-1.1186 -2.0455,0 -4.1338,0.1491 -6.2643,0.4473 -2.1308,0.2982 -3.9206,0.6818 -5.3694,1.1506 l 0,42.124 7.8622,1.3422 0,-15.0214 c 1.108,0.5114 2.1733,0.8629 3.196,1.0548 1.0228,0.1916 2.0455,0.2877 3.0683,0.2877 1.9604,0 3.6861,-0.373 5.1777,-1.1188 1.4913,-0.7458 2.738,-1.8111 3.7392,-3.1961 1.0015,-1.3847 1.7581,-3.0362 2.2694,-4.9538 0.5114,-1.9176 0.7671,-4.0481 0.7671,-6.3921"
id="path258"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 367.5986,1068.9772 c 0,2.6422 0.405,4.9646 1.2146,6.9676 0.8096,2.0026 1.8751,3.6754 3.1961,5.0176 1.3209,1.3425 2.8338,2.3544 4.5382,3.0364 1.7048,0.6818 3.4518,1.0228 5.2416,1.0228 4.4319,0 7.7984,-1.3105 10.0995,-3.9314 2.3012,-2.6206 3.4518,-6.4451 3.4518,-11.4738 0,-0.5114 -0.0108,-1.044 -0.0321,-1.5979 -0.0212,-0.5542 -0.0532,-1.0015 -0.0958,-1.3425 l -19.4958,0 c 0,-1.9601 0.8096,-3.505 2.429,-4.634 1.6192,-1.1294 3.7074,-1.694 6.2643,-1.694 1.5766,0 3.0787,0.1703 4.5062,0.5113 1.4278,0.341 2.6314,0.6818 3.6116,1.0228 l 1.0868,-6.7119 c -1.3637,-0.4686 -2.8125,-0.8629 -4.3466,-1.1823 -1.5341,-0.3198 -3.2601,-0.4794 -5.1778,-0.4794 -2.5568,0 -4.8472,0.3303 -6.8714,0.9907 -2.0242,0.6605 -3.75,1.6407 -5.1775,2.9404 -1.4275,1.2997 -2.5251,2.9084 -3.2922,4.826 -0.767,1.9176 -1.1505,4.1547 -1.1505,6.7116 z m 20.1353,3.1323 c 0,0.8096 -0.1066,1.5874 -0.3198,2.3329 -0.2131,0.7458 -0.5538,1.417 -1.0227,2.0137 -0.4686,0.5964 -1.0653,1.0761 -1.7898,1.438 -0.7242,0.3623 -1.6191,0.5434 -2.6847,0.5434 -1.0227,0 -1.9069,-0.1703 -2.6526,-0.5113 -0.7458,-0.341 -1.3638,-0.8096 -1.8536,-1.4063 -0.4901,-0.5967 -0.8737,-1.2784 -1.1506,-2.0455 -0.2772,-0.767 -0.4581,-1.5553 -0.5434,-2.3649 l 12.0172,0 z"
id="path260"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 425.5109,1076.2642 c -0.5113,0.1279 -1.1185,0.2557 -1.8218,0.3835 -0.703,0.1279 -1.4168,0.2452 -2.1413,0.3515 -0.7245,0.1066 -1.4275,0.1812 -2.1092,0.224 -0.6821,0.043 -1.2572,0.064 -1.7261,0.064 -1.1077,0 -2.1945,-0.053 -3.2598,-0.1599 -1.0656,-0.1063 -2.1521,-0.309 -3.2601,-0.6072 l 0,-22.3084 -7.9263,0 0,28.0613 c 2.0883,0.7671 4.2296,1.385 6.4242,1.8539 2.1946,0.4685 4.7622,0.703 7.7026,0.703 0.426,0 1.0332,-0.021 1.8215,-0.064 0.7886,-0.043 1.6302,-0.1174 2.5251,-0.2239 0.8949,-0.1066 1.8003,-0.2345 2.7165,-0.3836 0.9161,-0.1491 1.7365,-0.3515 2.461,-0.6072 l -1.4063,-7.287 z"
id="path262"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 433.5647,1083.2956 c 1.4063,0.3835 3.1428,0.7458 5.2098,1.0868 2.0668,0.3407 4.3146,0.5113 6.7436,0.5113 2.3862,0 4.3679,-0.3305 5.9446,-0.991 1.5766,-0.6604 2.8233,-1.5978 3.7394,-2.8125 0.9162,-1.2144 1.5662,-2.6631 1.9497,-4.3466 0.3835,-1.6833 0.5752,-3.5476 0.5752,-5.5931 l 0,-16.939 -7.8622,0 0,15.9163 c 0,2.8125 -0.3198,4.8367 -0.959,6.0726 -0.6392,1.2357 -1.9389,1.8536 -3.899,1.8536 -0.5967,0 -1.1826,-0.021 -1.7578,-0.064 -0.5754,-0.043 -1.1826,-0.1065 -1.8218,-0.1919 l 0,-23.5868 -7.8625,0 0,29.0841 z"
id="path264"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 463.4796,1068.9772 c 0,2.6422 0.4051,4.9646 1.2146,6.9676 0.8096,2.0026 1.8752,3.6754 3.1961,5.0176 1.3209,1.3425 2.8338,2.3544 4.5383,3.0364 1.7047,0.6818 3.4517,1.0228 5.2415,1.0228 4.432,0 7.7984,-1.3105 10.0996,-3.9314 2.3011,-2.6206 3.4517,-6.4451 3.4517,-11.4738 0,-0.5114 -0.0108,-1.044 -0.032,-1.5979 -0.0213,-0.5542 -0.0533,-1.0015 -0.0958,-1.3425 l -19.4959,0 c 0,-1.9601 0.8096,-3.505 2.429,-4.634 1.6192,-1.1294 3.7075,-1.694 6.2643,-1.694 1.5767,0 3.0787,0.1703 4.5063,0.5113 1.4278,0.341 2.6314,0.6818 3.6116,1.0228 l 1.0868,-6.7119 c -1.3638,-0.4686 -2.8126,-0.8629 -4.3467,-1.1823 -1.5341,-0.3198 -3.2601,-0.4794 -5.1777,-0.4794 -2.5569,0 -4.8473,0.3303 -6.8715,0.9907 -2.0242,0.6605 -3.7499,1.6407 -5.1775,2.9404 -1.4275,1.2997 -2.5251,2.9084 -3.2921,4.826 -0.7671,1.9176 -1.1506,4.1547 -1.1506,6.7116 z m 20.1353,3.1323 c 0,0.8096 -0.1066,1.5874 -0.3197,2.3329 -0.2132,0.7458 -0.5539,1.417 -1.0228,2.0137 -0.4685,0.5964 -1.0652,1.0761 -1.7898,1.438 -0.7242,0.3623 -1.6191,0.5434 -2.6846,0.5434 -1.0228,0 -1.9069,-0.1703 -2.6527,-0.5113 -0.7458,-0.341 -1.3638,-0.8096 -1.8536,-1.4063 -0.4901,-0.5967 -0.8736,-1.2784 -1.1506,-2.0455 -0.2772,-0.767 -0.458,-1.5553 -0.5434,-2.3649 l 12.0172,0 z"
id="path266"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 502.5991,1077.7346 -6.8394,0 0,6.5199 6.8394,0 0,7.5424 7.8625,1.2785 0,-8.8209 12.5923,0 0,-6.5199 -12.5923,0 0,-12.1451 c 0,-1.1081 0.1066,-2.0029 0.3194,-2.6847 0.2132,-0.6817 0.5114,-1.2146 0.8949,-1.5982 0.3836,-0.3835 0.8524,-0.6392 1.4063,-0.767 0.5542,-0.1279 1.1718,-0.1916 1.8539,-0.1916 0.7242,0 1.3955,0.021 2.0134,0.064 0.618,0.043 1.2144,0.1174 1.7898,0.224 0.5752,0.1063 1.1718,0.2662 1.7898,0.4793 0.618,0.2132 1.2889,0.4901 2.0135,0.8309 l 1.0868,-6.7757 c -1.4491,-0.5964 -3.0153,-1.0225 -4.6985,-1.2781 -1.6832,-0.2557 -3.3131,-0.3836 -4.8897,-0.3836 -1.8326,0 -3.4518,0.1491 -4.8581,0.4473 -1.4062,0.2982 -2.5996,0.8737 -3.5795,1.7258 -0.9803,0.8524 -1.7261,2.0562 -2.2374,3.6116 -0.5114,1.5554 -0.7671,3.5904 -0.7671,6.1044 l 0,12.337 z"
id="path268"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 527.4003,1068.9772 c 0,2.6422 0.4051,4.9646 1.2146,6.9676 0.8096,2.0026 1.8751,3.6754 3.1961,5.0176 1.3209,1.3425 2.8338,2.3544 4.5383,3.0364 1.7047,0.6818 3.4517,1.0228 5.2415,1.0228 4.432,0 7.7984,-1.3105 10.0996,-3.9314 2.3011,-2.6206 3.4517,-6.4451 3.4517,-11.4738 0,-0.5114 -0.0108,-1.044 -0.032,-1.5979 -0.0213,-0.5542 -0.0533,-1.0015 -0.0958,-1.3425 l -19.4959,0 c 0,-1.9601 0.8096,-3.505 2.429,-4.634 1.6192,-1.1294 3.7075,-1.694 6.2643,-1.694 1.5766,0 3.0787,0.1703 4.5063,0.5113 1.4278,0.341 2.6314,0.6818 3.6116,1.0228 l 1.0868,-6.7119 c -1.3638,-0.4686 -2.8126,-0.8629 -4.3467,-1.1823 -1.5341,-0.3198 -3.2601,-0.4794 -5.1777,-0.4794 -2.5569,0 -4.8473,0.3303 -6.8715,0.9907 -2.0242,0.6605 -3.7499,1.6407 -5.1775,2.9404 -1.4275,1.2997 -2.5251,2.9084 -3.2921,4.826 -0.7671,1.9176 -1.1506,4.1547 -1.1506,6.7116 z m 20.1353,3.1323 c 0,0.8096 -0.1066,1.5874 -0.3197,2.3329 -0.2132,0.7458 -0.5539,1.417 -1.0228,2.0137 -0.4685,0.5964 -1.0652,1.0761 -1.7898,1.438 -0.7242,0.3623 -1.6191,0.5434 -2.6847,0.5434 -1.0227,0 -1.9068,-0.1703 -2.6526,-0.5113 -0.7458,-0.341 -1.3638,-0.8096 -1.8536,-1.4063 -0.4901,-0.5967 -0.8736,-1.2784 -1.1506,-2.0455 -0.2772,-0.767 -0.4581,-1.5553 -0.5434,-2.3649 l 12.0172,0 z"
id="path270"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /><path
d="m 571.9533,1060.1561 c 1.9176,0 3.3876,0.1386 4.4104,0.4155 1.0227,0.277 1.5341,0.8416 1.5341,1.694 0,0.5539 -0.1599,1.0227 -0.4793,1.4063 -0.3198,0.3835 -0.7671,0.7137 -1.3425,0.9907 -0.5752,0.2769 -1.2357,0.5434 -1.9814,0.7991 -0.7458,0.2557 -1.5234,0.5113 -2.3332,0.767 -1.1506,0.3407 -2.3225,0.735 -3.5156,1.1826 -1.1931,0.4473 -2.2799,1.012 -3.2601,1.6937 -0.9799,0.6821 -1.7898,1.5449 -2.429,2.5889 -0.6392,1.044 -0.9587,2.3545 -0.9587,3.9311 0,1.2784 0.2449,2.4823 0.735,3.6116 0.4901,1.1294 1.2572,2.1305 2.3012,3.0042 1.044,0.8736 2.3757,1.5554 3.9952,2.0455 1.6191,0.4901 3.5583,0.7353 5.8166,0.7353 1.9605,0 3.782,-0.1494 5.4652,-0.4476 1.6835,-0.2982 3.1428,-0.7246 4.3787,-1.2784 l -1.2143,-6.6479 c -0.7246,0.2132 -1.8539,0.5647 -3.388,1.0548 -1.5341,0.4901 -3.2599,0.735 -5.1775,0.735 -2.003,0 -3.3559,-0.2449 -4.0589,-0.735 -0.7033,-0.4901 -1.0548,-1.0122 -1.0548,-1.5661 0,-0.4689 0.1599,-0.8841 0.4793,-1.2464 0.3198,-0.3623 0.7458,-0.6925 1.2785,-0.9907 0.5326,-0.2985 1.1506,-0.5859 1.8538,-0.8632 0.703,-0.2769 1.4593,-0.5431 2.2692,-0.7988 1.1505,-0.3835 2.3437,-0.8098 3.5796,-1.2784 1.2356,-0.4689 2.3544,-1.0548 3.3556,-1.7578 1.0015,-0.7032 1.8218,-1.5981 2.461,-2.6847 0.6393,-1.0868 0.959,-2.4185 0.959,-3.9951 0,-1.2359 -0.2344,-2.4078 -0.7033,-3.5158 -0.4688,-1.1078 -1.2464,-2.0668 -2.3332,-2.8763 -1.0865,-0.8096 -2.5035,-1.4488 -4.2505,-1.9177 -1.7473,-0.4688 -3.8994,-0.703 -6.4562,-0.703 -2.5994,0 -4.8152,0.2982 -6.6478,0.8946 -1.8323,0.5967 -3.3024,1.1294 -4.4104,1.5982 l 1.2143,6.5838 c 1.4916,-0.5964 3.079,-1.1506 4.7622,-1.662 1.6832,-0.5114 3.3985,-0.767 5.1458,-0.767"
id="path272"
style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none" /></g></svg>
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

61
11_orchestration/kubernetes/advanced.md Normal file
View File

@@ -0,0 +1,61 @@
# Kubernetes 高级特性
掌握了 Kubernetes 的核心概念（Pod, Service, Deployment）后，我们需要了解更多高级特性以构建生产级应用。
## Helm - 包管理工具
[Helm](https://helm.sh/) 被称为 Kubernetes 的包管理器（类似于 Linux 的 apt/yum）。它将一组 Kubernetes 资源定义文件打包为一个 **Chart**。
* **安装应用**：`helm install my-release bitnami/mysql`
* **版本管理**：轻松回滚应用的发布版本。
* **模板化**：支持复杂的应用部署逻辑配置。
## Ingress - 服务的入口
Service 虽然提供了负载均衡，但通常是 4 层（TCP/UDP）。**Ingress** 提供了 7 层（HTTP/HTTPS）路由能力，充当集群的网关。
* **域名路由**：基于 Host 将请求转发不同服务 (api.example.com -> api-svc, web.example.com -> web-svc)。
* **路径路由**：基于 Path 将请求转发 (/api -> api-svc, / -> web-svc)。
* **SSL/TLS**：集中管理证书。
常见的 Ingress Controller有 Nginx Ingress Controller, Traefik, Istio Gateway 等。
## Persistent Volume (PV) 与 StorageClass
容器内的文件是临时的。对于有状态应用（如数据库），需要持久化存储。
* **PVC (Persistent Volume Claim)**：用户申请存储的声明。
* **PV (Persistent Volume)**：实际的存储资源（NFS, AWS EBS, Ceph 等）。
* **StorageClass**：定义存储类，支持动态创建 PV。
## Horizontal Pod Autoscaling (HPA)
HPA 根据 CPU 利用率或其他指标（如内存、自定义指标）自动扩缩 Deployment 或 ReplicaSet 中的 Pod 数量。
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
```
## ConfigMap 与 Secret
* **ConfigMap**：存储非机密的配置数据（配置文件、环境变量）。
* **Secret**：存储机密数据（密码、Token、证书），在 Etcd 中加密存储。
通过将配置与镜像分离，保证了容器的可移植性。

188
11_orchestration/kubernetes/concepts.md Normal file
View File

@@ -0,0 +1,188 @@
# 基本概念
![](../_images/kubernetes_design.jpg)
* 节点（`Node`）：一个节点是一个运行 Kubernetes 中的主机。
* 容器组（`Pod`）：一个 Pod 对应于由若干容器组成的一个容器组，同个组内的容器共享一个存储卷(volume)。
* 容器组生命周期（`pos-states`）：包含所有容器状态集合，包括容器组状态类型，容器组生命周期，事件，重启策略，以及 replication controllers。
* Replication Controllers：主要负责指定数量的 pod 在同一时间一起运行。
* 服务（`services`）：一个 Kubernetes 服务是容器组逻辑的高级抽象，同时也对外提供访问容器组的策略。
* 卷（`volumes`）：一个卷就是一个目录，容器对其有访问权限。
* 标签（`labels`）：标签是用来连接一组对象的，比如容器组。标签可以被用来组织和选择子对象。
* 接口权限（`accessing_the_api`）：端口，IP 地址和代理的防火墙规则。
* web 界面（`ux`）：用户可以通过 web 界面操作 Kubernetes。
* 命令行操作（`cli`）：`kubectl`命令。
## 节点
在 `Kubernetes` 中，节点是实际工作的点，节点可以是虚拟机或者物理机器，依赖于一个集群环境。每个节点都有一些必要的服务以运行容器组，并且它们都可以通过主节点来管理。必要服务包括 Docker，kubelet 和代理服务。
### 容器状态
容器状态用来描述节点的当前状态。现在，其中包含三个信息：
#### 主机IP
主机 IP 需要云平台来查询，`Kubernetes` 把它作为状态的一部分来保存。如果 `Kubernetes` 没有运行在云平台上，节点 ID 就是必需的。IP 地址可以变化，并且可以包含多种类型的 IP 地址，如公共 IP，私有 IP，动态 IP，ipv6 等等。
#### 节点周期
通常来说节点有 `Pending`，`Running`，`Terminated` 三个周期，如果 Kubernetes 发现了一个节点并且其可用，那么 Kubernetes 就把它标记为 `Pending`。然后在某个时刻，Kubernetes 将会标记其为 `Running`。节点的结束周期称为 `Terminated`。一个已经 `Terminated` 的节点不会接受和调度任何请求，并且已经在其上运行的容器组也会删除。
#### 节点状态
节点的状态主要是用来描述处于 `Running` 的节点。当前可用的有 `NodeReachable` 和 `NodeReady`。以后可能会增加其他状态。`NodeReachable` 表示集群可达。`NodeReady` 表示 kubelet 返回 Status Ok 并且 HTTP 状态检查健康。
### 节点管理
节点并非 Kubernetes 创建，而是由云平台创建，或者就是物理机器、虚拟机。在 Kubernetes 中，节点仅仅是一条记录，节点创建之后，Kubernetes 会检查其是否可用。在 Kubernetes 中，节点用如下结构保存：
```json
{
"id": "10.1.2.3",
"kind": "Minion",
"apiVersion": "v1beta1",
"resources": {
"capacity": {
"cpu": 1000,
"memory": 1073741824
},
},
"labels": {
"name": "my-first-k8s-node",
},
}
```
Kubernetes 校验节点可用依赖于 ID。在当前的版本中，有两个接口可以用来管理节点：节点控制和 Kube 管理。
### 节点控制
在 Kubernetes 主节点中，节点控制器是用来管理节点的组件。主要包含：
* 集群范围内节点同步
* 单节点生命周期管理
节点控制有一个同步轮询，主要监听所有云平台的虚拟实例，会根据节点状态创建和删除。可以通过 `--node_sync_period`标志来控制该轮询。如果一个实例已经创建，节点控制将会为其创建一个结构。同样的，如果一个节点被删除，节点控制也会删除该结构。在 Kubernetes 启动时可用通过 `--machines`标记来显示指定节点。同样可以使用 `kubectl` 来一条一条的添加节点，两者是相同的。通过设置 `--sync_nodes=false`标记来禁止集群之间的节点同步，你也可以使用 api/kubectl 命令行来增删节点。
## 容器组
在 Kubernetes 中，使用的最小单位是容器组，容器组是创建，调度，管理的最小单位。 一个容器组使用相同的 Docker 容器并共享卷（挂载点）。一个容器组是一个特定应用的打包集合，包含一个或多个容器。
和运行的容器类似，一个容器组被认为只有很短的运行周期。容器组被调度到一组节点运行，直到容器的生命周期结束或者其被删除。如果节点死掉，运行在其上的容器组将会被删除而不是重新调度。（也许在将来的版本中会添加容器组的移动）。
### 容器组设计的初衷
### 资源共享和通信
容器组主要是为了数据共享和它们之间的通信。
在一个容器组中，容器都使用相同的网络地址和端口，可以通过本地网络来相互通信。每个容器组都有独立的 IP，可用通过网络来和其他物理主机或者容器通信。
容器组有一组存储卷（挂载点），主要是为了让容器在重启之后可以不丢失数据。
### 容器组管理
容器组是一个应用管理和部署的高层次抽象，同时也是一组容器的接口。容器组是部署、水平放缩的最小单位。
### 容器组的使用
容器组可以通过组合来构建复杂的应用，其本来的意义包含：
* 内容管理，文件和数据加载以及本地缓存管理等。
* 日志和检查点备份，压缩，快照等。
* 监听数据变化，跟踪日志，日志和监控代理，消息发布等。
* 代理，网桥
* 控制器，管理，配置以及更新
### 替代方案
为什么不在一个单一的容器里运行多个程序？
* 1.透明化。为了使容器组中的容器保持一致的基础设施和服务，比如进程管理和资源监控。这样设计是为了用户的便利性。
* 2.解偶软件之间的依赖。每个容器都可能重新构建和发布，Kubernetes 必须支持热发布和热更新（将来）。
* 3.方便使用。用户不必运行独立的程序管理，也不用担心每个应用程序的退出状态。
* 4.高效。考虑到基础设施有更多的职责，容器必须要轻量化。
### 容器组的生命状态
包括若干状态值：`pending`、`running`、`succeeded`、`failed`。
#### pending
容器组已经被节点接受，但有一个或多个容器还没有运行起来。这将包含某些节点正在下载镜像的时间，这种情形会依赖于网络情况。
#### running
容器组已经被调度到节点，并且所有的容器都已经启动。至少有一个容器处于运行状态（或者处于重启状态）。
#### succeeded
所有的容器都正常退出。
#### failed
容器组中所有容器都意外中断了。
### 容器组生命周期
通常来说，如果容器组被创建了就不会自动销毁，除非被某种行为触发，而触发此种情况可能是人为，或者复制控制器所为。唯一例外的是容器组由 succeeded 状态成功退出，或者在一定时间内重试多次依然失败。
如果某个节点死掉或者不能连接，那么节点控制器将会标记其上的容器组的状态为 `failed`。
举例如下。
* 容器组状态 `running`，有 1 容器，容器正常退出
* 记录完成事件
* 如果重启策略为：
* 始终：重启容器，容器组保持 `running`
* 失败时：容器组变为 `succeeded`
* 从不：容器组变为 `succeeded`
* 容器组状态 `running`，有1容器，容器异常退出
* 记录失败事件
* 如果重启策略为：
* 始终：重启容器，容器组保持 `running`
* 失败时：重启容器，容器组保持 `running`
* 从不：容器组变为 `failed`
* 容器组状态 `running`，有2容器，有1容器异常退出
* 记录失败事件
* 如果重启策略为：
* 始终：重启容器，容器组保持 `running`
* 失败时：重启容器，容器组保持 `running`
* 从不：容器组保持 `running`
* 当有2容器退出
* 记录失败事件
* 如果重启策略为：
* 始终：重启容器，容器组保持 `running`
* 失败时：重启容器，容器组保持 `running`
* 从不：容器组变为 `failed`
* 容器组状态 `running`，容器内存不足
* 标记容器错误中断
* 记录内存不足事件
* 如果重启策略为：
* 始终：重启容器，容器组保持 `running`
* 失败时：重启容器，容器组保持 `running`
* 从不：记录错误事件，容器组变为 `failed`
* 容器组状态 `running`，一块磁盘死掉
* 杀死所有容器
* 记录事件
* 容器组变为 `failed`
* 如果容器组运行在一个控制器下，容器组将会在其他地方重新创建
* 容器组状态 `running`，对应的节点段溢出
* 节点控制器等到超时
* 节点控制器标记容器组 `failed`
* 如果容器组运行在一个控制器下，容器组将会在其他地方重新创建
## Replication Controllers
## 服务
## 卷
## 标签
## 接口权限
## web界面
## 命令行操作

49
11_orchestration/kubernetes/design.md Normal file
View File

@@ -0,0 +1,49 @@
# 架构设计
任何优秀的项目都离不开优秀的架构设计。本小节将介绍 Kubernetes 在架构方面的设计考虑。
## 基本考虑
如果让我们自己从头设计一套容器管理平台，有如下几个方面是很容易想到的：
* 分布式架构，保证扩展性；
* 逻辑集中式的控制平面 + 物理分布式的运行平面；
* 一套资源调度系统，管理哪个容器该分配到哪个节点上；
* 一套对容器内服务进行抽象和 HA 的系统。
## 运行原理
下面这张图完整展示了 Kubernetes 的运行原理。
![Kubernetes 架构](../_images/k8s_architecture.png)
可见，Kubernetes 首先是一套分布式系统，由多个节点组成，节点分为两类：一类是属于管理平面的主节点/控制节点（Master Node）；一类是属于运行平面的工作节点（Worker Node）。
显然，复杂的工作肯定都交给控制节点去做了，工作节点负责提供稳定的操作接口和能力抽象即可。
从这张图上，我们没有能发现 Kubernetes 中对于控制平面的分布式实现，但是由于数据后端自身就是一套分布式的数据库 Etcd，因此可以很容易扩展到分布式实现。
## 控制平面
### 主节点服务
主节点上需要提供如下的管理服务：
* `apiserver` 是整个系统的对外接口，提供一套 RESTful 的 [Kubernetes API](https://kubernetes.io/zh/docs/concepts/overview/kubernetes-api/)，供客户端和其它组件调用；
* `scheduler` 负责对资源进行调度，分配某个 pod 到某个节点上。是 pluggable 的，意味着很容易选择其它实现方式；
* `controller-manager` 负责管理控制器，包括 endpoint-controller（刷新服务和 pod 的关联信息）和 replication-controller（维护某个 pod 的复制为配置的数值）。
### Etcd
这里 Etcd 即作为数据后端，又作为消息中间件。
通过 Etcd 来存储所有的主节点上的状态信息，很容易实现主节点的分布式扩展。
组件可以自动的去侦测 Etcd 中的数值变化来获得通知，并且获得更新后的数据来执行相应的操作。
## 工作节点
* kubelet 是工作节点执行操作的 agent，负责具体的容器生命周期管理，根据从数据库中获取的信息来管理容器，并上报 pod 运行状态等；
* kube-proxy 是一个简单的网络访问代理，同时也是一个 Load Balancer。它负责将访问到某个服务的请求具体分配给工作节点上的 Pod（同一类标签）。
![Proxy 代理对服务的请求](../_images/kube-proxy.png)

83
11_orchestration/kubernetes/intro.md Normal file
View File

@@ -0,0 +1,83 @@
# Kubernetes 简介
![](../_images/kubernetes_logo.png)
## 什么是 Kubernetes
Kubernetes（常简称为 K8s）是 Google 开源的容器编排引擎。如果说 Docker 解决了"如何打包和运送集装箱"的问题，那么 Kubernetes 解决的就是"如何管理海量集装箱的调度、运行和维护"的问题。
它不仅仅是一个编排系统，更是一个**云原生应用操作系统**。
> **名字由来**：Kubernetes 在希腊语中意为"舵手"或"飞行员"。K8s 是因为 k 和 s 之间有 8 个字母。
---
## 为什么需要 Kubernetes
当我们在单机运行几个容器时，Docker Compose 就足够了。但在生产环境中，我们需要面对：
- **多主机调度**：容器应该运行在哪台机器上？
- **自动恢复**：容器崩溃了怎么办？节点挂了怎么办？
- **服务发现**：容器 IP 变了，其他服务怎么找到它？
- **负载均衡**：流量大了，如何分发给多个副本？
- **滚动更新**：如何不中断服务升级应用？
Kubernetes 完美解决了这些问题。
---
## 核心概念
### Pod (豆荚)
Kubernetes 的最小调度单位。一个 Pod 可以包含一个或多个紧密协作的容器（共享网络和存储）。就像豌豆荚里的豌豆一样。
### Node (节点)
运行 Pod 的物理机或虚拟机。
### Deployment (部署)
定义应用的期望状态（如：需要 3 个副本，镜像版本为 v1）。K8s 会持续确保当前状态符合期望状态。
### Service (服务)
定义一组 Pod 的访问策略。提供稳定的 Cluster IP 和 DNS 名称，负责负载均衡。
### Namespace (命名空间)
用于多租户资源隔离。
---
## Docker 用户如何过渡
如果你已经熟悉 Docker，学习 K8s 会很容易：
| Docker 概念 | Kubernetes 概念 | 说明 |
|------------|----------------|------|
| Container | Pod | K8s 增加了一层 Pod 包装 |
| Volume | PersistentVolume | K8s 的存储更加抽象和强大 |
| Network | Service/Ingress| K8s 的网络模型更扁平 |
| Compose | Deployment + Service | 声明式配置的理念是一致的 |
---
## 架构
Kubernetes 也是 C/S 架构，由 **Master (控制平面)** 和 **Worker (工作节点)** 组成：
- **Control Plane**：负责决策（API Server, Scheduler, Controller Manager, etcd）
- **Worker Node**：负责干活（Kubelet, Kube-proxy, Container Runtime）
---
## 学习建议
Kubernetes 的学习曲线较陡峭。建议的学习路径：
1. **理解基本概念**：Pod, Deployment, Service
2. **动手实践**：使用 Minikube 或 Kind 在本地搭建集群
3. **部署应用**：编写 YAML 部署一个无状态应用
4. **深入原理**：网络模型、存储机制、调度算法
---
## 延伸阅读
- [Minikube 安装](../kubernetes/setup/README.md)：本地体验 K8s
- [Kubernetes 官网](https://kubernetes.io/)：官方文档

99
11_orchestration/kubernetes/practice.md Normal file
View File

@@ -0,0 +1,99 @@
# Kubernetes 实战练习
本章将通过一个具体的案例：部署一个 Nginx 网站，并为其配置 Service 和 Ingress，来串联前面学到的知识。
## 目标
1. 部署一个 Nginx Deployment。
2. 创建一个 Service 暴露 Nginx。
3. （可选）通过 Ingress 访问服务。
## 步骤 1：创建 Deployment
创建一个名为 `nginx-deployment.yaml` 的文件：
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.24
ports:
- containerPort: 80
```
应用配置：
```bash
kubectl apply -f nginx-deployment.yaml
```
## 步骤 2：创建 Service
创建一个名为 `nginx-service.yaml` 的文件：
```yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
type: NodePort # 使用 NodePort 方便本地测试
```
应用配置：
```bash
kubectl apply -f nginx-service.yaml
```
查看分配的端口：
```bash
kubectl get svc nginx-service
```
如果输出端口是 `80:30080/TCP`，你可以通过 `http://<NodeIP>:30080` 访问 Nginx。
## 步骤 3：模拟滚动更新 (Rolling Update)
修改 `nginx-deployment.yaml`，将镜像版本改为 `nginx:latest`。
```bash
kubectl apply -f nginx-deployment.yaml
```
观察更新过程：
```bash
kubectl rollout status deployment/nginx-deployment
```
## 步骤 4：清理资源
练习结束后，记得清理资源：
```bash
kubectl delete -f nginx-service.yaml
kubectl delete -f nginx-deployment.yaml
```

11
11_orchestration/setup/README.md Normal file
View File

@@ -0,0 +1,11 @@
# 部署 Kubernetes
目前，Kubernetes 支持在多种环境下使用，包括本地主机（Ubuntu、Debian、CentOS、Fedora 等）、云服务（[腾讯云](https://cloud.tencent.com/act/cps/redirect?redirect=10058&cps_key=3a5255852d5db99dcd5da4c72f05df61)、[阿里云](https://www.aliyun.com/product/kubernetes?source=5176.11533457&userCode=8lx5zmtu&type=copy)、[百度云](https://cloud.baidu.com/product/cce.html) 等）。
你可以使用以下几种方式部署 Kubernetes：
* kubeadm
* docker-desktop
* k3s
接下来的小节会对以上几种方式进行详细介绍。

45
11_orchestration/setup/dashboard.md Normal file
View File

@@ -0,0 +1,45 @@
# Kubernetes Dashboard
[Kubernetes Dashboard](https://github.com/kubernetes/dashboard) 是基于网页的 Kubernetes 用户界面。
![](https://d33wubrfki0l68.cloudfront.net/349824f68836152722dab89465835e604719caea/6e0b7/images/docs/ui-dashboard.png)
## 部署
执行以下命令即可部署 Dashboard：
```bash
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml
```
## 访问
通过命令行代理访问，执行以下命令：
```bash
$ kubectl proxy
```
到 http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/ 即可访问。
## 登录
目前，Dashboard 仅支持使用 Bearer 令牌登录。下面教大家如何创建该令牌：
```bash
$ kubectl create sa dashboard-admin -n kube-system
$ kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
$ ADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}')
$ DASHBOARD_LOGIN_TOKEN=$(kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}')
echo ${DASHBOARD_LOGIN_TOKEN}
```
将结果粘贴到登录页面，即可登录。
## 参考文档
* [官方文档](https://kubernetes.io/zh/docs/tasks/access-application-cluster/web-ui-dashboard/)

19
11_orchestration/setup/docker-desktop.md Normal file
View File

@@ -0,0 +1,19 @@
# Docker Desktop 启用 Kubernetes
使用 Docker Desktop 可以很方便的启用 Kubernetes。
## 启用 Kubernetes
在 Docker Desktop 设置页面，点击 `Kubernetes`，选择 `Enable Kubernetes`，稍等片刻，看到左下方 `Kubernetes` 变为 `running`，Kubernetes 启动成功。
![](https://github.com/docker/docs/raw/main/assets/images/desktop/settings-kubernetes.png)
> 注意：Kubernetes 的镜像存储在 `registry.k8s.io`，如果国内网络无法直接访问，可以在 Docker Desktop 配置中的 `Docker Engine` 处配置镜像加速器，或者利用国内云服务商的镜像仓库手动拉取镜像并 retag。
## 测试
```bash
$ kubectl version
```
如果正常输出信息，则证明 Kubernetes 成功启动。

52
11_orchestration/setup/k3s.md Normal file
View File

@@ -0,0 +1,52 @@
# K3s - 轻量级 Kubernetes
[K3s](https://k3s.io/) 是一个轻量级的 Kubernetes 发行版，由 Rancher Labs 开发。它专为边缘计算、物联网、CI、ARM 等资源受限的环境设计。K3s 被打包为单个二进制文件，只有不到 100MB，但通过了 CNCF 的一致性测试。
## 核心特性
* **轻量级**：移除过时的、非必须的 Kubernetes 功能（如传统的云提供商插件），使用 SQLite 作为默认数据存储（也支持 Etcd/MySQL/Postgres）。
* **单一二进制**：所有组件（API Server, Controller Manager, Scheduler, Kubelet, Kube-proxy）打包在一个进程中运行。
* **开箱即用**：内置 Helm Controller、Traefik Ingress controller、ServiceLB、Local-Path-Provisioner。
* **安全**：默认启用安全配置，基于 TLS 通信。
## 安装
### 脚本安装（Linux）
K3s 提供了极为便捷的安装脚本：
```bash
curl -sfL https://get.k3s.io | sh -
```
安装完成后，K3s 会自动启动并配置好 `systemd` 服务。
### 查看状态
```bash
sudo k3s kubectl get nodes
```
输出类似：
```
NAME STATUS ROLES AGE VERSION
k3s-master Ready control-plane,master 1m v1.28.2+k3s1
```
## 快速使用
K3s 内置了 `kubectl` 命令（通过 `k3s kubectl` 调用），为了方便，通常会建立别名或配置 `KUBECONFIG`。
```bash
# 读取 K3s 的配置文件
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
# 现在可以直接使用 kubectl
kubectl get pods -A
```
## 清理卸载
```bash
/usr/local/bin/k3s-uninstall.sh
```

81
11_orchestration/setup/kind.md Normal file
View File

@@ -0,0 +1,81 @@
# Kind - Kubernetes IN Docker
[Kind](https://kind.sigs.k8s.io/) (Kubernetes in Docker) 是一个使用 Docker 容器作为节点运行本地 Kubernetes 集群的工具。主要用于测试 Kubernetes 本身，也非常适合本地开发和 CI 环境。
## 为什么选择 Kind
* **轻量便捷**：只要有 Docker 环境即可，无需额外虚拟机。
* **多集群支持**：可以轻松在本地启动多个集群。
* **多版本支持**：支持指定 Kubernetes 版本进行测试。
* **HA 支持**：支持模拟高可用集群（多 Control Plane）。
## 安装 Kind
### macOS
```bash
brew install kind
```
### Linux / Windows (WSL2)
可以下载二进制文件：
```bash
# Linux AMD64
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
```
## 创建集群
最简单的创建方式：
```bash
kind create cluster
```
指定集群名称：
```bash
kind create cluster --name my-cluster
```
## 与集群交互
Kind 会自动将 kubeconfig 合并到 `~/.kube/config`。
```bash
kubectl cluster-info --context kind-kind
kubectl get nodes
```
## 高级用法：配置集群
创建一个 `kind-config.yaml` 来定制集群，例如映射端口到宿主机：
```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 80
hostPort: 8080
protocol: TCP
- role: worker
- role: worker
```
应用配置：
```bash
kind create cluster --config kind-config.yaml
```
## 删除集群
```bash
kind delete cluster
```

187
11_orchestration/setup/kubeadm-docker.md Normal file
View File

@@ -0,0 +1,187 @@
# 使用 kubeadm 部署 kubernetes(使用 Docker)
`kubeadm` 提供了 `kubeadm init` 以及 `kubeadm join` 这两个命令作为快速创建 `kubernetes` 集群的最佳实践。
## 安装 Docker
参考 [安装 Docker](../../install) 一节安装 Docker。
## 安装 **kubelet** **kubeadm** **kubectl**
### Ubuntu/Debian
```bash
$ apt-get update && apt-get install -y apt-transport-https
$ curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
$ cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
$ apt-get update
$ apt-get install -y kubelet kubeadm kubectl
```
### CentOS/Fedora
```bash
$ cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
$ sudo yum install -y kubelet kubeadm kubectl
```
## 修改内核的运行参数
```bash
$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
# 应用配置
$ sysctl --system
```
## 配置 kubelet
### 修改 `kubelet.service`
`/etc/systemd/system/kubelet.service.d/10-proxy-ipvs.conf` 写入以下内容
```bash
# 启用 ipvs 相关内核模块
[Service]
ExecStartPre=-/sbin/modprobe ip_vs
ExecStartPre=-/sbin/modprobe ip_vs_rr
ExecStartPre=-/sbin/modprobe ip_vs_wrr
ExecStartPre=-/sbin/modprobe ip_vs_sh
```
执行以下命令应用配置。
```bash
$ sudo systemctl daemon-reload
```
## 部署
### master
```bash
$ sudo kubeadm init --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
--pod-network-cidr 10.244.0.0/16 \
--v 5 \
--ignore-preflight-errors=all
```
* `--pod-network-cidr 10.244.0.0/16` 参数与后续 CNI 插件有关，这里以 `flannel` 为例，若后续部署其他类型的网络插件请更改此参数。
> 执行可能出现错误，例如缺少依赖包，根据提示安装即可。
执行成功会输出
```bash
...
[addons] Applied essential addon: CoreDNS
I1116 12:35:13.270407 86677 request.go:538] Throttling request took 181.409184ms, request: POST:https://192.168.199.100:6443/api/v1/namespaces/kube-system/serviceaccounts
I1116 12:35:13.470292 86677 request.go:538] Throttling request took 186.088112ms, request: POST:https://192.168.199.100:6443/api/v1/namespaces/kube-system/configmaps
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.199.100:6443 --token cz81zt.orsy9gm9v649e5lf \
--discovery-token-ca-cert-hash sha256:5edb316fd0d8ea2792cba15cdf1c899a366f147aa03cba52d4e5c5884ad836fe
```
### node 工作节点
在 **另一主机** 重复 **部署** 小节以前的步骤，安装配置好 kubelet。根据提示，加入到集群。
```bash
$ kubeadm join 192.168.199.100:6443 --token cz81zt.orsy9gm9v649e5lf \
--discovery-token-ca-cert-hash sha256:5edb316fd0d8ea2792cba15cdf1c899a366f147aa03cba52d4e5c5884ad836fe
```
## 查看服务
所有服务启动后，查看本地实际运行的 Docker 容器。这些服务大概分为三类：主节点服务、工作节点服务和其它服务。
### 主节点服务
* `apiserver` 是整个系统的对外接口，提供 RESTful 方式供客户端和其它组件调用；
* `scheduler` 负责对资源进行调度，分配某个 pod 到某个节点上；
* `controller-manager` 负责管理控制器，包括 endpoint-controller（刷新服务和 pod 的关联信息）和 replication-controller（维护某个 pod 的复制为配置的数值）。
### 工作节点服务
* `proxy` 为 pod 上的服务提供访问的代理。
### 其它服务
* Etcd 是所有状态的存储数据库；
## 使用
将 `/etc/kubernetes/admin.conf` 复制到 `~/.kube/config`
执行 `$ kubectl get all -A` 查看启动的服务。
由于未部署 CNI 插件，CoreDNS 未正常启动。如何使用 Kubernetes，请参考后续章节。
## 部署 CNI
这里以 `flannel` 为例进行介绍。
### flannel
检查 podCIDR 设置
```bash
$ kubectl get node -o yaml | grep CIDR
# 输出
podCIDR: 10.244.0.0/16
podCIDRs:
```
```bash
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.11.0/Documentation/kube-flannel.yml
```
## master 节点默认不能运行 pod
如果用 `kubeadm` 部署一个单节点集群，默认情况下无法使用，请执行以下命令解除限制
```bash
$ kubectl taint nodes --all node-role.kubernetes.io/master-
# 恢复默认值
# $ kubectl taint nodes NODE_NAME node-role.kubernetes.io/master=true:NoSchedule
```
## 参考文档
* [官方文档](https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)

399
11_orchestration/setup/kubeadm.md Normal file
View File

@@ -0,0 +1,399 @@
# 使用 kubeadm 部署 kubernetes(CRI 使用 containerd)
`kubeadm` 提供了 `kubeadm init` 以及 `kubeadm join` 这两个命令作为快速创建 `kubernetes` 集群的最佳实践。
> **版本说明**：Kubernetes 版本更新较快（约每 4 个月一个新版本），本文档基于 Kubernetes 1.35 编写。请访问 [Kubernetes 官方发布页](https://kubernetes.io/releases/) 获取最新版本信息。
## 安装 containerd
参考 [安装 Docker](../../install) 一节添加 apt/yum 源，之后执行如下命令。
```bash
# debian 系
$ sudo apt install containerd.io
# rhel 系
$ sudo yum install containerd.io
```
## 配置 containerd
新建 `/etc/systemd/system/cri-containerd.service` 文件
```
[Unit]
Description=containerd container runtime for kubernetes
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd --config /etc/cri-containerd/config.toml
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
```
新建 `/etc/cri-containerd/config.toml` containerd 配置文件
```toml
version = 2
# persistent data location
root = "/var/lib/cri-containerd"
# runtime state information
state = "/run/cri-containerd"
plugin_dir = ""
disabled_plugins = []
required_plugins = []
# set containerd's OOM score
oom_score = 0
[grpc]
address = "/run/cri-containerd/cri-containerd.sock"
tcp_address = ""
tcp_tls_cert = ""
tcp_tls_key = ""
# socket uid
uid = 0
# socket gid
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
[debug]
address = ""
format = "json"
uid = 0
gid = 0
level = ""
[metrics]
address = "127.0.0.1:1338"
grpc_histogram = false
[cgroup]
path = ""
[timeouts]
"io.containerd.timeout.shim.cleanup" = "5s"
"io.containerd.timeout.shim.load" = "5s"
"io.containerd.timeout.shim.shutdown" = "3s"
"io.containerd.timeout.task.state" = "2s"
[plugins]
[plugins."io.containerd.gc.v1.scheduler"]
pause_threshold = 0.02
deletion_threshold = 0
mutation_threshold = 100
schedule_delay = "0s"
startup_delay = "100ms"
[plugins."io.containerd.grpc.v1.cri"]
disable_tcp_service = true
stream_server_address = "127.0.0.1"
stream_server_port = "0"
stream_idle_timeout = "4h0m0s"
enable_selinux = false
selinux_category_range = 1024
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10"
stats_collect_period = 10
# systemd_cgroup = false
enable_tls_streaming = false
max_container_log_line_size = 16384
disable_cgroup = false
disable_apparmor = false
restrict_oom_score_adj = false
max_concurrent_downloads = 3
disable_proc_mount = false
unset_seccomp_profile = ""
tolerate_missing_hugetlb_controller = true
disable_hugetlb_controller = true
ignore_image_defined_volumes = false
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "overlayfs"
default_runtime_name = "runc"
no_pivot = false
disable_snapshot_annotations = false
discard_unpacked_layers = false
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
pod_annotations = []
container_annotations = []
privileged_without_host_devices = false
base_runtime_spec = ""
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
# SystemdCgroup enables systemd cgroups.
SystemdCgroup = true
# BinaryName is the binary name of the runc binary.
# BinaryName = "runc"
# BinaryName = "crun"
# NoPivotRoot disables pivot root when creating a container.
# NoPivotRoot = false
# NoNewKeyring disables new keyring for the container.
# NoNewKeyring = false
# ShimCgroup places the shim in a cgroup.
# ShimCgroup = ""
# IoUid sets the I/O's pipes uid.
# IoUid = 0
# IoGid sets the I/O's pipes gid.
# IoGid = 0
# Root is the runc root directory.
Root = ""
# CriuPath is the criu binary path.
# CriuPath = ""
# CriuImagePath is the criu image path
# CriuImagePath = ""
# CriuWorkPath is the criu work path.
# CriuWorkPath = ""
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
max_conf_num = 1
conf_template = ""
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/cri-containerd/certs.d"
[plugins."io.containerd.grpc.v1.cri".registry.headers]
# Foo = ["bar"]
[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = ""
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
tls_cert_file = ""
tls_key_file = ""
[plugins."io.containerd.internal.v1.opt"]
path = "/opt/cri-containerd"
[plugins."io.containerd.internal.v1.restart"]
interval = "10s"
[plugins."io.containerd.metadata.v1.bolt"]
content_sharing_policy = "shared"
[plugins."io.containerd.monitor.v1.cgroups"]
no_prometheus = false
[plugins."io.containerd.runtime.v2.task"]
platforms = ["linux/amd64"]
[plugins."io.containerd.service.v1.diff-service"]
default = ["walking"]
[plugins."io.containerd.snapshotter.v1.devmapper"]
root_path = ""
pool_name = ""
base_image_size = ""
async_remove = false
```
## 安装 **kubelet** **kubeadm** **kubectl** **cri-tools** **kubernetes-cni**
### Ubuntu/Debian
```bash
$ apt-get update && apt-get install -y apt-transport-https
$ curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
$ cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
$ apt-get update
$ apt-get install -y kubelet kubeadm kubectl
```
### CentOS/Fedora
```bash
$ cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
$ sudo yum install -y kubelet kubeadm kubectl
```
## 修改内核的运行参数
```bash
$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
# 应用配置
$ sysctl --system
```
## 配置 kubelet
### 修改 `kubelet.service`
`/etc/systemd/system/kubelet.service.d/10-proxy-ipvs.conf` 写入以下内容
```bash
# 启用 ipvs 相关内核模块
[Service]
ExecStartPre=-/sbin/modprobe ip_vs
ExecStartPre=-/sbin/modprobe ip_vs_rr
ExecStartPre=-/sbin/modprobe ip_vs_wrr
ExecStartPre=-/sbin/modprobe ip_vs_sh
```
执行以下命令应用配置。
```bash
$ sudo systemctl daemon-reload
```
## 部署
### master
```bash
$ systemctl enable cri-containerd
$ systemctl start cri-containerd
$ sudo kubeadm init \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
--pod-network-cidr 10.244.0.0/16 \
--cri-socket /run/cri-containerd/cri-containerd.sock \
--v 5 \
--ignore-preflight-errors=all
```
* `--pod-network-cidr 10.244.0.0/16` 参数与后续 CNI 插件有关，这里以 `flannel` 为例，若后续部署其他类型的网络插件请更改此参数。
> 执行可能出现错误，例如缺少依赖包，根据提示安装即可。
执行成功会输出
```bash
...
[addons] Applied essential addon: CoreDNS
I1116 12:35:13.270407 86677 request.go:538] Throttling request took 181.409184ms, request: POST:https://192.168.199.100:6443/api/v1/namespaces/kube-system/serviceaccounts
I1116 12:35:13.470292 86677 request.go:538] Throttling request took 186.088112ms, request: POST:https://192.168.199.100:6443/api/v1/namespaces/kube-system/configmaps
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.199.100:6443 --token cz81zt.orsy9gm9v649e5lf \
--discovery-token-ca-cert-hash sha256:5edb316fd0d8ea2792cba15cdf1c899a366f147aa03cba52d4e5c5884ad836fe
```
### node 工作节点
在 **另一主机** 重复 **部署** 小节以前的步骤，安装配置好 kubelet。根据提示，加入到集群。
```bash
$ systemctl enable cri-containerd
$ systemctl start cri-containerd
$ kubeadm join 192.168.199.100:6443 \
--token cz81zt.orsy9gm9v649e5lf \
--discovery-token-ca-cert-hash sha256:5edb316fd0d8ea2792cba15cdf1c899a366f147aa03cba52d4e5c5884ad836fe \
--cri-socket /run/cri-containerd/cri-containerd.sock
```
## 查看服务
所有服务启动后，通过 `crictl` 查看本地实际运行的容器。这些服务大概分为三类：主节点服务、工作节点服务和其它服务。
```bash
CONTAINER_RUNTIME_ENDPOINT=/run/cri-containerd/cri-containerd.sock crictl ps -a
```
### 主节点服务
* `apiserver` 是整个系统的对外接口，提供 RESTful 方式供客户端和其它组件调用；
* `scheduler` 负责对资源进行调度，分配某个 pod 到某个节点上；
* `controller-manager` 负责管理控制器，包括 endpoint-controller（刷新服务和 pod 的关联信息）和 replication-controller（维护某个 pod 的复制为配置的数值）。
### 工作节点服务
* `proxy` 为 pod 上的服务提供访问的代理。
### 其它服务
* Etcd 是所有状态的存储数据库；
## 使用
将 `/etc/kubernetes/admin.conf` 复制到 `~/.kube/config`
执行 `$ kubectl get all -A` 查看启动的服务。
由于未部署 CNI 插件，CoreDNS 未正常启动。如何使用 Kubernetes，请参考后续章节。
## 部署 CNI
这里以 `flannel` 为例进行介绍。
### flannel
检查 podCIDR 设置
```bash
$ kubectl get node -o yaml | grep CIDR
# 输出
podCIDR: 10.244.0.0/16
podCIDRs:
```
```bash
$ kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/v0.26.1/Documentation/kube-flannel.yml
```
## master 节点默认不能运行 pod
如果用 `kubeadm` 部署一个单节点集群，默认情况下无法使用，请执行以下命令解除限制
```bash
$ kubectl taint nodes --all node-role.kubernetes.io/master-
# 恢复默认值
# $ kubectl taint nodes NODE_NAME node-role.kubernetes.io/master=true:NoSchedule
```
## 参考文档
* [官方文档](https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)
* [Container runtimes](https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd)

3
11_orchestration/setup/systemd.md Normal file
View File

@@ -0,0 +1,3 @@
# 一步步部署 kubernetes 集群
可以参考 [opsnull/follow-me-install-kubernetes-cluster](https://github.com/opsnull/follow-me-install-kubernetes-cluster) 项目一步步部署 kubernetes 集群。