harbor 能力测试

发表于 更新于 分类于 本文字数： 1k 阅读时长 ≈ 4 分钟

测试环境

测试机型：

harbor机型：I2 IO优化型， 4C8G
测试镜像大小： 1.7G
测试工具：harbor perftest

测试集群：

准备k8s集群，通过 k8s node 节点拉取harbor镜像,

准备测试镜像：

1
2
3
docker pull garima0079/bigimage
docker tag garima0079/bigimage {harbor LB地址}/bigimage
docker push {harbor LB地址}/bigimage

测试工具准备:

1
2
3
git clone git@github.com:seamounts/perftest.git -d $GOPATH/src
cd $GOPATH/src/perftest
go build

准备测试集群node docker daemon 配置:

批量更新 k8s node docker daemon，将 harbor lb地址加入到 insecure-registries 列表

1
$GOPATH/src/perftest/cmd/harbor/script/prepare.sh

清除镜像缓存:

批量清除 k8s node 节点镜像缓存

1
$GOPATH/src/perftest/cmd/harbor/script/clear-image-cache.sh {harbor LB地址}/bigimage

测试指标

镜像无缓存并发拉取大镜像的平均耗时，99值耗时

harbor 实例数和拉取时间的关系

harbor 支持的最大并发

无缓存拉取大镜像测试

2 HA harbor测试

测试环境：
harbor 2个实例，主机I2 IO优化型，4C8G，内网LB，内网KS3

测试数据：
执行： $GOPATH/src//perftest harbor pullimage –image 10.21.1.4/hsxue/bigimage –kubeconfig /root/.kube/config, 在集群所有节点并发拉取镜像。

50 node 并发拉取测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Summary:
  Total:    73.3745 secs.
  Slowest:  73.3734 secs.
  Fastest:  0.8629 secs.
  Average:  49.8971 secs.
 
Response time histogram:
  0.8629 [1]    |∎
  8.1140 [0]    |
  15.3650 [0]   |
  22.6161 [0]   |
  29.8671 [0]   |
  37.1182 [0]   |
  44.3692 [0]   |
  51.6203 [31]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  58.8713 [17]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  66.1224 [0]   |
  73.3734 [1]   |∎
 
Latency distribution:
  99% in 57.8476 secs.

100 node 并发拉取测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Summary:
  Total:    127.8106 secs.
  Slowest:  127.8100 secs.
  Fastest:  46.4622 secs.
  Average:  70.0965 secs.
 
Response time histogram:
  46.4622 [1]   |∎∎
  54.5969 [18]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  62.7317 [18]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  70.8665 [19]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  79.0013 [17]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  87.1361 [12]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  95.2709 [8]   |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  103.4056 [2]  |∎∎∎∎
  111.5404 [3]  |∎∎∎∎∎∎
  119.6752 [1]  |∎∎
  127.8100 [1]  |∎∎
 
Latency distribution:
  99% in 127.8100 secs.

200 node 并发拉取测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Summary:
  Total:    180.1791 secs.
  Slowest:  180.1777 secs.
  Fastest:  53.9234 secs.
  Average:  112.8468 secs.
 
Response time histogram:
  53.9234 [1]   |∎
  66.5489 [10]  |∎∎∎∎∎∎∎∎∎∎∎∎
  79.1743 [16]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  91.7997 [28]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  104.4252 [30] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  117.0506 [32] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  129.6760 [23] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  142.3015 [21] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  154.9269 [20] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  167.5523 [15] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  180.1777 [4]  |∎∎∎∎∎
 
Latency distribution:
  99% in 173.5462 secs.

300 node 并发拉取测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Summary:
  Total:    246.9771 secs.
  Slowest:  246.9760 secs.
  Fastest:  53.7463 secs.
  Average:  152.4312 secs.
 
Response time histogram:
  53.7463 [1]   |
  73.0693 [10]  |∎∎∎∎∎∎∎∎
  92.3923 [22]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  111.7152 [37] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  131.0382 [34] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  150.3612 [34] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  169.6841 [50] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  189.0071 [34] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  208.3301 [42] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  227.6530 [21] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  246.9760 [15] |∎∎∎∎∎∎∎∎∎∎∎∎
 
Latency distribution:
  99% in 237.6006 secs.

400 node 并发拉取测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Summary:
  Total:    297.7355 secs.
  Slowest:  297.7326 secs.
  Fastest:  60.6159 secs.
  Average:  187.1423 secs.
 
Response time histogram:
  60.6159 [1]   |
  84.3275 [7]   |∎∎∎∎∎
  108.0392 [37] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  131.7509 [41] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  155.4626 [40] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  179.1742 [53] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  202.8859 [54] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  226.5976 [53] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  250.3092 [45] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  274.0209 [36] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  297.7326 [31] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
 
Latency distribution:
  99% in 295.2425 secs.

4 HA harbor测试

测试环境：
harbor 4个实例，主机I2 IO优化型，4C8G，内网LB，内网KS3

测试数据：
执行： $GOPATH/src//perftest harbor pullimage –image 10.21.1.4/hsxue/bigimage –kubeconfig /root/.kube/config, 在集群所有节点并发拉取镜像。

50 node 并发拉取测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Summary:
  Total:    68.5314 secs.
  Slowest:  68.5308 secs.
  Fastest:  41.2531 secs.
  Average:  45.7705 secs.
 
Response time histogram:
  41.2531 [1]   |∎
  43.9809 [27]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  46.7086 [11]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  49.4364 [4]   |∎∎∎∎∎
  52.1642 [2]   |∎∎
  54.8919 [1]   |∎
  57.6197 [1]   |∎
  60.3475 [0]   |
  63.0752 [0]   |
  65.8030 [1]   |∎
  68.5308 [1]   |∎
 
Latency distribution:
  99% in 64.3012 secs.

100 node 并发拉取测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Summary:
  Total:    115.8375 secs.
  Slowest:  115.8369 secs.
  Fastest:  46.4803 secs.
  Average:  70.5216 secs.
 
Response time histogram:
  46.4803 [1]   |∎∎
  53.4160 [20]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  60.3516 [12]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  67.2873 [12]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  74.2229 [15]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  81.1586 [11]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  88.0943 [11]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  95.0299 [8]   |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  101.9656 [7]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  108.9012 [0]  |
  115.8369 [3]  |∎∎∎∎∎∎
 
Latency distribution:
  99% in 115.8369 secs.

300 node 并发拉取测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

Summary:
  Total:    240.1224 secs.
  Slowest:  240.1149 secs.
  Fastest:  53.0441 secs.
  Average:  138.6291 secs.
 
Response time histogram:
  53.0441 [1]   |
  71.7512 [15]  |∎∎∎∎∎∎∎∎∎∎
  90.4583 [27]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  109.1654 [40] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  127.8724 [55] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  146.5795 [38] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  165.2866 [32] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  183.9937 [41] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  202.7008 [27] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  221.4078 [18] |∎∎∎∎∎∎∎∎∎∎∎∎∎
  240.1149 [6]  |∎∎∎∎
 
Latency distribution:
  99% in 237.0826 secs.

harbor 性能总结

2 HA harbor 性能

2.1 并发拉取镜像数和拉取延迟关系

1

拉取 1.7G 大镜像所需要的时间和并发拉取node的数量基本成线性正比的关系，并发度每增加100，拉取时间就相应增加一分钟。

2.2 并发拉取镜像数和cpu 使用率关系

2

并发拉取量每增加 100，cpu 使用率就增加 10%，当并发达到 500 时，cpu使用率为50%左右 。可知当拉取镜像的并发数达到500以上时，对 harbor 的压力会比较大。

推荐增加harbor 实例个数来缓解压力。

2 HA harbor 和 4HA harbpr 拉取镜像平均时间对比

50 nodes 100 nodes 300 nodes
2HA harbor 49.8971 s 70.0965 s 152.4312 s
4HA harbor 45.7705 s 70.5216 s 138.6291 s

增加 harbor 实例个数，拉取镜像所花费时间没有明显的的减少。时间增加应该和后端存储的性能和网络带宽有关。