Client-go 的 UpdateStatus 接口你用对了吗?

最近发现一个问题,发现使用 client-go 并行调用 UpdateStatus 接口与 PATCH 更新 annotation 会出现覆盖写的问题,这与之前的理解不一致,这里做一个记录。

先说结论:client-go 的 UpdateStatus 会 PUT 更新 Pod 的 Status 字段与 metadata 字段。

问题现象

使用 client-go 在两个协程并行调用 UpdateStatus 接口与 PATCH 接口更新 annotation,发现 annotation 更新失效。

问题分析

通过打印 client-go 的 response body 信息(klog 的 flag 设置 v=8 以上),发现两个协程先调用 PATCH 接口更新 annotation 成功,资源的 ResourceVersion 为1,但紧接着马上发出了一个 PUT status 的接口,资源的 ResourceVersion 更新为2,此时的 annotation 为老版本的 annotation,导致 PATCH 接口的更新失效了。

问题根因

client-go 的 UpdateStatus 接口本质为调用 APIServer 的 PUT {resource}/status 接口,按照官方 API 文档的说法,Update 接口不会更新 status,所以额外提供了 PUT {resource}/status 接口。

The PUT and POST verbs on objects MUST ignore the “status” values, to avoid accidentally overwriting the status in read-modify-write scenarios. A /status subresource MUST be provided to enable system components to update statuses of resources they manage.

查看 APIServer 代码的具体实现,

pkg/registry/core/pod/storage/storage.go 为初始化 Pod 相关接口的 handler

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
func NewStorage(optsGetter generic.RESTOptionsGetter, k client.ConnectionInfoGetter, proxyTransport http.RoundTripper, podDisruptionBudgetClient policyclient.PodDisruptionBudgetsGetter) (PodStorage, error) {

store := &genericregistry.Store{
NewFunc: func() runtime.Object { return &api.Pod{} },
NewListFunc: func() runtime.Object { return &api.PodList{} },
PredicateFunc: registrypod.MatchPod,
DefaultQualifiedResource: api.Resource("pods"),

CreateStrategy: registrypod.Strategy,
UpdateStrategy: registrypod.Strategy,
DeleteStrategy: registrypod.Strategy,
ReturnDeletedObject: true,

TableConvertor: printerstorage.TableConvertor{TableGenerator: printers.NewTableGenerator().With(printersinternal.AddHandlers)},
}
options := &generic.StoreOptions{
RESTOptions: optsGetter,
AttrFunc: registrypod.GetAttrs,
TriggerFunc: map[string]storage.IndexerFunc{"spec.nodeName": registrypod.NodeNameTriggerFunc},
Indexers: registrypod.Indexers(),
}
if err := store.CompleteWithOptions(options); err != nil {
return PodStorage{}, err
}

statusStore := *store
statusStore.UpdateStrategy = registrypod.StatusStrategy
ephemeralContainersStore := *store
ephemeralContainersStore.UpdateStrategy = registrypod.EphemeralContainersStrategy

bindingREST := &BindingREST{store: store}
return PodStorage{
Pod: &REST{store, proxyTransport},
Binding: &BindingREST{store: store},
LegacyBinding: &LegacyBindingREST{bindingREST},
Eviction: newEvictionStorage(store, podDisruptionBudgetClient),
Status: &StatusREST{store: &statusStore},
EphemeralContainers: &EphemeralContainersREST{store: &ephemeralContainersStore},
Log: &podrest.LogREST{Store: store, KubeletConn: k},
Proxy: &podrest.ProxyREST{Store: store, ProxyTransport: proxyTransport},
Exec: &podrest.ExecREST{Store: store, KubeletConn: k},
Attach: &podrest.AttachREST{Store: store, KubeletConn: k},
PortForward: &podrest.PortForwardREST{Store: store, KubeletConn: k},
}, nil
}

而 StatusUpdate 接口与普通 Update 接口的区别主要是 statusStore.UpdateStrategy = registrypod.StatusStrategy

深入观察两种策略的不同,发现 Update 使用的 registrypod.Strategy 会使用老 Pod 的 Status。

1
2
3
4
5
6
7
8
// PrepareForUpdate clears fields that are not allowed to be set by end users on update.
func (podStrategy) PrepareForUpdate(ctx context.Context, obj, old runtime.Object) {
newPod := obj.(*api.Pod)
oldPod := old.(*api.Pod)
newPod.Status = oldPod.Status

podutil.DropDisabledPodFields(newPod, oldPod)
}

StatusUpdate 接口使用的 registrypod.StatusStrategy 策略会使用老 Pod 的 Spec。

1
2
3
4
5
6
7
8
9
10
func (podStatusStrategy) PrepareForUpdate(ctx context.Context, obj, old runtime.Object) {
newPod := obj.(*api.Pod)
oldPod := old.(*api.Pod)
newPod.Spec = oldPod.Spec
newPod.DeletionTimestamp = nil

// don't allow the pods/status endpoint to touch owner references since old kubelets corrupt them in a way
// that breaks garbage collection
newPod.OwnerReferences = oldPod.OwnerReferences
}

而出问题的 annotation 即不属于 Spec,也不属于 Status,故两个操作会产生协程冲突。

解决方案

将 annotation 更新的 Pod 接口与 UpdateStatus 接口整合。