大网数据广告活动
服务器运维问答

Letting systemd service to stop later without blocking on `systemctl stop`

· 52CE 我爱测

I have bunch of services responsible for running action consumed from the queue. I want to be able to restart services gently (without interrupting action which're already running) It can be solved with handling SIGTERM sent by s…

问题原文

原帖链接 · Jakub Kuszneruk · CC BY-SA 4.0

I have bunch of services responsible for running action consumed from the queue.

I want to be able to restart services gently (without interrupting action which're already running)

It can be solved with handling SIGTERM sent by systemd and saving information that the program should exit after current action's processed.
There's other minor problem with the fact that after some time defined as TimeoutStopSec in service configuraiton file systemd will send additional SIGKILL to brutally terminate my process.
I can easily avoid it with setting TimeoutStopSec=infinity. Then systemctl stop'll wait until script terminates itself, which may last even more than an hour, and leads me to main problem.

I don't want systemctl command to wait until script will end

It looks like SendSIGKILL=no configuration does the job. This results with retrying SIGTERM after TimeoutStopSec, then creating new worker, and leaving the old one running.

journalctl log

May 06 14:14:43 jaku systemd[1]: Stopping Jaku test worker...
May 06 14:14:43 jaku python3[31597]: * 15 <frame object at 0x14d8108>
May 06 14:14:53 jaku systemd[1]: jaku-test-worker.service: State 'stop-sigterm' timed out. Skipping SIGKILL.
May 06 14:14:53 jaku python3[31597]: * 15 <frame object at 0x14d8108>
May 06 14:15:03 jaku systemd[1]: jaku-test-worker.service: State 'stop-final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.
May 06 14:15:03 jaku systemd[1]: jaku-test-worker.service: Failed with result 'timeout'.
May 06 14:15:03 jaku systemd[1]: Stopped Jaku test worker.
May 06 14:15:03 jaku systemd[1]: jaku-test-worker.service: Found left-over process 31597 (python3) in control group while starting unit. Ignoring.
May 06 14:15:03 jaku systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 06 14:15:03 jaku systemd[1]: Started Jaku test worker.
jaku@jaku:/nfs/home/jaku/tmp$ ps aux | grep "sig.py"
jaku     31597 99.9  0.0  31884  9916 ?        Rs   14:00  15:10 /usr/bin/python3 /home/jaku/tmp/sig.py
jaku     32359  100  0.0  31884 10032 ?        Rs   14:15   0:43 /usr/bin/python3 /home/jaku/tmp/sig.py
jaku     32483  0.0  0.0  15968  1040 pts/7    S+   14:15   0:00 grep --color=auto sig.py

Solution looks like it's doing its job, but I'm worried about this sentence:

This usually indicates unclean termination of a previous run, or service implementation deficiencies.

Am I missing something or it's really the best solution?

Reference:

已采纳解答

原帖链接 · Jakub Kuszneruk · CC BY-SA 4.0

It looks like there's no way around it, here are some related threads.
But... My assumption that I don't want systemctl command to wait until script will end was wrong.

I wanted this command to be short, because it had to be part of jenkins deployment, and I didn't want deploy process to take more than several minutes.
What I didn't know is that interrupting systemctl command doesn't stop process of turning service down, so possible solution is:

running systemctl command with time limit e.g. timeout 60 systemctl restart services-preifx-* || echo "processes will be restart in background"

Now TimeoutStopSec can be set to some high value (like 10h) to prevent service restarting forever.

Additionally KillMode=process must be set, not to interrupt any child processes.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2020-05-06)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。