问题原文
原帖链接 · Jakub Kuszneruk · CC BY-SA 4.0
I have bunch of services responsible for running action consumed from the queue.
I want to be able to restart services gently (without interrupting action which're already running)
It can be solved with handling SIGTERM sent by systemd and saving information that the program should exit after current action's processed.
There's other minor problem with the fact that after some time defined as TimeoutStopSec in service configuraiton file systemd will send additional SIGKILL to brutally terminate my process.
I can easily avoid it with setting TimeoutStopSec=infinity. Then systemctl stop'll wait until script terminates itself, which may last even more than an hour, and leads me to main problem.
I don't want systemctl command to wait until script will end
It looks like SendSIGKILL=no configuration does the job.
This results with retrying SIGTERM after TimeoutStopSec, then creating new worker, and leaving the old one running.
journalctl log
May 06 14:14:43 jaku systemd[1]: Stopping Jaku test worker...
May 06 14:14:43 jaku python3[31597]: * 15 <frame object at 0x14d8108>
May 06 14:14:53 jaku systemd[1]: jaku-test-worker.service: State 'stop-sigterm' timed out. Skipping SIGKILL.
May 06 14:14:53 jaku python3[31597]: * 15 <frame object at 0x14d8108>
May 06 14:15:03 jaku systemd[1]: jaku-test-worker.service: State 'stop-final-sigterm' timed out. Skipping SIGKILL. Entering failed mode.
May 06 14:15:03 jaku systemd[1]: jaku-test-worker.service: Failed with result 'timeout'.
May 06 14:15:03 jaku systemd[1]: Stopped Jaku test worker.
May 06 14:15:03 jaku systemd[1]: jaku-test-worker.service: Found left-over process 31597 (python3) in control group while starting unit. Ignoring.
May 06 14:15:03 jaku systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 06 14:15:03 jaku systemd[1]: Started Jaku test worker.
jaku@jaku:/nfs/home/jaku/tmp$ ps aux | grep "sig.py"
jaku 31597 99.9 0.0 31884 9916 ? Rs 14:00 15:10 /usr/bin/python3 /home/jaku/tmp/sig.py
jaku 32359 100 0.0 31884 10032 ? Rs 14:15 0:43 /usr/bin/python3 /home/jaku/tmp/sig.py
jaku 32483 0.0 0.0 15968 1040 pts/7 S+ 14:15 0:00 grep --color=auto sig.py
Solution looks like it's doing its job, but I'm worried about this sentence:
This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Am I missing something or it's really the best solution?
Reference:
已采纳解答
原帖链接 · Jakub Kuszneruk · CC BY-SA 4.0
It looks like there's no way around it, here are some related threads.
But... My assumption that I don't want systemctl command to wait until script will end was wrong.
I wanted this command to be short, because it had to be part of jenkins deployment, and I didn't want deploy process to take more than several minutes.
What I didn't know is that interrupting systemctl command doesn't stop process of turning service down, so possible solution is:
running systemctl command with time limit e.g. timeout 60 systemctl restart services-preifx-* || echo "processes will be restart in background"
Now TimeoutStopSec can be set to some high value (like 10h) to prevent service restarting forever.
Additionally KillMode=process must be set, not to interrupt any child processes.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
