Parallel graceful shutdown for `ThreadPoolTaskExecutor` and `ThreadPoolTaskScheduler`

[jgslima]

Applications that need to perform graceful shutdown of tasks submitted to ThreadPoolTaskExecutor and/or ThreadPoolTaskScheduler, may make use of the setting awaitTerminationMillis (and possibly waitForTasksToCompleteOnShutdown).

However, application may take a very long time to actually finish if these conditions apply:

• application uses both ThreadPoolTaskExecutor and ThreadPoolTaskScheduler (and possibly multiple ThreadPoolTaskExecutors).
• submitted tasks are not quick.
• there are SmartLifecycle beans which implements lenghty stopping.

Real examples are web applications that use such components and make use of the web container "graceful shutdown" feature of Spring Boot. The overall termination sequence of an application is:

1. SIGTERM is sent.
2. SmartLifecycle asynchronous stopping triggers the web container graceful shutdown.
3. SmartLifecycle asynchronous stopping blocks and waits for the web container shutdown.
4. Context closing proceeds, invoking DisposableBean/@PreDestroy methods, so, say:
   1. ThreadPoolTaskExecutor's destroy() is called, blocking and awaiting for the tasks to finish. If the application uses multiple ThreadPoolTaskExecutors, for each one of them an awaiting occurs.
   2. ThreadPoolTaskScheduler's destroy() is called, blocking and awaiting for the tasks to finish.

The proposal here is to create the possibility to have some form to make all the pools to finish their tasks in parallel. Ideally, in parallel with the stopping of other SmartLifecycle beans.

In Kubernetes spring web applications that fits in the scenario above, the total time took by a Pod to finish ends up being too high (which is aggravated due to the need to configure a preStop hook to give time to kubeproxy to note the pod deletion). This has real effects: as, rigthly, in a rollout Kubernetes does not wait for old Pods to actually finish in order to create new ones, applications with a large number of pods and with a large termination time, end up having a large number of Pods actually running in the rollout (new Pods and many still being terminated). We have seen this triggering a cluster auto scale to make the cluster able to handle this large number of Pods that occur during the rollout.

[jgslima]

Implementation considerations

Probably the implementation might make ExecutorConfigurationSupport to implement SmartLifecycle.

In my opinion, the implementation of stop does not have to be assynchronous. So, only the stop() method would be implemented (leaving stop(Runnable callback) with its default, synchronous implementation).

In the stop() method, the ExecutorService would be asked to shut down (but without calling here awaitTerminationIfNecessary()). This would make all thread pools to start their shutdown. And, in web applications that use graceful shutdown, this would happen in parallel with the graceful shutdown blocking as well.

After, the destroy() would see that the ExecutorService is already shut down and would call executor.awaitTermination(). In this way, the calls to executor.awaitTermination() would not happen in parallel, but it does not really matter, since all thread pools will be finishing their tasks in parallel. The total time took to all the calls to destroy() would be, roughly, the greatest (worse) time took by some of the pools.

[jgslima]

Hello people.
Any thoughts on this?
Thanks.

[jgslima]

@rstoyanchev , I do not want to contribute to this vast amount of issues in the spring project. As it seems, the community creates an amount of issues greater than the capacity of yours to analyze them.

Feel free to just close the issue.

[jgslima]

Spring team,
I am considering closing this issue, unless you want to keep it open.
Did you have chance to analyze this?

Here, the lenghty application shutdown remains.

Thanks.

[jgslima]

If I work on the code and submit a PR for this, it would be considered?

[jhoeller]

In the context of Project CRaC support, we are revisiting our own lifecycle adapters, and we ran into the executor/scheduler implementations as well. It is somewhat more complicated with stop/restart having to be possible, so a shutdown on stop() would have to be followed by a recreation of the thread pool on start(), or we'd have to use a different trigger to stop the currently running tasks while keeping the thread pool alive.

We'll try to sort this out for our 6.1.

[jhoeller]

The challenge with the Lifecycle contract is that it serves several purposes, and even one more now with Checkpoint Restore. ThreadPoolTaskExecutor and in particular ThreadPoolTaskScheduler are not really ideal candidates for implementing Lifecycle since the underlying ThreadPoolExecutor has no stop or restart functionality, just shutdown. So whenever a Lifecycle.stop() call comes in but later followed up by Lifecycle.start(), we could only really reinitialize the ThreadPoolExecutor completely, losing all of it previously scheduled tasks... which we'd have to re-schedule somehow.

Up until this point, we treated ThreadPoolTaskExecutor and ThreadPoolTaskScheduler as general delegates which do not participate in lifecycle management on their own. Instead, whoever submits tasks to the executor/scheduler is responsible for pausing/cancelling/rescheduling them whenever necessary. For graceful shutdown purposes, such callers could implement Lifecycle themselves and stop their tasks early, so that the target executor does not have any actual tasks left whenever it eventually gets disposed. This is the approach that we applied ourselves in DefaultMessageListenerContainer and co.

For Project CRaC, it is not actually necessary to stop/restart thread pools since the active threads are a part of the checkpoint image anyway. So we currently only stop DefaultMessageListenerContainer and co, winding down its async workers but not the target thread pool that it is using. That said, it would not hurt to stop and restart thread pools themselves as well which is what we're currently investigating. If such a thread pool restart turns out to be impossible, a regular Lifecycle implementation would not be appropriate since it would receive stop calls in restart scenarios as well, not just for graceful shutdown.

@jgslima what kind of tasks are typically left to await termination for in your scenario, and how did those tasks get submitted? Is this about late-submitted one-off tasks during the shutdown process, or rather about periodic tasks kicking in once more?

[jhoeller]

In terms of an early shutdown signal for executors and schedulers, it does not have to come from Lifecycle, it could also come from ContextClosedEvent which arrives even before the lifecycle stop calls (and does not have any restart semantics attached to it). This would effectively run in parallel with SmartLifecycle stopping, awaiting actual termination in the destruction step then.

[jhoeller]

Experimenting with this a bit, we seem to be able to implement the general Lifecycle stop/start semantics with a pause/resume approach in (Scheduled)ThreadPoolExecutor subclasses (along the lines of the PausableThreadPoolExecutor suggested in the ThreadPoolExecutor javadoc) which would address our Checkpoint Restore requirements as well.

This combines nicely with a ContextClosedEvent-driven early shutdown signal, addressing graceful shutdown scenarios then (which are otherwise not covered by such a pause/resume approach for stop/start). For ThreadPoolTaskExecutor and ThreadPoolTaskScheduler, we should be able to provide an out-of-the-box solution for all those scenarios in 6.1.