MidPoint 4.0 and later
Sometimes a task gets stuck, presenting as being in
STALLED state. Often the cause lies in a connector not responding in a timely manner (or never). The best way how to diagnose these situations is to take and inspect a thread dump. However, this usually requires an access to the server console, which is not always available.
Therefore in 4.0 midPoint has a new (a bit experimental) functionality: showing and recording thread dumps.
Showing thread dumps from GUI
The functionality is available from the Internals configuration menu item:
You can either show all threads in JVM:
Or show only threads corresponding to the tasks currently executing on this node (this is probably the most useful option):
Or you can request that thread dumps of currently executing tasks are attached to those tasks in the form of
<diagnosticInformation> property. You can inspect these elements by exporting the task repository objects or showing them via GUI.
You can test this feature e.g. using the following samples containing long-running tasks:
- https://github.com/Evolveum/midpoint/blob/master/samples/tasks/task-scheduling.xml#L53 (use Task2, maybe modified by prolonging the time to run by enlarging the
- https://github.com/Evolveum/midpoint/blob/master/samples/tasks/bulk-actions/script-delay-for-testing-thread-dumps.xml (waits 10 seconds for each object found in repo)
Getting thread dumps via REST
Sometimes the system might be unreachable via GUI e.g. because of resource shortage. If at least the REST interface is working you can get thread dump like this:
The first command retrieves all the threads, the second one gives threads related to all running tasks (on this node), and the third one gives threads related to a specific running task (again, it must be running on the node receiving the REST request).
Getting thread dumps automatically
Current midPoint setting is that as soon as it detects that the task is stalled, it records the threads dump into the task's
<diagnosticInformation> property. This is repeated each hour until the task starts making progress or is stopped.
The current (hard-wired) setting is that the last 5 dumps are kept for each task.