Problem
Applicable product versions: Charon-SSP 3.0 and higher on version 7.x of Red Hat, CentOS, or Oracle Linux.
Symptom: When starting Charon-SSP, it produces an error such as one of the following:
2020-06-02 12:49:34 ERROR Thread Unable to create I/O thread. 2020-06-02 12:49:34 ERROR vtimer Failed to create schedule worker
or
2020-06-10 18:19:23 ERROR Thread Unable to create I/O thread with round-robin priority. 2020-06-10 18:19:23 ERROR vtimer Failed to create schedule worker.
The emulator may start, but then hang while trying to boot the guest OS, or it may stop after encountering the error.
When does the error occur: This problem typically occurs after upgrading the Red Hat or CentOS system, or after enabling additional software (e.g., Docker). However, it may also occur on a freshly installed Linux systems.
Identification of Problem Source
The cause of the problem is that CPU accounting is enabled (implicitly or explicitly) on the Linux host system. This prevents the emulator from starting real-time threads. Services that implicitly enable CPU accounting may not be obvious.
To identify service(s) that implicitly enable CPU accounting, run the following command as the root user:
# find /etc/systemd/system.conf /etc/systemd/system /usr/lib/systemd -type f | xargs grep -e CPUAccounting -e CPUWeight -e StartupCPUWeight \ -e CPUShares -e StartupCPUShares -e CPUQuota |grep -v -e :# -e "^Binary file"
If you see output similar to the following example from Oracle Linux, this shows services that implicitly enable CPU accounting:
/usr/lib/systemd/system/unified-monitoring-agent_config_downloader.service:CPUQuota=40%
/usr/lib/systemd/system/unified-monitoring-agent.service:CPUQuota=40%
Solution
The cause of the problem is that CPU accounting is enabled on the Linux host system.
Possible solutions are shown below:
- If possible disable or remove the software that enables CPU accounting and reboot Linux before starting the emulator again.
- If you cannot remove or disable the offending software, you can enable the following workarounds:
- Remove the limitation on real-time scheduling:
# sysctl -w kernel.sched_rt_runtime_us=-1
(add to sysctl.conf to make permanent).
This introduces the risk that a run-away real-time process could starve other processes and in extreme cases, lock up the system. The default value of this parameter is950000
(=0.95
seconds). - Disable CPU accounting at boot by adding the following parameter to the kernel boot line:
cgroup_disable=cpu
Test this during an interactive boot first. If satisfied with the result, make it permanent. For example, by using:# grubby --update-kernel=ALL --args="
.cgroup_disable=cpu
"
Please note: this setting may prevent some software packages from running that are dependent on CPU accounting.
- Remove the limitation on real-time scheduling:
Reinstalling the host operating system may solve the problem for a short time if the problem occurred after an upgrade or other change, but this is not a practical solution as the problem can reoccur again with the next change.
Related articles