Problem

Applicable product versions: Charon-SSP 3.0 and higher on version 7.x of Red Hat, CentOS, or Oracle Linux.

Symptom: When starting Charon-SSP, it produces an error such as one of the following:

2020-06-02 12:49:34 ERROR Thread  Unable to create I/O thread.
2020-06-02 12:49:34 ERROR vtimer  Failed to create schedule worker

or

2020-06-10 18:19:23 ERROR Thread   Unable to create I/O thread with round-robin priority.
2020-06-10 18:19:23 ERROR vtimer   Failed to create schedule worker.

The emulator may start, but then hang while trying to boot the guest OS, or it may stop after encountering the error.

When does the error occur: This problem typically occurs after upgrading the Red Hat or CentOS system, or after enabling additional software (e.g., Docker). However, it may also occur on a freshly installed Linux systems.

Identification of Problem Source

The cause of the problem is that CPU accounting is enabled (implicitly or explicitly) on the Linux host system. This prevents the emulator from starting real-time threads. Services that implicitly enable CPU accounting may not be obvious.

To identify service(s) that implicitly enable CPU accounting, run the following command as the root user:

# find /etc/systemd/system.conf /etc/systemd/system /usr/lib/systemd -type f | xargs grep -e CPUAccounting -e CPUWeight -e StartupCPUWeight \
-e CPUShares -e StartupCPUShares -e CPUQuota |grep -v -e :# -e "^Binary file"

If you see output similar to the following example from Oracle Linux, this shows services that implicitly enable CPU accounting:

/usr/lib/systemd/system/unified-monitoring-agent_config_downloader.service:CPUQuota=40%

/usr/lib/systemd/system/unified-monitoring-agent.service:CPUQuota=40%

Depending on the service and the environment, it may or may not be possible to disable or uninstall the service. For example, the service shown in the OCI example above is required for Compute Instance Monitoring. It is automatically installed if Instance Monitoring is on, and it may be undesirable to disable it. This question must be reviewed and answered with knowledge of the customer requirements.

Solution

The cause of the problem is that CPU accounting is enabled on the Linux host system.

 Possible solutions are shown below:

• If possible disable or remove the software that enables CPU accounting and reboot Linux before starting the emulator again.
• If you cannot remove or disable the offending software because of technical or customer environment restrictions, you can enable one of the following workarounds:
  • Remove the limitation on real-time scheduling:
    # sysctl -w kernel.sched_rt_runtime_us=-1
    To make the configuration permanent, add kernel.sched_rt_runtime_us=-1 to a file in /etc/sysctl.d (e.g., 90-rt.conf), or to sysctl.conf (deprecated in newer Linux versions).
    This configuration introduces the risk that a run-away real-time process could starve other processes and in extreme cases, lock up the system. The default value of this parameter is 950000 (=0.95 seconds).
  • Disable CPU accounting at boot by adding the following parameter to the kernel boot line:
    cgroup_disable=cpu
    Test this during an interactive boot first. If satisfied with the result, make it permanent. For example, by using:
    # grubby --update-kernel=ALL --args="cgroup_disable=cpu".
    Please note: this setting may prevent some software packages from running that are dependent on CPU accounting.

Reinstalling the host operating system may solve the problem for a short time if the problem occurred after an upgrade or other change, but this is not a practical solution as the problem can reoccur again with the next change.

Related articles

Related articles appear here based on the labels you select. Click to edit the macro and add or change labels.