Processor affinity, clusters, runmasks, and inherit masks
A question that often arises in a multiprocessor (and thus, multicore) environment is:
Can I make it so that one processor handles the GUI, another handles the database,
and the other two handle the realtime functions?
The answer is: Yes, absolutely.
This is done through processor affinity, the ability to associate a thread with a particular processor or cluster of processors.
A cluster is a group of associated processors. There are pre-defined clusters, including one that represents all the processors on the system, and a set of clusters where each one represents a different processor. Thus, on a system with N processors, there are always (at least) N+1 clusters.
The startup program may define custom clusters, either directly in the program code, or by configuring them through the -c command option. This allows processors to be grouped based on hardware properties. For example, on a machine with heterogeneous processors, the startup program might define one cluster for the faster processors (or CPUs) and another for the slower ones. A compute-bound thread could then set its affinity mask (i.e., its association with a particular cluster) to the faster cluster.
pidin syspage=cluster
For details about the naming of clusters, refer to the -c option in the startup-* entry in the Utilities Reference.
- No processor may be in more than eight clusters; it is recommended that it belong to as few clusters as possible (e.g., perhaps four for a complex system)
- Each thread can be associated with only one cluster
- A thread can run only on processors that belong to the cluster indicated by its runmask
Runmasks can be set explicitly through functions, as explained in the next section,
or implicitly by causing any thread that receives a message to inherit the sender's runmask,
as explained in Client runmask inheritance
below.
However, setting the runmask affects the processor affinity for only the calling thread,
not further threads that it may create. To change the processor affinity of any further threads,
you use an inherit mask, as explained in
Setting inherit masks
below.
Setting runmasks
When a thread starts up, its runmask depends on whether it was created as an additional thread in
an existing process, or as the first (main) thread of a new process.
In the first case, the runmask is the inherit mask of the creator thread
(i.e., the thread that called a function such as thrd_create() or
pthread_create()).
In the second case, the runmask can be defined by attribute-setting functions (refer to
Runmasks and inherit masks used in new processes
below).
You can change the runmask for a thread by associating it with a particular cluster by calling pthread_cluster_set(). You can retrieve the runmask for a particular cluster via pthread_cluster_get_runmask().
You can also manually specify the runmask for the calling thread by issuing the ThreadCtl() _NTO_TCTL_RUNMASK command. In this case, though, you must be sure that the runmask you specify corresponds to a valid cluster—a thread cannot be bound to an arbitrary set of processors. If this runmask matches a cluster, the call succeeds; otherwise, it fails with EINVAL.
Setting inherit masks
When a thread starts up, its inherit mask is set to the inherit mask of the creator thread. You can change the inherit mask for a thread (along with its runmask) by calling pthread_cluster_set_and_inherit() and naming the clusters that you want to associate the thread and its children with.
You can also manually specify the inherit mask (along with the runmask) for the calling thread by issuing the ThreadCtl() _NTO_TCTL_RUNMASK_GET_AND_SET_INHERIT command. However, you must be sure that the inherit mask and runmask you specify correspond to valid clusters. If both masks match clusters, the call succeeds; otherwise, it fails with EINVAL.
Runmasks and inherit masks used in new processes
If you're starting a new process, you can specify a runmask by calling posix_spawnattr_setxflags() to set the POSIX_SPAWN_EXPLICIT_CPU extended flag, then pthread_spawnattr_setrunmask_np() to set the runmask for a new process, and then posix_spawn() to create the process.
If you specify the runmask attribute, then when the new process starts up, the main thread's runmask and inherit mask are given that runmask value. If you don't define a runmask, then when the new process starts up, the main thread's runmask and inherit mask are set to the inherit mask of the creator thread (i.e., the one that called posix_spawn()). In both cases, if no further changes are made in the new process (i.e., none of the mask-setting functions described in previous sections are called), then this results in normal inherit mask behavior, meaning all threads in the new process are given the runmask and inherit mask of the main thread.
Client runmask inheritance
When creating a channel, you can enable the _NTO_CHF_INHERIT_RUNMASK flag to cause any thread that receives a message on this channel to inherit the sender's runmask. With this client runmask inheritance, the server's runmask has no effect on the server's execution behavior when it receives messages. This is because the server is either RECEIVE-blocked and, hence, not running, or running with the runmask of the client whose message it's handling.
If this flag is set, then when a server thread receives a pulse on the channel, the thread's runmask is set to the thread's inherit mask.
For more information, refer to the _NTO_CHF_INHERIT_RUNMASK description in the ChannelCreate() entry in the C Library Reference.