Spank Spank

NAMESPANK - Slurm Plug-in Architecture for Node and job (K)control DESCRIPTIONThis manual briefly describes the capabilities of the Slurm Plug-inArchitecture for Node and job Kontrol (SPANK) as well as the SPANKconfiguration file: (By default: plugstack.conf.)SPANK provides a very generic interface for stackable plug-inswhich may be used to dynamically modify the job launch code inSlurm. SPANK plugins may be built without access to Slurm sourcecode. They need only be compiled against Slurm's spank.h header file,added to the SPANK config file plugstack.conf,and they will be loaded at runtime during the next job launch. Thus,the SPANK infrastructure provides administrators and other developersa low cost, low effort ability to dynamically modify the runtimebehavior of Slurm job launch.NOTE: All SPANK plugins should be recompiled when upgrading Slurmto a new major release. The SPANK API is not guaranteed to be ABIcompatible between major releases. Any SPANK plugin linking to any of theSlurm libraries should be carefully checked as the Slurm APIs and headerscan change between major releases. SPANK PLUGINSSPANK plugins are loaded in up to five separate contexts during aSlurm job. Briefly, the five contexts are:localIn local context, the plugin is loaded by srun. (i.e. the "local"part of a parallel job).remoteIn remote context, the plugin is loaded by slurmstepd. (i.e. the "remote"part of a parallel job).allocatorIn allocator context, the plugin is loaded in one of the job allocationutilities salloc, sbatch or scrontab.slurmdIn slurmd context, the plugin is loaded in theslurmd daemon itself. Note: Plugins loaded in slurmd contextpersist for the entire time slurmd is running, so if configuration ischanged or plugins are updated, slurmd must be restarted for the changesto take effect.job_scriptIn the job_script context, plugins are loaded in the context of thejob prolog or epilog. Note: Plugins are loaded in job_scriptcontext on each run on the job prolog or epilog, in a separate addressspace from plugins in slurmd context. This means there is nostate shared between this context and other contexts, or even betweenone call to slurm_spank_job_prolog or slurm_spank_job_epilogand subsequent calls.In local context, only the init, exit, init_post_opt, andlocal_user_init functions are called. In allocator context, only theinit, exit, and init_post_opt functions are called.Similarly, in slurmd context, only the init and slurmd_exitcallbacks are active, and in the job_script context, only the job_prologand job_epilog callbacks are used.Plugins may query the context in which they are running with thespank_context and spank_remote functions defined in.SPANK plugins may be called from multiple points during the Slurm joblaunch. A plugin may define the following functions:slurm_spank_initCalled just after plugins are loaded. In remote context, this is justafter job step is initialized. This function is called before any pluginoption processing.slurm_spank_job_prologCalled at the same time as the job prolog. If this function returns anon-zero value and the SPANK plugin that contains it is required in theplugstack.conf, the node that this is run on will be drained.slurm_spank_init_post_optCalled at the same point as slurm_spank_init, but after alluser options to the plugin have been processed. The reason that theinit and init_post_opt callbacks are separated is so thatplugins can process system-wide options specified in plugstack.conf inthe init callback, then process user options, and finally take someaction in slurm_spank_init_post_opt if necessary.In the case of a heterogeneous job, slurm_spank_init is invoked onceper job component.slurm_spank_local_user_initCalled in local (srun) context only after alloptions have been processed.This is called after the job ID and step IDs are available.This happens in srun after the allocation is made, but beforetasks are launched.slurm_spank_user_initCalled after privileges are temporarily dropped. (remote context only)slurm_spank_task_init_privilegedCalled for each task just after fork, but before all elevated privilegesare dropped. (remote context only)slurm_spank_task_initCalled for each task just before execve (2). If you are restricting memorywith cgroups, memory allocated here will be in the job's cgroup. (remotecontext only)slurm_spank_task_post_forkCalled for each task from parent process after fork (2) is complete.Due to the fact that slurmd does not exec any tasks until alltasks have completed fork (2), this call is guaranteed to run beforethe user task is executed. (remote context only)slurm_spank_task_exitCalled for each task as its exit status is collected by Slurm.(remote context only)slurm_spank_exitCalled once just before slurmstepd exits in remote context.In local context, called before srun exits.slurm_spank_job_epilogCalled at the same time as the job epilog. If this function returns anon-zero value and the SPANK plugin that contains it is required in theplugstack.conf, the node that this is run on will be drained.slurm_spank_slurmd_exitCalled in slurmd when the daemon is shut down.All of these functions have the same prototype, for example: int slurm_spank_init (spank_t spank, int ac, char *argv[])Where spank is the SPANK handle which must be passed back toSlurm when the plugin calls functions like spank_get_item andspank_getenv. Configured arguments (See CONFIGURATIONbelow) are passed in the argument vector argv with argumentcount ac.SPANK plugins can query the current list of supported slurm_spanksymbols to determine if the current version supports a given plugin hook.This may be useful because the list of plugin symbols may grow in thefuture. The query is done using the spank_symbol_supported function,which has the following prototype: int spank_symbol_supported (const char *sym);The return value is 1 if the symbol is supported, 0 if not.SPANK plugins do not have direct access to internally defined Slurmdata structures. Instead, information about the currently executingjob is obtained via the spank_get_item function call. spank_err_t spank_get_item (spank_t spank, spank_item_t item, ...);The spank_get_item call must be passed the current SPANKhandle as well as the item requested, which is defined by thepassed spank_item_t. A variable number of pointer arguments are alsopassed, depending on which item was requested by the plugin. Alist of the valid values for item is kept in the spank.h headerfile. Some examples are:S_JOB_UIDUser id for running job. (uid_t *) is third arg of spank_get_itemS_JOB_STEPIDJob step id for running job. (uint32_t *) is third arg of spank_get_item.S_TASK_EXIT_STATUSExit status for exited task. Only valid from slurm_spank_task_exit.(int *) is third arg of spank_get_item.S_JOB_ARGVComplete job command line. Third and fourth args to spank_get_itemare (int *, char ***).See spank.h for more details.SPANK functions in the local and allocator environment shoulduse the getenv, setenv, and unsetenv functions to view andmodify the job's environment.SPANK functions in the remote environment should use thespank_getenv, spank_setenv, and spank_unsetenv functions toview and modify the job's environment. spank_getenvsearches the job's environment for the environment variablevar and copies the current value into a buffer bufof length len. spank_setenv allows a SPANKplugin to set or overwrite a variable in the job's environment,and spank_unsetenv unsets an environment variable inthe job's environment. The prototypes are: spank_err_t spank_getenv (spank_t spank, const char *var, char *buf, int len); spank_err_t spank_setenv (spank_t spank, const char *var, const char *val, int overwrite); spank_err_t spank_unsetenv (spank_t spank, const char *var);These are only necessary in remote context since modifications ofthe standard process environment using setenv (3), getenv (3),and unsetenv (3) may be used in local context.Functions are also available from within the SPANK plugins toestablish environment variables to be exported to the SlurmPrologSlurmctld, Prolog, Epilog and EpilogSlurmctldprograms (the so-called job control environment).The name of environment variables established by these calls will be prependedwith the string SPANK_ in order to avoid any security implicationsof arbitrary environment variable control. (After all, the job controlscripts do run as root or the Slurm user.).These functions are available from local context only. spank_err_t spank_job_control_getenv(spank_t spank, const char *var, char *buf, int len); spank_err_t spank_job_control_setenv(spank_t spank, const char *var, const char *val, int overwrite); spank_err_t spank_job_control_unsetenv(spank_t spank, const char *var);See spank.h for more information.Many of the described SPANK functions available to plugins returnerrors via the spank_err_t error type. On success, the return valuewill be set to ESPANK_SUCCESS, while on failure, the return valuewill be set to one of many error values defined in slurm/spank.h. TheSPANK interface provides a simple function const char * spank_strerror(spank_err_t err);which may be used to translate a spank_err_t value into itsstring representation.The slurm_spank_log function can be used to print messages back to theuser at an error level. This is to keep users from having to rely on theslurm_error function, which can be confusing because it prepends"error:" to every message. SPANK OPTIONSSPANK plugins also have an interface through which they may defineand implement extra job options. These options are made available tothe user through Slurm commands such as srun(1), salloc(1),and sbatch(1). If the option is specified by the user, its value isforwarded and registered with the plugin in slurmd when the job is run.In this way, SPANK plugins may dynamically provide new options andfunctionality to Slurm.Each option registered by a plugin to Slurm takes the form ofa struct spank_option which is declared in as struct spank_option char * name; char * arginfo; char * usage; int has_arg; int val; spank_opt_cb_f cb; ;Wherenameis the name of the option. Its length is limited to SPANK_OPTION_MAXLENdefined in .arginfois a description of the argument to the option, if the option does takean argument.usageis a short description of the option suitable for --help output.has_arg0 if option takes no argument, 1 if option takes an argument, and2 if the option takes an optional argument. (See getopt_long (3)).valA plugin-local value to return to the option callback function.cbA callback function that is invoked when the plugin option isregistered with Slurm. spank_opt_cb_f is typedef'd in as typedef int (*spank_opt_cb_f) (int val, const char *optarg, int remote);Where val is the value of the val field in the spank_optionstruct, optarg is the supplied argument if applicable, and remoteis 0 if the function is being called from the "local" host (e.g. host wheresrun or sbatch/salloc are invoked) or 1 from the "remote" host(host where slurmd/slurmstepd run) but only executed by slurmstepd(remote context) if the option was registered for such context.Plugin options may be registered with Slurm usingthe spank_option_register function. This function is only validwhen called from the plugin's slurm_spank_init handler, andregisters one option at a time. The prototype is spank_err_t spank_option_register (spank_t sp, struct spank_option *opt);This function will return ESPANK_SUCCESS on successful registrationof an option, or ESPANK_BAD_ARG for errors including invalid spank_thandle, or when the function is not called from the slurm_spank_initfunction. All options need to be registered from all contexts in whichthey will be used. For instance, if an option is only used in local (srun)and remote (slurmd) contexts, then spank_option_registershould only be called from within those contexts. For example: if (spank_context() != S_CTX_ALLOCATOR) spank_option_register (sp, opt);If, however, the option is used in all contexts, the spank_option_registerneeds to be called everywhere.In addition to spank_option_register, plugins may also export optionsto Slurm by defining a table of struct spank_option with thesymbol name spank_options. This method, however, is not supportedfor use with sbatch and salloc (allocator context), thusthe use of spank_option_register is preferred. When using thespank_options table, the final element in the array must befilled with zeros. A SPANK_OPTIONS_TABLE_END macro is providedin for this purpose.When an option is provided by the user on the local side, either by command lineoptions or by environment variables, Slurm will immediately invoke theoption's callback with remote=0. This is meant for the plugin to do localsanity checking of the option before the value is sent to the remote side duringjob launch. If the argument the user specified is invalid, the plugin shouldissue an error and issue a non-zero return code from the callback. The pluginshould be able to handle cases where the spank option is set multiple timesthrough environment variables and command line options. Environment variablesare processed before command line options.On the remote side, options and their arguments are registered justafter SPANK plugins are loaded and before the spank_inithandler is called. This allows plugins to modify behavior of all pluginfunctionality based on the value of user-provided options.As an alternative to use of an option callback and global variable,plugins can use the spank_option_getopt option to check forsupplied options after option processing. This function has the prototype: spank_err_t spank_option_getopt(spank_t sp, struct spank_option *opt, char **optargp);This function returns ESPANK_SUCCESS if the option defined in thestruct spank_option opt has been used by the user. If optargpis non-NULL then it is set to any option argument passed (if the optiontakes an argument). The use of this method is required to processoptions in job_script context (slurm_spank_job_prolog andslurm_spank_job_epilog). This function is valid in the following contexts:slurm_spank_job_prolog, slurm_spank_local_user_init, slurm_spank_user_init,slurm_spank_task_init_privileged, slurm_spank_task_init, slurm_spank_task_exit,and slurm_spank_job_epilog. CONFIGURATIONThe default SPANK plug-in stack configuration file isplugstack.conf in the same directory as slurm.conf(5),though this may be changed via the Slurm config parameterPlugStackConfig. Normally the plugstack.conf fileshould be identical on all nodes of the cluster.The config file lists SPANK plugins,one per line, along with whether the plugin is required oroptional, and any global arguments that are to be passed tothe plugin for runtime configuration. Comments are preceded with '#'and extend to the end of the line. If the configuration fileis missing or empty, it will simply be ignored.NOTE: The SPANK plugins need to be installed on the machines thatexecute slurmd (compute nodes) as well as on the machines that execute joballocation utilities such as salloc, sbatch, etc (login nodes).The format of each non-comment line in the configuration file is: required/optional plugin arguments For example: optional /usr/lib/slurm/test.soTells slurmd to load the plugin test.so passing no arguments.If a SPANK plugin is required, then failure of any of theplugin's functions will cause slurmd, or the job allocator command toterminate the job, while optional plugins only cause a warning.If a fully-qualified path is not specified for a plugin, then thecurrently configured PluginDir in slurm.conf(5) is searched.SPANK plugins are stackable, meaning that more than one plugin maybe placed into the config file. The plugins will simply be calledin order, one after the other, and appropriate action taken onfailure given that state of the plugin's optional flag.Additional config files or directories of config files may be includedin plugstack.conf with the include keyword. The includekeyword must appear on its own line, and takes a glob as its parameter,so multiple files may be included from one include line. Forexample, the following syntax will load all config files in the/etc/slurm/plugstack.conf.d directory, in local collation order: include /etc/slurm/plugstack.conf.d/*which might be considered a more flexible method for building upa spank plugin stack.The SPANK config file is re-read on each job launch, so editingthe config file will not affect running jobs. However care shouldbe taken so that a partially edited config file is not read by alaunching job. ErrorsWhen SPANK plugin results in a non-zero result, the following changes will result:CommandFunction Context Exitcode Drains Node Fails jobsrun slurm_spank_init local 1 no yessrun slurm_spank_init_post_opt local 1 no yessrun slurm_spank_local_user_init local 1 no nosrun slurm_spank_user_init remote 0 no nosrun slurm_spank_task_init_privileged remote 1 no yessrun slurm_spank_task_post_fork remote 0 no nosrun slurm_spank_task_init remote 1 no yessrun slurm_spank_task_exit remote 0 no nosrun slurm_spank_exit local 0 no yessalloc slurm_spank_init allocator 1 no yessalloc slurm_spank_init_post_opt allocator 1 no yessalloc slurm_spank_init local 1 no yessalloc slurm_spank_init_post_opt local 1 no yessalloc slurm_spank_local_user_init local 1 no yessalloc slurm_spank_user_init remote 0 no nosalloc slurm_spank_task_init_privileged remote 1 no yessalloc slurm_spank_task_post_fork remote 0 no nosalloc slurm_spank_task_init remote 1 no yessalloc slurm_spank_task_exit remote 0 no nosalloc slurm_spank_exit local 0 no yessalloc slurm_spank_exit allocator 0 no yessbatch slurm_spank_init allocator 1 no yessbatch slurm_spank_init_post_opt allocator 1 no yessbatch slurm_spank_init local 1 no yessbatch slurm_spank_init_post_opt local 1 no yessbatch slurm_spank_local_user_init local 1 no yessbatch slurm_spank_user_init remote 0 yes nosbatch slurm_spank_task_init_privileged remote 1 no yessbatch slurm_spank_task_post_fork remote 0 yes nosbatch slurm_spank_task_init remote 1 no yessbatch slurm_spank_task_exit remote 0 no nosbatch slurm_spank_exit local 0 no nosbatch slurm_spank_exit allocator 0 no noNOTE: The behavior for ProctrackType=proctrack/pgid may result intimeouts for slur