slycat.web.server.remote

Functions for managing cached remote ssh sessions.

Slycat makes extensive use of ssh and the Slycat Agent to access remote resources located on the high performance computing platforms used to generate ensembles. This module provides functionality to create cached remote ssh / agent sessions that can be used to retrieve data from remote hosts. This functionality is used in a variety of ways:

• Web clients can browse the filesystem of a remote host.
• Web clients can create a Slycat model using data stored on a remote host.
• Web clients can retrieve images on a remote host (an essential part of the Parameter Image Model).
• Web clients can retrieve video compressed from still images on a remote host.

When a remote session is created, a connection to the remote host over ssh is created, an agent is started (only if the required configuration is present), and a unique session identifier is returned. Callers use the session id to retrieve the cached session and communicate with the remote host / agent. A “last access” time for each session is maintained and updated whenever the cached session is accessed. If a session times-out (a threshold amount of time has elapsed since the last access) it is automatically deleted, and subsequent use of the expired session id will fail.

Each session is bound to the IP address of the client that created it - only the same client IP address is allowed to access the session.

class slycat.web.server.remote.Session(client, username, hostname, ssh, sftp, agent=None)

Bases: object

Encapsulates an open session connected to a remote host.

Examples

Calling threads must serialize access to the Session object. To facilitate this, a Session is a context manager - callers should always use a with statement when accessing a session:

>>> with slycat.web.server.remote.get_session(sid) as session:
...   print session.username

accessed

Return the time the session was last accessed.

browse(path, file_reject, file_allow, directory_reject, directory_allow)

cancel_job(jid)

Submits a command to the slycat-agent to cancel a running job on a cluster running SLURM.

Parameters:jid (int) – Job ID Returns:response – A dictionary with the following keys: jid, output, errors Return type:dict

checkjob(jid)

Submits a command to the slycat-agent to check the status of a submitted job to a cluster running SLURM.

Parameters:jid (int) – Job ID Returns:response – A dictionary with the following keys: jid, status, errors Return type:dict

client

Return the IP address of the client that created the session.

close()

get_file(path, **kwargs)

get_image(path, **kwargs)

get_job_output(jid, path)

Submits a command to the slycat-agent to fetch the content of the a job’s output file from a cluster running SLURM.

Note that the expected format for the output file is slurm-[jid].out.

Parameters:jid (int) – Job ID Returns:response – A dictionary with the following keys: jid, output, errors Return type:dict

get_remote_job_status(jid)

check of the status of a job running on an agent with a hostanemd session :param jid: job id :return:

get_user_config()

Submits a command to the slycat-agent to fetch the content of a user’s .slycatrc file in their home directory.

Returns:response – A dictionary with the configuration values Return type:dict

get_video(vsid)

get_video_status(vsid)

hostname

Return the remote hostname accessed by the session.

launch(command)

Submits a single command to a remote location via the slycat-agent or SSH.

Parameters:command (string) – Command Returns:response – A dictionary with the following keys: command, output, errors Return type:dict

run_agent_function(wckey, nnodes, partition, ntasks_per_node, time_hours, time_minutes, time_seconds, fn, fn_params, uid)

Submits a command to the slycat-agent to run a predefined function on a cluster running SLURM.

Parameters:

• wckey (string) – Workload characterization key
• nnodes (int) – Number of nodes requested for the job
• partition (string) – Name of the partition where the job will be run
• ntasks_per_node (int) – Number of tasks to run on a node
• ntasks (int) – Number of tasks allocated for the job
• ncpu_per_task (int) – Number of CPUs per task requested for the job
• time_hours (int) – Number of hours requested for the job
• time_minutes (int) – Number of minutes requested for the job
• time_seconds (int) – Number of seconds requested for the job
• fn (string) – Name for the Slycat agent function
• fn_params (dict) – Additional params for the agent function

Returns:

response – A dictionary with the following keys: jid, errors

Return type:

dict

run_remote_command(command)

run a remote command from an HPC source running a slycat agent. the command could be things such as starting an hpc script or batch job or something as simple as moving files. the only requirement is that the script is in our list of trusted scripts.

this_func()->calls agent_command_func()->which runs_shell_command() -> which launches_script()-> sends_response_to_agent()->sends_response_to_server() ->sends_status_response_to_client()

Parameters:

• self –
• command – json form of a command to be run

{

“scripts”: //pre defined scripts that are registerd with the server [{

“script_name”:”script_name”, // key for the script lookup “parameters”: [{key:value},…] // params that are fed to the script

},…] “hpc”: // these are the hpc commands that may be add for thing such as slurm {

“is_hpc_job”:bol, // determins if this should be run as an hpc job “parameters”:[{key:value},…] // things such as number of nodes

}

} :return: {“msg”:”message from the agent”, “error”: boolean}

set_user_config(config)

Submits a command to the slycat-agent to set the content of a user’s .slycatrc file in their home directory.

Returns:response Return type:dict

sftp

submit_batch(filename)

Submits a command to the slycat-agent to start an input batch file on a cluster running SLURM.

Parameters:filename (string) – Name of the batch file Returns:response – A dictionary with the following keys: filename, jid, errors Return type:dict

username

Return the username used to create the session.

slycat.web.server.remote.cache_object(pid, key, content_type, content)

slycat.web.server.remote.check_session(sid)

Return a true if session is active

If the session has timed-out or doesn’t exist, returns false

Parameters:sid (string) – Unique session identifier returned by slycat.web.server.remote.create_session(). Returns: Return type:boolean

slycat.web.server.remote.create_session(hostname, username, password, agent)

Create a cached remote session for the given host.

Parameters:

• hostname (string) – Name of the remote host to connect via ssh.
• username (string) – Username for ssh authentication.
• password (string) – Password for ssh authentication.
• agent (bool) – Used to require / prevent agent startup.

Returns:

sid – A unique session identifier.

Return type:

string

slycat.web.server.remote.delete_session(sid)

Delete a cached remote session.

Parameters:sid (string, required) – Unique session identifier returned by slycat.web.server.remote.create_session().

slycat.web.server.remote.get_session(sid)

Return a cached remote session.

If the session has timed-out or doesn’t exist, raises a 404 exception.

Parameters:sid (string) – Unique session identifier returned by slycat.web.server.remote.create_session(). Returns:session – Session object that encapsulates the connection to a remote host. Return type:slycat.web.server.remote.Session

slycat.web.server.remote.get_session_server(client, sid)

Return a cached remote session.

If the session has timed-out or doesn’t exist, raises a 404 exception.

Parameters:sid (string) – Unique session identifier returned by slycat.web.server.remote.create_session(). Returns:session – Session object that encapsulates the connection to a remote host. :param client: Return type:slycat.web.server.remote.Session