Routing job submissions between disparate compute environments

[0001] Technical Field

[0002] The present disclosure relates to high performance computing systems, and more particularly to a method and system for automatically directing a workload or job away from busy or more loaded distributed computing environments to idle or less loaded environments and/or environments that are dynamically scalable.

[0003] Description of Related Art

[0004] Job scheduling environments enable the distribution of heterogeneous compute workloads across large compute environments. Compute environments within large enterprises tend to have the following characteristics:

• • [0005] Static size;
  • [0006] Typically built out of physical machines;
  • [0007] Largely homogeneous configuration;
  • [0008] Heavily connected within the same cluster;
  • [0009] Loosely connected with other regional clusters;
  • [0010] Poorly connected to clusters in other geographic locations;
  • [0011] Shared storage space typically not accessible between clusters; and
  • [0012] It is common to see hot spots where one cluster is busy and another is idle.

[0013] As a result of variations in regional cluster size and regional workload demand, it is attractive to run jobs in other regions or geographic locations. However, since the workloads tend to be tightly coupled by network and storage constraints, it is difficult to build a functional workload that spans resources across these zones of high performance compute, networking, and storage resources.

[0014] The current art in job scheduling deals with reconfiguring physical hosts based upon job scale (U.S. Pat. No. 8,544,017), without considering the availability of other computing environments besides local, or internal, servers, including remote, or external/cloud, resources.

[0015] The present disclosure addresses the shortcomings of the prior art by disclosing a system and method for transparently (without requiring additional user or submitter input) routing computational jobs between disparate computing environments, in either a single distribution or through a multi-pass routing. In addition, the present disclosure addresses the need to ensure the data in a submitted job is present in a remote environment via the creation of transfer jobs whose completion guarantees that the data is transferred, or synchronized, either by the transfer job itself or an asynchronous system.

[0016] In one aspect of the disclosure, the command-line software, API calls, or other submission interface that submit a workload (job) to compute resources itself determines whether a job should be placed on local resources or moved to one or many remote computing clusters based upon factors that can include, for example, application availability, local queue wait times for the job versus data transfer times plus remote queue wait times for the jobs in various remote clusters. Based upon this routing decision, in one aspect of the system, the software could move all the individual jobs in the submission to the optimal local or remote environment, or in another aspect of the invention, iteratively route a subset of the jobs and reconsider the optimal place for the remainder over a plurality of iterations.

[0017] In another aspect of the disclosure, the command-line software, API calls, or other submission interface could request a routing decision by sending a submission description to a separate software component via a network or other API call(s) and submits the jobs corresponding to the response to the submission description.

[0018] In another aspect of the disclosure, when the submitted job is placed on servers that are not local to the data, the submitted job relies upon, the input data and output, or results, data are guaranteed to be synchronized by adding the submitted job to a workflow that executes the synchronization or synchronization check for inputs in a transfer job step before, and a similar transfer job step after to synchronize the outputs after the submitted job. This workflow includes a first job whose completion signals that all data required to start the job is present on the remote system, having executed the data synchronization itself or alternately checking to see if the data has been transferred by an asynchronous process, and it is appropriate to start the submitted job. That is, the data on the remote system is verified as step independent step of the data transfer. The computing job that was submitted is then processed, and the optional final transfer job completes when any generated output data is transferred back to the original local, or internal, compute environment.

[0019] The present disclosure provides a system and method for any developer of high performance compute applications to make use of compute resources across an internal enterprise and/or multiple infrastructure-as-a-service (IaaS) cloud environments, seamlessly. This is done by treating the each individual cluster of computers, internal or external to the closed enterprise network, as regions of computational power with well-known performance characteristics surrounded by a zone of performance and reliability uncertainty. This system transfers the data, or ensures the data is transferred asynchronously, and migrates the workload to remote clusters as if it existed and was submitted locally. The decision making and execution of this workflow is implemented as a complete transparent process to the developer and application. That is, no input, other than the job, is required of the user. In the event that no optimal cluster exists in a routing table, the decision making can optionally create a cluster in the cloud using provisioning APIs, and add that cluster to the routing table. Thus, the system can dynamically modify the available cluster environment in response to existing conditions. The complexities of the data and job migration are not exposed to the developer or application. The developer need only make their application function in a single region and the present system automatically handles the complexities of migrating the job to other regions.

[0020] The present approach places compute geographically separated compute resources in the same scheduling environment and treats local and remote environments as equivalent. Two factors in the incumbent approach contribute to make the present approach a superior solution for the problem. Factor one: operations across questionable WAN links that execute under the assumption of low latency and high bandwidth will consistently fail. Factor two: performance characteristics of global shared storage devices are typically so slow that they result in the perception of failure due to lack of rapid progress on any job in the workload. By avoiding both of these pitfalls, the present approach ensures jobs can flow between environments more readily, and the execution of these jobs can proceed with the speed and reliability that the developer would expect when running on an internal cluster located in one region. The only additional costs paid in the scenario where the present system is used is the migration of data from the local to the remote cluster to support the job execution along with the transfer of result data back to the origination region after the computation has completed at the remote region.

[0021] The system continuously gathers detailed performance and use data from the clusters, and uses this data to make decisions related to job routing based on parameters such as:

• • [0022] The desire of the user or automated workload to direct the jobs to an internal environment based on security, performance, and regulatory compliance considerations;
  • [0023] Tolerance of the cost of running in an external, dynamic computing environment such as Amazon's AWS service;
  • [0024] Current utilization of all compute resources across the entire computing landscape; and
  • [0025] Bandwidth available between clusters for data transfer.

[0026] The matchmaking algorithm used to determine the eventual compute job routing is configurable to account for a variety of dynamic properties. The present system performs meta-scheduling for workloads by applying all the knowledge it has about the jobs being submitted and the potential clusters that could run the jobs and routing jobs to the appropriate regions automatically, without application, developer or end user intervention. The job meta-scheduling decision happens at submit time and immediately routes the jobs out to schedulers that then have the responsibility for running the work on execute machines in specific regions, either internal or external, static or dynamic.

[0027] The present system allows for clusters and jobs scheduled in to them, to run completely independent of each other, imparting much greater stability as the need for constant, low-latency communication is not required to maintain a functional environment. The system also allows for these clusters to function entirely outside of the scope of this architecture, providing for a mix of completely local workloads and jobs that flow in and out from other clusters via the meta-scheduling algorithm. This allows legacy interoperability and flexibility when it comes to security: in cases where it is not desirable for jobs to be scheduled to run remote to their point of submission by the system, the job may be tagged such that the system will submit only to a local region or specified set of validated regions. The system also promotes high use rates among widely distributed pools of computational resources, with more workloads submitted through the meta-scheduling algorithm resulting in greater overall utilization.

[0028] The present disclosure provides:

[0029] A process for determining a proper routing of a job submission, whether as a set of service requests or command-lines to be executed, to a proper cluster environment, with appropriate routing between a plurality of different local or remote environments, or clusters, based upon attributes about the state of the clusters (such as time waiting in the queue, how “full” a given cluster is of jobs, network bandwidth, geographic location, availability of a given environment for the submitted job (such as security or access issues), user direction (explicit tagging or option setting, including the queue the job is submitted to), and submission hints of other kinds including administrator set parameters and business logic, using a matchmaking, or ranking, process between the available options.

[0030] It is understood the process can include routing the submitted jobs all at once or a fraction at a time (iteratively) between potential cluster environments. Further, is it is understood the routing may be changed prior to execution of all the submitted jobs, as corresponding to existing or updated attributes.

[0031] The present disclosure includes a process for moving a submitted job to a remote cluster, or environment, which creates a new workflow that has a transfer job whose completion indicates the data the submitted job relies upon is in the remote environment, either through synchronous (just in time) or asynchronous transfer (beforehand), then running the submitted jobs, then a dependent final transfer job whose completion indicates the data the job created is submitted back to the cluster the job was submitted from originally. However, it is also understood the data the job created can be presented or submitted to a different cluster than received the job.

[0036] A preferred embodiment of a process and system according to the invention will be described, but the invention is not limited to this embodiment. It is understood that the submitted job routing process to local and remote resources performed by the system may be performed on all the submitted jobs at once (completely), or in multiple passes on a fraction of the jobs (e.g. “iteratively”). Additionally, the data transfer workflow process performed by the system to move input and output data from and back to the local cluster can be performed synchronously (e.g., “just in time,” JIT) or asynchronously (before by another process).

[0037] A preferred embodiment where a job is submitted through a command-line or API that mimics the prior native command-line and API interface for a given job scheduler, or cluster, prompting a routing decision between a local computing environment and/or a plurality of remote environments, where attributes of the submitted jobs, the remote clusters, and the applications in question yield a placement (or routing) decision. If the submitted job remains in a local (or internal) environment or cluster because the local or internal environment is available or otherwise is the proper placement for the submitted job, then the job is submitted for execution.

[0038] If no optimal cluster exists in a routing table of available clusters, the decision making can optionally create a cluster in the cloud using provisioning APIs, and add that cluster to the routing table. Otherwise if one or more remote clusters, or environments, from the routing table and matchmaking process are the right places to put the job, the submitted job becomes dependent upon an initial transfer job that moves synchronously or checks an asynchronous transfer for, the data of the submitted job into the remote cluster or clusters, then the portions of the submitted job going to each remote cluster are executed once that transfer job is completed, with a similar final transfer job ensuring the data has moved back to the original location of the submitted job (data), or another directed final location.

[0039] Referring to FIG. 1, a block diagram, which shows a process of job submission by an end user or a job submission portal. A job refers to a unit of work or set of instructions given to an operating system to execute.

[0040] The processes and components in this system are:

• • [0041] Communication methods that are commoditized and easily passed through regional and Internet-facing firewalls.
  • [0042] Detailed monitoring of the OS-level and scheduler-level health that allows for job routing decisions based on collected health metrics.
  • [0043] Command-lines/API/webpage/webservice (A1) that are configured to emulate the native utilities of the scheduler, or cluster, to provide aggregated job, host, and queue monitoring across the entire computing environment and not just for one region.
  • [0044] Additional metadata about the job submission that can be passed in through configuration files, job submission directives, and native scheduler variables.
  • [0045] The job submission command-lines/API/webpage/webservice gathers (B2) environment information (B3) derived variables pulled from a dry run of the routing/submission and (B1) user input metadata.
  • [0046] In the case where the server environment (C1) is unavailable or takes too long to respond, the job submission executable will always execute the submission locally (D1). This way, job submission always occurs within a predefined time interval.
  • [0047] In the case where a task runs on multiple clusters, the output will be differentiated through the workflow using a prefix designated by the cluster name as defined in the system.
  • [0048] The output of the job submission should be identical to the output produced by the native scheduler commands. This way, users, workloads, or APIs that leverage this system can interoperate transparently with this system. This output is returned by the server environment during typical execution (D2)

[0049] Referring to FIG. 2, a block diagram is provided showing the routing engine and the variables that can drive the decision. The processes and components in this system are:

• • [0050] GUI dashboards within a server architecture that can be used to configure, manage, and monitor the job routing and submission behaviors. (A1)
  • [0051] Default submission configurations that can be used by administrators to configure the system to conform to their specific policies and expectations. (B1)
  • [0052] Metadata that can be defined for the scheduling environments such as, but not limited to, security domain, authorized user groups, geographic location, available shared storage space, advertised applications, current capacity, oversubscription thresholds and dynamic execute node capabilities. This metadata can be input during configuration and/or derived in realtime by the monitoring environment. (B5)
  • [0053] Preference may be given to routing the job locally if it is most expedient. (B2, B3, B4)
  • [0054] The full matchmaking routine is only entered if local routing is not immediately apparent. (B5)
  • [0055] The routing decision is ultimately used to process the actual submission, no matter what cluster unit is chosen. (C1)

[0056] FIG. 3 is a block diagram which shows the workflow of a remote job submission including data transfer and scheduler interaction.

[0057] The processes and components in this system are:

• • [0058] A hub-and-spoke design for a central server to communicate with one or more cluster units. (A1)
  • [0059] The key decision during the submission is whether or not the routing is local or remote, because this dictates the requirement to move data. (B1, A2)
  • [0060] Cluster units can represent both internal clusters of machines and external clusters of machines, with statically allocated or dynamically allocated lists of computational resources.
  • [0061] A ticket-based data transfer mechanism that can provide both internal-initiated and external-initiated data transfers on either a scheduled or on-demand basis. (B2, B4) (D1, D2)
  • [0062] Secure, reliable communication between the central server and the remote nodes for command execution. (C1, B3)
  • [0063] Proper error handling of any potential failure before, during, or after a committed job submission. If any errors are encountered, submit locally as a failsafe. (C1)

[0064] FIG. 4 is a block diagram that shows the process of backfilling work onto a partially idle internal cluster when submitting to a remote cluster. The processes and components in this system are:

• • [0065] When a job submission is committed to a particular cluster unit, there is an opportunity for further load balancing. Although the bulk of the workload is designated for the remote cluster unit, a subset of the workload may be carved off to run on local resources that are immediately available, decreasing the overall runtime. (A1)
  • [0066] This branch of behavior is only taken if the following is true: (1) the submission is not a tightly coupled parallel job (2) the submission is a job array (3) the ability to split task arrays is enabled within the system.
  • [0067] The system counts the number of available execution slots, counts the running jobs, and calculates the available slots. (B1)
  • [0068] The job array is split such that the local cluster is filled first (C1) and the remainder of jobs is submitted to selected remote cluster(s). (C2)

[0069] These two or more submissions created for (C1) and (C2) are processed, and the workflow proceeds as described above.

[0070] It is also contemplated that the system can provide, or obtain, user authorization across multiple security domains. Specifically, remote computing environments have distinct authorization and entitlement domains for users. The system handles domain traversal transparently, without requiring individualized user authentication for each domain. Upon first use by a new user, the system requests authentication against available remote computing environments (such as via password or encryption key). Future uses by the user can then authenticate automatically unless the privileges of the user have been revoked. The routing decision for the respective job submissions will select only among, or filter authenticated remote environments for the submitting user.

[0071] In a further configuration, the system provides for multi-pass routing for job arrays. When a very large job array is submitted to the system, the sum of all available slots in all clusters may be insufficient to run all jobs immediately. In this case, the system may perform multiple meta-scheduling passes for a single job array. Each meta-scheduling pass proceeds to split the job array further as described in FIG. 4. The system will continue ranking clusters and scheduling portions of the array to the best matching cluster until the entire job array has been scheduled.

[0072] Thus, the system provides for submitting workload of submitted jobs within an internal or cloud cluster (computing environment), wherein the submission precisely, or within predetermined limits, mimics the behavior of a job scheduler-based job submission corresponding to a knowledge of the operation of the job scheduler, the system analyzes as much metadata as possible about the workload being submitted, prior to the workload being submitted to the intended scheduler.

[0073] The system provides a job routing mechanism coupled with a scheduler monitoring solution that can account for a flexible number of environment parameters to make a corresponding decision about job routing or workload distribution between a local or set of remote clusters, or schedulers, that are set forth in a routing table.

[0074] The system provides automated remote access to perform seamless (without requiring submitter action) input data transfer, remote command execution and job monitoring once the job routing decision is made.

[0075] The system provides for a set of jobs to be run on multiple heterogeneous environments and transparently deposit the results in a consolidated area when complete, without requiring performance related input (other than the job) from the user.

[0076] A system is provided for submitting a workload having associated metadata within a cloud computing environment, wherein the behavior of a scheduler-based job submission is mimicked corresponding to the knowledge of an operation of the job scheduler at least a portion of the available metadata corresponding to the submitted workload (job) before moving the job to an intender scheduler. In select configurations, the portion of the metadata is analyzed to determine the processing of the workload or job.

[0077] It is contemplated that a job routing mechanism can be coupled with a scheduler monitoring solution that can account for a flexible number of environment parameters to make a real time decision about job or workload routing between a local set of remote clusters, or schedulers, that are listed in a routing table. The routing table may be static or dynamic, as a new cluster can be created using commonly available cloud provisioning APIs and place the newly created cluster (or scheduler of the created cluster) in the routing table.

[0078] The system also provides for automated remote access to perform seamless data transfer (without requiring user control input), remote command execution, and job monitoring once a job routing decision is made by the system.

[0079] Therefore, a set of jobs can run on multiple heterogeneous environments, wherein the results (end product of the job) are transparently deposited in a consolidated area upon job completion.

[0080] The submitted job input and/or output data can be transferred synchronously as part of an initial transfer job, wherein the dependent submitted jobs are then run, and result data is transferred back as part of a dependent final transfer job to the originating cluster (or other designated cluster). Alternatively, it is understood, where submitted job input and/or output data are transferred asynchronously by another process, the transferred data is verified by the initial and final transfer jobs.

[0081] While a preferred embodiment of the invention has been shown and described with particularity, it will be appreciated that various changes in design and formulas and modifications may suggest themselves to one having ordinary skill in the art upon being apprised of the present invention. It is intended to encompass all such changes and modifications as fall within the scope and spirit of the appended claims.