How to add Open Job Description in your render pipeline

Open Job Description (OpenJD) is a new open specification introduced by Amazon Web Services (AWS) in January, 2024 that makes job submissions portable across any rendering pipeline. OpenJD includes three Python libraries and tools packages that anyone can use to create integrations with their render pipeline. Openjd-cli provides a command-line tool to develop and run OpenJD jobs on your own workstation or render farm; and openjd-model and openjd-sessions provide functionality to integrate OpenJD into your own render farm and pipeline software.

This blog post describes show how we used the openjd-model-for-python and openjd-sessions-for-python libraries in AWS Thinkbox Deadline 10 software to accept and process OpenJD jobs. Because the software uses Python 3, Deadline’s out-of-the-box scripts and plugins can be customized using its Python based API. In fact, OpenJD will work with any scheduler that has an open interface to accept jobs, and any creative tool that has a command line interface (CLI). To illustrate this, our team wrote a prototype plugin to test an OpenJD job submission to Deadline 10 via command line submission. While this was for internal experimentation, we wanted to share relevant portions of code that demonstrate where OpenJD would interoperate with a render scheduler like Deadline 10.

Running jobs in Thinkbox Deadline 10

All jobs in Deadline 10 run as an instance of a plugin. Each job is given a set of frames to process. To submit a job, you provide job and plugin init files as metadata, and you can attach additional auxiliary files.

We explored how to map OpenJD onto this Deadline job structure, as OpenJD breaks units of work into steps. We chose to submit a single step of an OpenJD job as a Deadline 10 job. We specified metadata about which step inside a job template to run inside the plugin info file, and attached the job template itself as an auxiliary file. We used Deadline 10 frame numbers for the OpenJD step parameter space.

To perform renders, we need to be able to submit auxiliary files and custom job submission files for each step in our OpenJD Job as a Job in Deadline 10, so we are using manual job submission. We use the Deadline 10 standard deadlinecommand CLI for the submission process. We go into more detail later in this blog post as we walk through how an OpenJD job is submitted.

Figure 1: Deadline Deployment in AWS

Adding OpenJD libraries to Deadline 10

Note that in the following example we’re using a non-production, internal installation of the software as an experiment to validate our specific use case. Deadline 10’s python installation does not include full support for OpenJD’s openjd-model and openjd-session libraries at this time. You can try experimenting with the creation of OpenJD plug-ins using AWS Deadline Cloud.

Installing our prototype OpenJD plugin

We install our prototype advanced OpenJD Plugin. Our plugin contains an icon file, an options file, param file, and the Python code. The Deadline 10 documentation gives an overview of plugins and how you can write one yourself.

Next, we log into any machine that has the Deadline 10 repository mounted and copy the required files to repo/custom/plugins directory. Here is what the directory looks like after we have done the installation:

Submitting a job to Deadline 10 manually

As a demonstration, we used the blender-ffmpeg.yaml sample found on the Open Job Description GitHub page.

We created a temporary submission directory for our job on the filesystem, and created job_info.txt and plugin_info.txt files that provide metadata to Deadline 10 and to our plugin. In job_info.txt, we tell Deadline 10 which plugin to use along with key value pairs like the parameter frames that were passed in the OpenJD template:

The file plugin_info.txt takes this form:

We also created a parameter_values.json file. This file contains the values of the job parameters defined in the job template, and will be used by the plugin to construct the OpenJD Job from the job template.

We zipped up the bundle of files for the blender-ffmpeg.yaml template and submitted it along with the required files using deadlinecommand:

Now that our job has been submitted, our prototype OpenJD Plugin gets called.

Breakdown of the prototype OpenJD plugin

Our plugin is invoked to run our job once Deadline 10 sees that we’ve referenced OpenJDPlugin in job_info.txt. The plugin has start and end job hooks that allow us to write OpenJD-specific code for our render. In Deadline 10, the StartJob hook is called when a job is picked up by a worker. It gathers information like the source path format, the job template directory, the step we want to run, parameters, and more. If there is a file bundle, we unzip that to a temporary directory. We then create a job using the OpenJD model for python library, apply path mapping, and create an OpenJD session. The OpenJD Wiki describes how jobs run within the context of sessions, environments and steps.

In our code, we create a session and pass the session id, params, path mapping rules, and a callback for ongoing status. We enter the environments for each job and step if they exist. The environment defines actions—command-lines with arguments to run when the environment is entered or exited—and may also define environment variables that become available in all subsequent actions run in the session.

Next, Deadline 10 calls our RenderTasks() method where we call the OpenJD session.run_task(). Now the OpenJD Python libraries take over and execute the step named in our plugin_info.txt file listed previously.

While our task is running, OpenJD sends progress to our plugin using the action_callback() we passed to our session upon creation. We use Deadline 10 plugin SetProgress() and SetStatusMessage() methods to let the render farm know what is happening.

When the job ends, Deadline 10 will call the EndJob() method of our plugin. We reverse through our environments, exiting them in reverse order for cleanup. We then clean up our temporary directory where OpenJD operated on the template and bundled files.

Following an OpenJD Job through Deadline 10

Let’s examine what happens during job submission, processing, completion, and how you know the integration is working with Deadline 10. As shown previously, we used deadlinecommand to submit the job to Deadline. Following is the output, the result which is successful, and a job id returned from submission.

Once our job is submitted, it’s picked up by a Deadline 10 worker. You can see from the following output that it’s executing our OpenJDRunStep.py plugin installed earlier. The log illustrates that the Deadline 10 job gets called, and the output of Blender is rendered inside an OpenJD Session, according to the actions we’ve defined in our OpenJD Template. Finally, you’ll note completion of the render task.

Our job shows up in the Deadline 10 Monitor just like any other job. We see it by selecting our job in the monitor (following), right clicking on the task in the Tasks panel, and selecting “View Task Reports”.

Conclusion

This post describes how we wrote a prototype plugin to add support for OpenJD in Deadline 10. If you would like to try it out yourself—and if your render manager can run a command as a job—then the openjd-cli provides all you need to run an OpenJD session on your workers. The OpenJD specification repository includes a number of sample files, including the template we used in this blog post. Please take the time to read further about how jobs are constructed and how jobs are run to get a better understanding of what’s going on under the hood with these templates. If you have comments on the specification itself, we invite you to participate in the RFC process or the discussion forums.