DCP (Distributive Compute Protocol) is a market-based compute orchestration platform where users can contribute or utilize existing infrastructure to execute compute-heavy workloads, earning cash or optimizing resource usage. Previously, DCP only supported JavaScript and WebAssembly since workers are based on JavaScript engines, making it challenging for users of other languages to run workloads without complex WebAssembly toolchains or big JavaScript re-writes.
A lack of support for native Python workloads in DCP has been a major missing component. Data Scientists and Researchers love Python due its rich ecosystem for scientific compute. Porting or re-writing python projects to WebAssembly and JavaScript is a non-starter for users who depend on this ecosystem and need a simple solution. Researchers can't afford to spend time compiling a Python interpreter to WebAssembly, interfacing with it through JavaScript, and managing Python dependencies themselves.
We're excited to announce the launch of the Pyodide Worktime, a new DCP Worktime which adds first-class support for Python execution on DCP.
With the Pyodide Worktime, Job Deployers can write their compute workloads in Python and leverage a long list of popular Python packages supported by Pyodide like Numpy, SciPy, Pandas, PIL, OpenCV, and more. The Pyodide Worktime also supports an in-memory filesystem enabling project source code and resources to be uploaded via Tarballs and modified during execution of Job's slice.
The Pyodide Worktime is proud to be based on the incredible Pyodide project. Pyodide is a Python interpreter based on CPython compiled to WebAssembly. It's embeddable in JavaScript runtimes and browsers and provides high-level interoperation between JavaScript and Python types. Being based on Pyodide means we get support for an ever-growing list of Python packages which have been ported to WebAssembly.
import dcp
dcp.init()
from dcp import compute_for
def workfn(datum):
import dcp
dcp.progress()
return datum * datum
my_j = compute_for([1, 2, 3, 4, 5], workfn)
my_j.exec()
res = my_j.wait()
print(res) # 1 4 9 16 25
const dcp = require('dcp-client').initSync();
const { compute } = dcp;
const workfn = `
def workfn(datum):
import dcp
dcp.progress()
return datum * datum
`;
const myJob = compute.for([1, 2, 3, 4, 5], workfn)
// specify the worktime here, by default, the JS SDK uses Map Basic
myJob.worktime = 'pyodide';
myJob.exec().then(console.log); // 1 4 9 16 25
The Pyodide Worktime is embedded and shipped with every DCP Worker. If you start a new browser-based worker instance on https://dcp.work, Pyodide will load and the worker will be able to execute either Map Basic and Pyodide workloads (along with additional future Worktimes yet to be released).
Despite now being embedded in every DCP Worker, the Pyodide Worktime began life as a really big JavaScript Work Function. This made prototyping simple during development of the worktime and is how we intend future DCP Worktimes to be developed.
The easiest way to try is through Distributive's Python SDK. Install it with pip3 install dcp