How I solved PyTorch's cross-platform nightmare

The Hacker News discussion on PyTorch packaging and cross-platform compatibility reveals several key themes:

PyTorch's Packaging Complexity and "Nightmare" Status

A central theme is the significant difficulty and frustration associated with packaging PyTorch-based projects due to its complex dependencies and need for hardware-specific optimization. Many users echoed the sentiment that setting this up is a "nightmare." One user vividly described their struggles:

"When I was a PhD student, I already had 12 years of using and administrating Linuxes as my personal OS, and I'd already had my share of package manager and dependency woes. But managing Python, PyTorch, and CUDA dependencies were relatively new to me. Sometimes I'd lose an evening here or there to something silly. But I had one week especially dominated by these woes, to the point where I'd have dreams about package management problems at the terminal. They were mundane dreams but I'd chalk them up as nightmares. The worst was having the pleasant dream where those problems went away forever, only to wake up to realize that was not the case." - lynndotpy

This sentiment is further supported by another user who noted the inherent difficulty: "The last 20% of packaging is 800% of your time." - doctorpangloss.

The Role of Hardware and Performance in PyTorch's Complexity

Users frequently attributed PyTorch's packaging challenges to its deep commitment to performance optimization across a wide array of hardware, including various CPUs and numerous GPUs. This necessity for hardware-specific builds inevitably "leaks into packaging," as one commenter put it:

"Torch simply has to work this way because it cares about performance on a combination of multiple systems and dozens of GPUs. The complexity leaks into packaging." - gobdovan

This point was elaborated on by another user who suggested that the difficulty arises from optimization rather than a lack of suitable cross-platform tools:

"Or, looked at a different way, Torch has to work this way because Python packaging has too narrow of an understanding of platforms which treats many things that are materially different platforms as the same platform." - dragonwriter

The comparison was made to solving "the assembly language nightmare," highlighting the low-level hardware considerations involved. - mdaniel

Limitations and Nuances of "Cross-Platform" in Python Packaging

There was a back-and-forth regarding the definition and implementation of "cross-platform" in the context of Python packaging, particularly concerning manylinux. Some users questioned the scope of "cross-platform" when it seemed to only encompass a limited set of Linux distributions favored by the original author:

"So, you're doubling down on OP's misnomer of 'cross platform means whatever platforms I use', eh?" - mdaniel

However, others defended the utility of tools like manylinux, explaining its purpose and how it facilitates broader compatibility:

"I do not know what you are objecting to. I have successfully packaged wheels for MacOS and Ubuntu with manylinux. I presume it works with some others too, but that is what I can personally attest to. Alpine is catered to by the musllinux project. You should be specific about which distributions you have in mind." - esafak

A more charitable interpretation was offered, suggesting the author of the original article focused on the platforms they personally used, which is a natural starting point:

"I think a more charitable interpretation of TFA would be: 'I Have Come Up With A Recipe for Solving PyTorch's Cross-Platform Nightmare.' That is: there's nothing stopping the author from building on the approach he shares to also include Windows/FreeBSD/NetBSD/whatever. It's his project (FileChat), and I would guess he uses Linux. It's natural that he'd solve this problem for the platforms he uses, and for which wheels are readily available." - cstrahan

Alternative Packaging and Environment Management Solutions

Several alternative tools and approaches were proposed to mitigate the challenges of managing PyTorch dependencies:

Conda/Mamba: Many users advocated for conda and mamba, particularly the conda-forge builds, as being more reliable for detecting hardware and installing the correct versions of complex dependencies. "In my opinion, anything that touch compiled packages like pytorch should be packaged with conda/mamba on conda-forge. I found it is the only package manager for python which will reliably detect my hardware and install the correct version of every dependency." - kwon-young "Likewise, this was my experience. If ever I need to 'pip anything' I know I'm in for a bad time. Conda is built for literally this exact problem. Still not a breeze, but much better than trying to manually freeze all your pip dependencies." - levocardia
Pixi: pixi was highlighted as a modern tool that aims to simplify the management of both conda and pypi packages, offering better system for hardware feature detection. "Try pixi! Pixi is a much more sane way for building with conda + pypi packages in a single tool that makes this so much easier for torch development, regardless if you get the condaforge or pypi builds of pytorch." - zbowling "Check out Pixi! Pixi is an alternative to the common conda and pypi frontends and has better system for hardware feature detection and get the best version of Torch for your hardware that is compatible across your packages (except for AMD at the moment). It can pull in the condaforge or pypi builds of pytorch and help you manage things automagically across platforms. ... The condaforge builds of PyTorch also are a bit more sane." - zbowling
torchruntime: This library was presented as a solution specifically designed to help users install the correct PyTorch version based on their system's OS, GPU vendor, and model. "Might help here, it's designed precisely for this purpose. pip install torchruntime torchruntime install torch It figures out the correct torch to install on the user's PC, factoring in the OS (Win, Linux, Mac), the GPU vendor (NVIDIA, AMD, Intel) and the GPU model (especially for ROCm, whose configuration varies per generation and ROCm version). And it tries to support quite a number of older GPUs as well, which are pinned to older versions of torch." - cmdr2
Rust-based alternatives (like Burn): For those seeking to simplify deployment across platforms, switching to languages like Rust with libraries designed for cross-platform compatibility was suggested. "Check out https://github.com/tracel-ai/burn project! It makes deploying models across different platforms easy. It uses Rust instead of Python." - antimora
ctypes and standalone compilation: One maintainer of a PyTorch operator package found success by decoupling their compiled C/C++/CUDA code from Python and PyTorch dependencies, using ctypes to load libraries and requiring users to install PyTorch separately. "I have tried various approaches over the years (including the ones endorsed by PyTorch), but the only solution I have found that seems to work flawlessly for everyone who uses it is to have no Python or PyTorch dependence in the compiled code, and to load the compiled libraries using ctypes. I use an old version of nvcc to compile the CUDA, use manylinux2014 for the Linux builds, and ask users to install PyTorch themselves before installing my package." - userabchn

Packaging Constraints from PyPI

A technical point was raised regarding PyPI's policies on direct references in package metadata, which can impact how PyTorch or related projects are packaged.

"Note that https://peps.python.org/pep-0440/#direct-references says: 'Public index servers SHOULD NOT allow the use of direct references in uploaded distributions. Direct references are intended as a tool for software integrators rather than publishers.' This means that PyPI will not accept your project metadata as you currently have it configured." - di

Questioning Distribution Package Usage

One user floated the idea of using distribution-provided packages for PyTorch, posing a question about potential downsides:

"Is there a problem using distro packages for Pytorch? What are the downsides of using the official Fedora Pytorch for example?" - tuna74