HN Distilled

Essential insights from Hacker News discussions

PYX: The next step in Python packaging

Original Article

Hacker News Discussion

Here's a summary of the themes in the Hacker News discussion:

Astral's Business Model and Sustainability

A significant portion of the discussion revolves around Astral's strategy of offering paid, hosted services while keeping their core tools (like uv, Ruff, and Ty) free and open-source. Many users express optimism about this approach, seeing it as a sustainable way to fund development and ensure the continued availability of their excellent tools.

  • "We're hoping to address this concern by building a separate sustainable commercial product rather than monetizing our open source tools." (woodruffw)
  • "I think having a credible, proven business model is a feature of an open source project - without one there are unanswered questions about ongoing maintenance. I'm glad to see Astral taking steps towards that." (simonw)
  • "If Astral can figure out a way to make money using a private registry (something that is used mainly by companies), then they'll have to resources to keep building their amazing open-source projects — Ruff and uv. That's a huge win for Python." (yoavm)
  • "I was surprised to see the community here on HN responding so cautiously. Been developing in python for about a decade now- whenever astral does something I get excited!" (mnazzaro)

However, there's also a contingent of users who are naturally skeptical, citing past experiences with open-source projects that shifted to commercial models or were acquired, leading to changes in features or availability. Concerns are raised about investor pressure influencing future decisions.

  • "The issue is that there isn't a clean business model that will produce the kind of profits that will satisfy their VCs - not that there isn't any business model that will help support a business like theirs." (MoreQARespect)
  • "We've been fed promises like these before. They will inevitably get acquired. Years of documentation, issues, and pull requests will be deleted with little-to-no notice. An exclusively commercial replacement will materialize from the new company that is inexplicably missing the features you relied on in the first place." (monster_truck)
  • "Maybe next year investors change the CEO and entire management and they start monetizing the open source tools. There is no way of knowing. But history tells us that there is a non-trivial chance of this happening." (abdullahkhalids)
  • "Yes, at every company I have been with investors, sooner or later they make the calls, what the founders wanted never has much weight a few years in." (dandellion)

The Need for Improved Python Packaging and Tools

A strong theme throughout the discussion is the long-standing desire for better, faster, and more cohesive Python packaging tools. Many users highlight the pain points and complexities of the current ecosystem, often comparing it unfavorably to the packaging solutions in other languages like Go or Rust.

  • "I'm brush up with Python for a new job, and boy what a ride. Not because of the language itself but the tooling around packages. I'm coming from Go and TS/JS and while these two ecosystems have their own pros and cons, at least they are more or less straightforward to get onboarded (there are 1 or 2 tools you need to know about). In Python there are dozens of tools/concepts related to packaging: pip, easy_install, setuptools, setup.py, pypy, poetry, uv, venv, virtualenv, pipenv, wheels, ..." (dakiol)
  • "Pip is broken and has been for years, they're uninterested in fixing the search. Or even removing the search or replacing it with a message/link to the package index." (nilamo)
  • "Python package management is (largely) solving problems that Go and Node packaging are not even trying to address." (formerly_proven)
  • "The Pythons ecosystem has a ton of tools, and none of them are perfect." (beshrkayali)

The introduction of uv as a faster alternative to pip is frequently mentioned, with users praising its performance and feature set. The new project, pyx, is seen as a potential extension of this improvement, particularly by addressing the complexities of managing dependencies with compiled components, like those found in machine learning libraries.

  • "They already built uv, which works extremely well for that." (anon7000)
  • "Given how great uv is, I’d like to use pyx, but first it would be good if they could provide a solid argument for using it." (normerdfe)
  • "It's easy to compete with Nix tooling, but pretty hard to compete with the breadth of nixpkgs." (conradev)
  • "The speed of Ruby with the memory management of Java, what's not to love? Also, now you have two problems." (nickserv)

The Complexity and Challenges of Managing Dependencies, Especially with Compiled Code

A specific pain point repeatedly raised is the difficulty of managing Python packages that have compiled components, particularly those related to machine learning and GPU acceleration (e.g., PyTorch, CUDA, FlashAttention). This complexity often leads to frustrating setup processes and dependency conflicts.

  • "I'm guessing it's the right PyTorch and FlashAttention and TransformerEngine and xformers and all that for the machine you're on without a bunch of ninja-built CUDA capability pain." (benreesman)
  • "It would be nice if they contributed improvements upstream, but then they can't capture revenue from doing it. I guess it's better to have an alternative and improved PyPI, than to have no improvements and a sense of pride." (nerdponx)
  • "This is so true! On Windows (and WSL) it is also exacerbated by some packages requiring the use of compilers bundled with outdated Visual Studio versions, some of which are only available by manually crafting download paths. I can't wait for a better dev experience." (m_kos)
  • "The problem is that most of the Python community is not working on these kind of complex issues. And frankly, a lot of the people who are actually working on these kinds of issues are not in the python community." (m_kos)

pyx is seen as a potential solution by offering curated, hardware-specific indices and aiming for better metadata querying, which could streamline the installation process for these historically challenging dependencies.

  • "The repository API allows server-driven content negotiation[1], so pyx can service specialized requests while also honoring the normal 503/691 ones." (woodruffw)
  • "Two concrete things I'm excited about: 1) curated, compatibility-tested indices per accelerator (CUDA/ROCm/CPU) so teams stop bikeshedding over torch/cu* matrixes, and 2) making metadata queryable so clients can resolve up front and install in parallel." (bytehumi)
  • "Also +1 to keeping tools OSS and monetizing the hosted service—clear separation builds trust." (bytehumi)

Concerns about Licensing and Future Changes

While Astral stresses that their tools will remain free and open-source, the discussion touches upon the nuances of licensing and community expectations regarding changes.

  • "The playbook is well known and users should be informed." (klysm)
  • "FWIW, I think the full paragraph around that snippet is important context: ... pyx itself is not a tool, it's a service." (woodruffw)
  • "You're saying that would be more painful in a viral license setting, right? If so I agree, although I think there's a pretty long track record of financially incentivized companies being willing to take that pain. MongoDB's AGPL transition comes to mind." (woodruffw)

The lack of a CLA for Astral's tools is noted as a point of difference concerning potential future relicensing, contrasting with companies that might use CLAs to facilitate such changes. However, the primary strategy emphasized is the separation of the open-source tools from the commercial services.

The "Python Packaging Problem" and the Desire for a Unifying Tool

Many participants express frustration with the perceived fragmentation and lack of a single, authoritative "obvious way" to handle Python packaging, echoing the sentiment of the XKCD comic 927.

  • "Why is that hard for the Python community to come up with a single tool to rule them all?" (dakiol)
  • "I'm honestly tired of the Python packing situation. I breathe a sigh of relief in language like Go and Rust with an “authoritative” built-in solution." (callc)
  • "I'm surprised to hear that. Ruby was the first language in my life/career where I felt good about the dependency management and packaging solution." (chao-)
  • "Design by committee is hard. Especially building consensus around reconsidering what at this point should be considered bad design. Python has a few decades of that." (jillesvangurp)

Astral's development of uv and now pyx is seen by some as a positive force in addressing this, potentially leading to better standards or at least offering compelling alternatives that push the ecosystem forward.

Nuances of PyPI and Repository Standards

The discussion also delves into the technical specifics of how Python package repositories work, particularly the limitations of PyPI's current simple repository API (PEP 503 and PEP 691) and how new services like pyx might extend or improve upon it.

  • "For example, there's no metadata API, you have to actually fetch each wheel file and inspect it to figure out certain things about the packages you're trying to resolve/install." (nerdponx)
  • "Your information is out of date. ... PyPI implements PEP 658 here. You can pre-download just the dependency info instead. ... But yes, there is no reverse-search support." (zahlman)
  • "will pyx describe a server protocol that could be implemented by others, or otherwise provide software that others can use to host their own servers? ... pyx uses the same PEP 503/691 interfaces as every other Python index..." (zahlman)
  • "The repository API allows server-driven content negotiation[1], so pyx can service specialized requests while also honoring the normal 503/691 ones." (woodruffw)

The concept of "GPU-aware" registries and the potential for more sophisticated metadata querying to improve dependency resolution and installation speed is a key takeaway.

Community Reaction and Trust in Astral

There's a noticeable split in the community's reaction. A segment expresses significant trust and excitement for Astral's work, citing their track record with uv and Ruff. Others remain cautious due to past experiences with open-source monetization strategies and the influence of venture capital.

  • "Astral has an excellent track record on the engineering and design side, so I expect that whatever they do in this space will basically make sense, it will eventually be codified in a PEP, and PyPI will implement the same endpoint so that other tools like pip and poetry can adopt it." (mikepurvis)
  • "I don't know; I was looking at TS tutorials the other day and there seemed to be at least half a dozen "bundlers" with different tutorails suggesting different ones to use." (aidenn0)
  • "This is effectively what Charlie said they were going to build last September when quizzed about their intended business model on Mastodon: ..." (simonw)
  • "But history tells us that there is a non-trivial chance of this happening." (abdullahkhalids)