This discussion revolves around the FFmpeg project, touching upon its performance, development model, API accessibility, and the intricacies of its low-level optimizations, particularly assembly language.
Commitment to Performance and Scale
Several users highlight FFmpeg's dedication to performance and the significant impact this has given its widespread use. The project's ability to achieve such performance is seen as a testament to its quality and the work of its contributors.
- "I canât imagine the scale that FFMPEG operates at. A small improvement has to be thousands and thousands of hours of compute saved. Insanely useful project." - cr125rider
- "Their commitment to performance is a beautiful thing." - prisenco
However, one user humorously points out the prerequisite for such developer resources:
- "Seems so easy! You only need the entire world even tangentially related to video to rely solely on your project for a task and you too can have all the developers you need to work on performance!" - byteknight
This leads to a discussion about the reliance of critical software on unpaid labor in the open-source world.
- "You know friend, if open source actually worked like that I wouldnât be so allergic to releasing projects. But it doesnât - a large swath of the economy depends on unpaid labour being treated poorly by people who wonât or canât contribute." - hluska
The Role and Importance of Performance Optimization
The discussion also touches on whether all software needs to be performance-critical. While some argue that unnecessary optimization is wasteful, others contend that even seemingly non-critical applications have performance goals.
- "Yeah no, I'd like non-performance critical programs to focus on other things than performance thank you" - Almondsetat
- "Surely all programs are performance critical. Any program we think isn't is just a program where the performance met the criteria already." - EliRivers
- "That would be an enormous waste of time. 99.9% of software doesn't have to be anywhere near optimal. It just has to not be wasteful. Sadly lots of software is blatantly wasteful. But it doesn't take fancy assembly micro optimization to fix it, the problem is typically much higher level than that. It's more like serialized network requests, unnecessarily high time complexities, just lots of unnecessary work and unnecessary waiting. Once you have that stuff solved you can start looking at lower level optimization, but by that point most apps are already nice and snappy so there's no reason to optimize further." - sfn42
FFmpeg's API and Usability
A significant portion of the conversation centers on FFmpeg's API and how developers interact with it. While the command-line interface (CLI) is powerful, its complexity is a point of discussion, with some advocating for more traditional API approaches.
- "It'd be nice, though, to have a proper API (in the traditional sense, not SaaS) instead of having to figure out these command lines in what's practically its own programming language...." - zahlman
Counterarguments clarify that FFmpeg does provide C APIs through libraries like libavcodec and libavformat, which are the foundation for the CLI tool itself.
- "FFMpeg does have an API. It ships a few libraries (libavcodec, libavformat, and others) which expose a C api that is used in the ffmpeg command line tool." - codys
The utility of these low-level C APIs for different programming environments is also debated, with some finding Python subprocess calls more practical for certain use cases, especially regarding error handling.
- "They're relatively low level APIs. Great if you're a C developer, but for most things you'd do in python just calling the command line probably does make more sense." - Wowfunhappy
- "If you are processing user data, the subprocess approach makes it easier to handle bogus or corrupt data. If something is off, you can just kill the subprocess. If something is wrong with the linked C api, it can be harder to handle predictably." - javier2
Recommendations for better Python bindings, like pyav, are also shared.
- "For future reference, if you want proper python bindings for ffmpeg you should use pyav. * To be more precise, these are bindings for the libav libraries that underlie ffmpeg" - ansk
The complexity of the CLI is noted, but the emerging utility of AI in generating FFmpeg commands from natural language is also a positive observation.
- "I get why the CLI is so complicated, but I will say AI has been great at figuring out what I need to run given an English language input. It's been one of the highest value uses of AI for me." - xxpor
Low-Level Optimization and Assembly Language
The discussion delves into FFmpeg's use of assembly language for performance optimization, particularly in its libavutil/x86/x86inc.asm file. The sophistication of these optimizations, including the use of macro-preprocessors and architecture-specific code, is noted.
- "There is serious abuse of nasm macro-preprocessor. Going to be tough to move away to another assembler." - sylware
The portability of these assembly instructions across different CPUs is clarified, with the understanding that FFmpeg maintains separate assembly files for various architectures (x86-64, ARM, etc.) and includes generic C fallbacks.
- "They don't. It's just x86-64." - KeplerBoy (referring to tutorials)
- "The lessons yes, but the repo contains assembly for the 5-6 architectures in wide use in consumer hardware today. Separate files of course." - ahartmetz
- "I think there's a generic C fallback, which can also serve as a baseline. But for the big (targeted) architectures, there one handwritten assembly version per arch." - CannotCarrot
The dynamic dispatching mechanism for selecting the most optimal assembly function based on CPU features (like AVX, SSE4, or even specific CPU generations) is explained.
- "On startup, it runs cpuid and assigns each operation the most optimal function pointer for that architecture. In addition to things like âsupports avxâ or âsupports sse4â some operations even have more explicit checks like âis a fifth generation celeronâ. The level of optimization in that case was optimizing around the cache architecture on the cpu iirc. Source: I did some dirty things with chromes native client and ffmpeg 10 years ago." - faluzure
The process of identifying performance bottlenecks and the debate between handwritten assembly versus compiler-generated intermediate representations (like LLVM IR) also arises. The consensus leans towards assembly being necessary for highly specific architectural optimizations, where C and compilers might fall short, particularly in vectorized code. However, it's stressed that measuring and benchmarking are crucial for validating assembly's superiority.
- "IME, not really. I've done a fair bit of hand-written assembly and it exclusively comes up when dealing with architecture-specific problems - for everything else you can just write C (unless you hit one of the edge cases where C semantics don't allow you to express something in C, but those are rare). For example: C and C++ compilers are really, really good at writing optimized code in general. Where they tend to be worse are things like vectorized code which requires you to redesign algorithms such that they can use fast vector instructions, and even then, you'll have to resort to compiler intrinsics to use the instructions at all, and even then, compiler intrinsics can lead to some bad codegen. So your code winds up being non-portable, looks like assembly, and has some overhead just because of what the compiler emits (and can't optimize). So you wind up just writing it in asm anyway, and get smarter about things the compiler worries about like register allocation and out-of-order instructions. But the real problem once you get into this domain is that you simply cannot tell at a glance whether hand written assembly is "better" (insert your metric for "better here) than what the compiler emits. You must measure and benchmark, and those benchmarks have to be meaningful." - duped
Educational Content and Contribution
The discussion also includes feedback on the FFmpeg assembly lessons, with users suggesting improvements to make them more accessible to newcomers. Suggestions include starting with a quick introduction to assemblers like NASM and including prerequisite math lessons for better context. The intent of the lessons is also debated, with some seeing them as generic assembly introductions rather than directly relating to FFmpeg internals.
- "Shame this doesn't start with a quick introduction to running the examples with an actual assembler like NASM." - commandlinefan
- "Why not include the required or targeted math lessons needed for the FFmpeg Assembly Lessons in the GitHub repository? It'd be easier for people to get started if everything was in one place :)" - SilentM68
- "I was expecting to read pearls of wisdom gleaned from all the hard work done on the project, but Iâm not really getting how this relates to ffmpeg. The few chapters I saw seemed to be pretty generic intro to assembly language type stuff." - WhitneyLand
- "NTA but if the assumption is that the reader has only a basic understanding of C programming and wants to contribute to a video codec there is a lot of ground that needs to be covered just to get to how the cooley/tukey algorithm works and even that's just the basic fundamentals." - snickerbockers
- "I read the repo more as "go through this if you want to have a greater understanding of how things work on a lower level inside your computer". In other words, presumably it's not only intended for people who want to contribute to a video codec/other parts of ffmpeg. But I'm also NTA, so could be wrong." - byryan