HN Distilled

Essential insights from Hacker News discussions

The hidden JTAG in a Qualcomm/Snapdragon device’s USB port

Original Article

Hacker News Discussion

Here's a summary of the themes from the Hacker News discussion, with direct quotes where appropriate:

Debugging Accessibility and Security Concerns on Mobile Chipsets

A central theme is the contrast between older, less secure, and often cumbersome debugging methods on mobile chipsets, and the emerging trend of more integrated and accessible solutions like JTAG over USB. Users note that previous methods, such as Qualcomm's "hand-rolled set of read/write/execute primitives exposed over USB," were "hilariously undersecured" and allowed for unauthorized root access. MediaTek is mentioned as still employing older methods that require specific "incantations" to expose serial ports. The new approaches are seen as a positive step towards making debugging functionality accessible, but there's an expectation that manufacturers will disable these features in production devices.

  • mmastrac stated, "This is a much better experience than the previous Qualcomm debug experience, which was a hand-rolled set of read/write/execute primitives exposed over USB. It was hilariously undersecured, allowing a few of us to continually get root on various Qualcomm models."
  • mmastrac also commented, "In seriousness, these debug ports are seriously lacking in most mobile chipsets. MediaTek still has the old-style approach in many of their devices, requiring some incantations which expose serial over USB, but not in the way you think -- it's serial over USB pins!"
  • mmastrac further elaborated, "I've done tonnes of work with mobile chipsets and security and this seems like they've finally started down the road to making this functionality accessible. Don't be surprised if you don't see this supported out of the box in most places, though. Most OEMs will certainly disable this once they've adapted their bootloaders to it. The big G doesn't like debuggability in end user devices."

The Cost and Practicality of JTAG Probes

The discussion highlights the significant cost and logistical challenges associated with traditional JTAG debugging. Users point out that JTAG probes can cost thousands of dollars, representing a substantial investment that often requires lengthy requisition processes within companies. This can lead to situations where limited probes are "jealously guarded like priceless jewels" before a product ships, forcing teams to plan far in advance or compete for scarce resources. Furthermore, accessing JTAG ports on production units can be difficult, sometimes requiring "destructive entry" if the connector isn't exposed through the device enclosure.

  • Veserv noted, "Most of those boards have a separate physical JTAG connector (at least in development kits, this article indicates JTAG over USB is disabled in production systems anyways so no difference there) which is what they are expecting you to use for low-level debugging. It only costs like 1,000 $ for a JTAG probe which is like 1 fully-burdened engineer-day of cost. Even fully featured probes enabling hardware trace and time-travel debugging only cost like 1 engineer-week."
  • AlotOfReading shared, "The probes cost enough to exceed individual purchasing limits at hardware companies, which means you need to go through the requisition process. That takes long enough that you have to plan ahead and you don't order more as your needs increase. Then everyone's fighting for the limited probes right before a ship date and they get jealously guarded like priceless jewels."
  • AlotOfReading added, "JTAG also isn't usually exposed through enclosures, so using the probe on a field unit might require destructive entry depending on the application."
  • Veserv criticized, "Well the problem there is companies who are too stupid to invest in cheap tooling with massive ROI for their developers. A pretty constant problem in software development."
  • Veserv also edited to say, "The article even mentions how the 'Qualcomm Landing Team at Linaro', which seems to be the team that works with pre-production hardware to get them working on launch day, has a development process where 'debuggers have never been a staple of our work for all the typical reasons you'd expect (cost and complexity being the main ones)'."

JTAG over USB as a More Accessible Alternative

In contrast to traditional JTAG, the concept of JTAG over USB is viewed as a significant improvement in terms of accessibility and convenience, especially for use with production-ready units. This method allows debugging capabilities to be enabled and accessed through the standard USB connection, bypassing the need for physical JTAG connectors and expensive external probes. While acknowledging that this feature is often disabled in production systems, its existence on boards, even those with fused-out JTAG, is seen as valuable for specific prototyping phases.

  • bri3d argued, "There's generally an entire phase of prototyping where engineers will be using production boards but still need JTAG, which is why it's fused and why these kinds of features exist. It's a lot easier to have your lower-level software team (drivers/BSP, perf, etc.) sitting with production-ready units provisioned with engineering keys and debug enabled than to have them having to use some kind of case-off JTAG header setup, cost aside."
  • Veserv qualified, "And I am not knocking JTAG over USB. It is certainly convenient and beneficial since you can enable it in production or deployed units. They just do not have the cheap tools that are the intended way to access that capability."

Enabling Debugging Features on Specific Platforms (Pixel, Chromebook)

The discussion delves into specific implementations of debugging interfaces on particular platforms. For Google's Pixel devices, it's noted that serial debugging is available over the SBU pins and can be enabled by unlocking the device and using a fastboot command. On Chromebooks, a more advanced system uses the SBU pins with a specialized "SuzyQ" cable for a suite of debugging facilities, historically used for unbricking purposes. The ability to access or enable these features, especially on production units, is a key point of interest.

  • IAmLiterallyAB stated, "Google exposes serial Serial over the SBU pins on all the Pixel devices"
  • bri3d explained, "It's just a UART; you can use the UART to debug the device in various ways. On Pixel devices, the UART is not configured or brought up by default in locked production mode (as things should be), but by unlocking the device and then using fastboot oem uart enable you can flip the bits to turn it on."
  • bri3d continued, "By default I think it's still configured as the kernel console in the kernel command line, so once it's enabled it will show the kernel debug output and present a TTY."
  • bri3d also added, "On Chromebook devices there's a more complicated and fancy debugging system where the SBU pins can be muxed to the security processor's USB host interface by presenting a debug cable called a SuzyQ, which presents a whole suite of debugging facilities. This used to be used quite frequently for unbricking purposes."

Accessing Fuses and Low-Level Bootloaders (EDL, eFuses)

A technical aspect that emerges is the importance of understanding how fuses, especially eFuses, are read on production devices and the role of low-level bootloaders like Qualcomm's Emergency Download Mode (EDL). The ability to dump "qfprom fuses" is noted to specifically require an EL3 loader, and the privilege level of most device programmers for newer devices like the OnePlus 6 is questioned in this regard. The existence of EDL loaders for various production devices and the possibility of recompiling kernels to access /dev/mem for fuse reading are also discussed.

  • tripdout questioned, "How are they reading the eFuses on a production OnePlus 6? Do they have a Qualcomm-signed EL3 EDL loader?"
  • tripdout further elaborated, "But reading QFUSES specifically requires an EL3 loader 'edl qfp qfp.bin -> To dump qfprom fuses (only on EL3 loaders)' and I don't believe most devices programmers (especially as relatively new as the OnePlus 6) run under that privilege level."
  • zorgmonkey pointed to resources, noting, "yeah EDL loaders for a bunch of production devices exist here [0] also more on various XDA Forum posts for stuff like unbricking guides."

Analogies to Other Vendor Debugging (Apple)

The discussion briefly touches upon the idea that similar "magical USB shenanigans" for debugging are also present with other vendors, with Apple being cited as an example. The "Chimp Cable" is mentioned as a specific instance of Apple's complex debugging interfaces.