Show HN: Spegel, a Terminal Browser That Uses LLMs to Rewrite Webpages

This discussion revolves around a tool proposed as an alternative way to browse the web, using Large Language Models (LLMs) to process and present web content in a more user-friendly, often markdown-based, format. The themes that emerge highlight both the potential benefits and significant challenges of such an approach.

Reimagining the Browser and User Experience

A central theme is the potential for LLM-powered tools to create a new paradigm for web consumption, moving away from the traditional "show me everything pixel-perfect" browser. Users envision personalized, intelligent agents that understand individual needs and prior knowledge, offering curated and summarized content. There's a strong sentiment that this aligns with the original vision of user agents acting on behalf of the user, rather than prioritizing content author preferences.

• Personalization and Curation: The idea is to have the tool act as a personalized agent that filters and processes information based on user preferences and existing knowledge. As myfonj puts it, "It seems we're approaching the moment when our individual personal agent, when asked about a new page, will tell us: 'Well, there's nothing new of interest for you, frankly...'" This vision extends to understanding what a user already "knows" from previous interactions or marked content.
• Beyond Pixel Matching: The discussion contrasts current browsers, which aim for visual fidelity across devices, with the proposed approach. myfonj notes, "The fact that browsers nowadays are usually expected to represent something 'pixel-perfect' to everyone with similar devices is utterly against the original intention." The new paradigm focuses on content processing and user understanding.
• Accessibility: The potential for LLM-driven tools to aid accessibility is also mentioned. andoando shares, "I've been thinking of doing this exactly this, but for as a screen reader for accessibility reasons."

The Role of LLMs in Content Processing

The utility and capabilities of LLMs themselves are a major focus. Participants debate whether LLMs are essential for the task or if simpler methods suffice, while also acknowledging their power in transforming content and circumventing SEO-driven websites.

• LLMs as Filters/Transformers: Many users see LLMs as powerful tools for stripping away "cruft" and "SEO slop" from web pages, presenting cleaner, more digestible content. mromanuk praises this, stating, "I definitely like the LLM in the middle, it’s a nice way to circumvent the SEO machine and how Google has optimized writing in recent years. Removing all the cruft from a recipe is a brilliant case for an LLM. And I suspect more of this is coming: LLMs to filter."
• Efficiency and Alternatives: There's discussion about the efficiency of using LLMs for simple tasks like HTML to Markdown conversion. insane_dreamer questions, "Interesting, but why round-trip through an LLM just to convert HTML to Markdown?" The suggestion of using tools like Pandoc or headless browsers (Puppeteer, Selenium) for initial rendering and then an LLM for summarization is raised as a potentially more computationally efficient alternative. markstos elaborates, "That's not to say you need an LLM, there are projects like Puppeteer that are like headless browsers that can return the rendered HTML, which can then be sent through an HTML to Markdown filter. That would be less computationally intensive."
• LLM Limitations and Hallucinations: The inherent unreliability and tendency for LLMs to "hallucinate" are a significant concern. A critical example emerges when mossTechnician points out a major recipe alteration: "Pounds of lamb become kilograms (more than doubling the quantity of meat), a medium onion turns large, one celery stalk becomes two, six cloves of garlic turn into four, tomato paste vanishes, we lose nearly half a cup of wine, beef stock gets an extra ¾ cup, rosemary is replaced with oregano." simedw later confirms this was due to content truncation leading to hallucination. andrepd strongly criticizes this, saying, "You have an algorithm that rewrites textA to textB (so nice), where textB potentially has no relation to textB (oh no). Were it anything else this would mean 'you don't have an algorithm to rewrite textA to textB', but for gen ai? Apparently this is not a fatal flaw, it's not even a flaw at all!"

JavaScript and Dynamic Web Content

The pervasive use of JavaScript to render modern web pages presents a major technical hurdle. The discussion explores how tools like this can handle Single Page Applications (SPAs) and dynamic content.

• The JavaScript Problem: Many users highlight that modern websites rely heavily on JavaScript, making simple HTML parsing insufficient. anonu asks, "Don't you need javascript to make most webpages useful?" jazzyjackson agrees, suggesting "to even get to the content on a lot of SPAs this would need to be running a headless browser to render the page, before extracting the static content unfortunately."
• Handling Dynamic Content: Solutions proposed include using headless browsers like Puppeteer or Selenium to render pages with JavaScript before processing. 098799 sketches this approach: "Here's a sketch: ... -- selenium drives your actual browser under the hood." deepdarkforest also notes, "The main problem with these approaches is that most sites now are useless without JS or having access to the accessibility tree."
• Alternative Approaches: Some argue that JavaScript is not inherently necessary for useful webpages, with inetknght stating, "The web was useful for long before javascript was around. I literally hate javascript -- not the language itself but the way it is used." However, others clarify that while possible to create without JS, many existing sites depend on it for functionality.

Extensibility and Future Functionality

Participants suggest various ways to expand the tool's capabilities, ranging from handling more complex web interactions to integrating with other command-line tools and environments.

• Enhanced Web Interaction: Suggestions include adding support for POST requests and scripting. wrsh07 comments, "Handling post requests, enabling scripting, etc could all be super powerful."
• Integration with Other Tools: The idea of integrating with terminal-based browsers like Emacs' Eww or Lynx is floated. pepperonipboy suggests, "Could work great with emacs' eww!" and sammy0910 shares a similar project for Emacs.
• Caching and Collaboration: The concept of caching processed content for reuse, potentially in a peer-to-peer fashion, is discussed. __MatrixMan__ ponders, "It would be cool of it were smart enough to figure out whether it was necessary to rewrite the page on every visit. There's a large chunk of the web where one of us could visit once, rewrite to markdown, and then serve the cleaned up version to each other without requiring a distinct rebuild on each visit."

The Broader Implication of LLM-Mediated Browsing

The discussion touches upon the meta-implications of using LLMs to mediate web access, touching on the potential for new forms of abstraction, the economics of LLM usage, and the philosophical shift in how we interact with information.

• Abstraction and "Broken Systems": One perspective is that this is "another layer of abstraction on top of an already broken system." b0a04gl states, "this is another layer of abstraction on top of an already broken system. you're running html through an llm to get markdown that gets rendered in a terminal browser? that's like... three format conversions just to read text." However, others see this as a valid response to a broken system (MangoToupe), or simply part of modern computing.
• Cost and Efficiency: The cost of using LLMs, particularly at scale, is a concern. hirako2000 asks, "Do you also like what it costs you to browse the web via an LLM potentially swallowing millions of tokens per minutes?"
• The "Bubble" Effect: A more critical observation is the potential for LLM-mediated browsing to create personalized "bubbles" that reinforce existing views, offering a "rose coloured glasses" experience. Bluestein expresses this concern: " this is where the 'bubble' seals itself 'from the inside' and custom (or cloud, biased) LLMs sear the 'bubble' in.- The ultimate rose (or red, or blue or black ...) coloured glasses.-"