This Hacker News discussion revolves around several key themes concerning Amazon Web Services (AWS), primarily focusing on S3 and VPC interactions, but also touching upon the evolving nature of AWS services and their associated costs and complexities.
S3 Key Prefixing and Hotspot Avoidance
A central point of contention is the necessity of randomizing the first part of S3 object keys to avoid hotspots. While the original poster suggested this is no longer strictly true, discussion participants offered differing views, highlighting that the auto-balancing mechanisms are not instantaneous and bursts of writes with the same key prefix can still lead to throttling. AWS staff themselves have reportedly recommended different prefixes even recently, suggesting a potential lag in their own guidance keeping pace with service changes.
- One user, cldcntrl, stated, "You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots."
- In response, cldcntrl elaborated, "The whole auto-balancing thing isn't instant. If you have a burst of writes with the same key prefix, you'll get throttled."
- hnlmorg shared an experience: "Not the OP but I’ve had AWS-staff recommend different prefixes even as recently as last year. If key prefixes don’t matter much any more, then it’s a very recent change that I’ve missed."
- cldcntrl corroborated this, saying, "That's right, same for me as of only a few months ago."
- williamdclt offered a counterpoint: "Might just be that the AWS staff wasn't up to date on this."
- hnlmorg countered again, "That’s possible but they did consult with the storage team prior to our consultation. But I don’t know what conversations did or did not happen behind the scenes."
AWS's Pricing and Complexity Strategy Leveraging User Ignorance
A significant portion of the discussion criticizes AWS for what is perceived as a deliberate strategy of leveraging user knowledge gaps for profit, particularly concerning VPCs, NAT Gateways, and VPC endpoints. Users expressed frustration that common, beneficial configurations, like traffic staying within the datacenter when accessing S3 from a VPC in the same region, incur unnecessary NAT Gateway costs by default. The requirement to opt into VPC endpoints, often described as an "absurd" and "baffling" design choice, is seen as a way to monetize basic connectivity and trap less informed users.
- SOLAR_FIELDS articulated this frustration: "You know what's still stupid? That if you have an S3 bucket in the same region as your VPC that you will get billed on your NAT Gateway to send data out to the public internet and right back in to the same datacenter. There is simply no reason to not default that behavior to opt out vs opt in (via a VPC endpoint) beyond AWS profiting off of people's lack of knowledge in this realm. The amount of people who would want the current opt-in behavior is... if not zero, infinitesimally small."
- andrewmcwatters broadly commented, "Everything about AWS is oriented to take advantage of people sleeping, how is this any surprise?"
- afandian shared the pain of configuration: "Having experienced the joy of setting up VPC, subnets and PrivateLink endpoints the whole thing just seems absurd. They spent the effort of branding private VPC endpoints "PrivateLink". Maybe it took some engineering effort on their part, but it should be the default out of the box, and an entirely unremarkable feature."
- afandian also raised a point about necessity: "In fact, I think if you have private subnets, the only way to use S3 etc is Private Link (correct me if I'm wrong)."
- tux3 framed this as "price segmentation," suggesting, "People who are price insensitive will not invest the time to fix it."
- nbngeorcjhe rephrased this aggressively: "This just sounds like a polite way of saying 'we're taking peoples' money in exchange for nothing of value, and we can get away with it because they don't know any better'."
- dmart advocated for a simpler approach: "VPC endpoints in general should be free and enabled by default. That you need to pay extra to reach AWS' own API endpoints from your VPC feels egregious."
Evolving Security Defaults and the Challenge of Publicly Accessible S3
The discussion also touches upon AWS's move to make "Block Public Access" the default for new S3 buckets. While universally acknowledged as a positive security measure given the history of data breaches, it creates friction for users who legitimately need to serve files publicly. The challenge lies not just in the existence of hurdles, but in their constant shifting, making it difficult to maintain a consistent process for enabling public access.
- simonw noted the dual nature of the change: "S3: 'Block Public Access is now enabled by default on new buckets.' On the one hand, this is obviously the right decision. The number of giant data breeches caused by incorrectly configured S3 buckets is enormous. But... every year or so I find myself wanting to create an S3 bucket with public read access to I can serve files out of it. And every time I need to do that I find something has changed and my old recipe doesn't work any more and I have to figure it out again from scratch!"
- SOLAR_FIELDS expressed a different perspective: "I honestly don't mind that you have to jump through hurdles to make your bucket publically available and that it's annoying. That to me seems like a feature, not a bug."
- simonw countered the "feature" aspect: "Sure... but last time I needed to jump through those hurdles I lost nearly an hour to them! I'm still not sure I know how to do it if I need to again."
- dghlsakjl summarized the core issue for users like simonw: "I think the OPs objection is not that hurdles exist but that they move them every time you try and run the track."
The Evolving Landscape of AWS Networking Options and Their Cost/Complexity Trade-offs
A brief exchange occurred regarding the evolution of AWS networking solutions, with Transit Gateway (TGW) and VPC sharing being mentioned as improvements over older methods like VPC peering. However, the cost implications and additional complexity of these newer services were also highlighted.
- aaronblohowiak questioned the cost-effectiveness: ">VPC peering used to be annoying; now there are better options like Transit Gateway, VPC sharing between accounts, resource sharing between accounts, and Cloud WAN. TGW is... twice as expensive as vpc peering?"
- alFReD-NSH pointed out the interconnectedness of cost and architecture: "And vpc sharing is free. Cost and architecture are tied."
- Hikikomori provided more detail on TGW costs: "More than twice as same AZ is free with peering. But if you're big enough you can get better deals on cost. But unlike peering TGW traffic flows through an additional compute layer so it has additional cost."
General Observations on AWS Documentation and Service Evolution
The discussion concluded with a couple of broader points about AWS, including the desire for better linkage between claims and supporting documentation, and a humorous take on how quickly knowledge about AWS can become outdated. There was also a technical query about the actual mechanics of Glacier restores, questioning the premise that they are no longer "painfully slow" without considering the underlying hardware.
- gurjeet expressed a common user wish: "It would've been nice if each of those claims in the article also linked to either the relevant announcement or to the documentation. If I'm interested in any of these headline items, I'd like to learn more."
- bob1029 challenged a statement about Glacier restores: "> Glacier restores are also no longer painfully slow. Wouldn't this always depend on the length of the queue to access the robotic tape library? Once your tape is loaded it should move really quickly:" followed by a link to IBM tape library specifications.