The conversation

zug_zugHNSource author
Lessons learnedExperience excerpt · English

Absolutely, when I worked at (semi-well-known unicorn) a half-dozen years ago on the …

Absolutely, when I worked at (semi-well-known unicorn) a half-dozen years ago on the data-engineering team the manager told me "Hey we want to use spark next quarter, that's a huge initiative." And I immediately asked, "in what capacity?" And the answer was don't-know/doesn't-matter, it's just important that we can say we're using it. I really wish I understood where that was coming from (his manager resume-building? somebody getting a kickback?)

Original discussionCommand-line Tools can be 235x Faster than your Hadoop Cluster (2014)276 source comments · at import
3

What was this responding to?

Parent context · original excerpt

BE
benrutterHN

This times a zillion! I think there's been a huge industry push to convince managers and more junior engineers that spark and distributed tools are the correct way to do data engineering . I think its a similar pattern to web dev influencers have convinced everyone to build huge hydrated-spa-framework-craziness where a static site would do. My advice to get out of this mess: - Managers, don't ask for specific solutions (spark, react). Ask for clever engineers to solve problems and optimise / track what you vare about (cost, performance etc). You hired them to know best, and they probably do. - Technical leads, if your manager is saying "what about hyperscale?" You don't have to say "our existing solution will scale forever". It's fine to say, "our pipelines handle dataset up to 20GB, we don't expect to see anything larger soon, and if we do we'll do x/y/z to meet that scale". Your manage…

Full original

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 3

Direct replies available in this import. Original-language excerpts.

SP
spauldoHN

That's when you rewrite your codebase in the SPARK dialect of Ada and play innocent when your management questions you about it.

Full original
CO
coliveiraHN

They'll never say it's resume building or kickbacks, they'll invent some technically sounding and/or business reason to achieve the same result.

Full original
TH
thwartedHN

The most interesting part is that you can say you're doing/using something entirely independent of if you actually are. Sure, that's a lie, but so is only using something so you can say you're using it (sure, they admitted to you that was the reason, but that won't be the reason they put on LinkedIn).

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.