The conversation

Business & incomeExperience excerpt · English

>Big Data today is not: I don't have enough storage or compute

>Big Data today is not: I don't have enough storage or compute. It is for me. Six times per year I go out to the field for two weeks to do data acquisition. In the field we do a dual-aircraft synthetic aperture radar collection over four bands and dual polarities. That means two aircraft each with one radar system containing eight 20TiB 16-drive RAID-0 SSD storage devices. We don't usually fill up the RAIDs so we generate about 176TiB of data per day and over the two weeks we do 7 flight, or 1.2PiB per deployment or 7.2PiB per year. We can only fly every other day because it takes a day between flights to offload the data via fiber onto storage servers that are usually haphazardly crammed into the corner of a hangar next to the apron. It is then duplicated to a second server for safekeeping and at the end of the mission everything is shipped back to our HQ for storage and processing. The data is valuable, but not "billions" valuable. …

Original discussionBig data is dead (2023)457 source comments · at import
3

What was this responding to?

Parent context · original excerpt

ZU

This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.

Full original

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 3

Direct replies available in this import. Original-language excerpts.

SR

Would https://docs.daos.io/v2.4 make sense for your use case (assuming affordable and available) ?

Full original
ED

Why not Minio? Yes, the network will still be the bottleneck but you can use whatever he works best and it is remarkably simple.

Full original
YA

> We can only fly every other day because it takes a day between flights to offload the data via fiber onto storage servers What’s preventing you from buying a second set of SSD drives and swapping them into planes end of day? The planes would then be able to fly every day while the previous day’s data is being offloaded on the ground, condensing the two weeks into one. Swaps could even be quite easy if you replace full enclosures rather than each drive individually. I'm probably missing something, though!

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.