>Big Data today is not: I don't have enough storage or compute
>Big Data today is not: I don't have enough storage or compute. It is for me. Six times per year I go out to the field for two weeks to do data acquisition. In the field we do a dual-aircraft synthetic aperture radar collection over four bands and dual polarities. That means two aircraft each with one radar system containing eight 20TiB 16-drive RAID-0 SSD storage devices. We don't usually fill up the RAIDs so we generate about 176TiB of data per day and over the two weeks we do 7 flight, or 1.2PiB per deployment or 7.2PiB per year. We can only fly every other day because it takes a day between flights to offload the data via fiber onto storage servers that are usually haphazardly crammed into the corner of a hangar next to the apron. It is then duplicated to a second server for safekeeping and at the end of the mission everything is shipped back to our HQ for storage and processing. The data is valuable, but not "billions" valuable. …
Original discussionBig data is dead (2023)457 source comments · at import