At the annual Cloud Technology Spring Festival Gala - Amazon Web Services' Re:Invent, in the most watched CEO keynote speech, the first product released was actually a storage product, which was Amazon S3 Express One Zone.
At the same time, I found out that this is a storage product that violates advertising laws.
Amazon Web Services CEO Adam announced that Amazon S3 Express One Zone is currently the fastest object storage solution on the cloud.
Accelerated acceleration, ultra-low latency object storage is coming!
Amazon S3 Express One Zone is an enhanced version of S3 object storage that delivers up to 10x the performance of Standard S3, capable of processing hundreds of thousands of requests per second (the exact number is not yet known) with a constant latency of a few milliseconds.
It is mainly used for latency-sensitive workloads such as financial transaction analysis, real-time ad recommendation, fraud detection, machine learning, Xi training, etc., which often require millions of data accesses in a short period of time and high latency requirements.
Previously, in order to provide this performance, users needed to move data from S3 to some cache acceleration solution. However, this is when users need to manage multiple storage infrastructures, and things can be complicated.
And now, with Amazon S3 Express One Zone.
In fact, it uses dedicated hardware and software to accelerate it, and it is stored in a single Availability Zone, allowing users to manually choose to place hot data particularly close to high-performance computing power, including EC2, ECS, and EKS, to better reduce latency.
I remember Amazon Web Services acquired the all-flash startup E8 in 2019, and since then, I haven't heard about E8 again. As for whether the technology of E8 was used this time. You guess!?
Low latency improves the performance of small objects
Thanks to its low latency, Amazon S3 Express One Zone delivers significantly better performance for small objects.
In general, object storage may not perform as well as other storage systems when handling large numbers of small files, and frequently reading and writing large numbers of small files can result in degraded storage system performance.
This is because the object storage system has a certain amount of latency overhead in processing metadata when processing each object. In addition, it takes a certain amount of time to read or write an object.
If the size of the object is relatively small, then the basic latency overhead accounts for a higher proportion of the entire process, the proportion of system overhead is higher, and the read and write bandwidth at this time is also relatively low.
When the object size is relatively large, most of the time in the entire read and write process is to transmit data, the proportion of overhead is less, and the read and write bandwidth will be higher.
Therefore, when the system latency is relatively low, small objects benefit significantly, and the performance can be improved by up to 10 times compared to the standard version of S3.
The low latency of S3 helps users reduce costs
Higher-performance tiers of S3 object storage usually mean higher costs (more information at the end of this article), but really, what Amazon Web Services wants to say is that it saves money!
When users use burst and on-demand instances, if they are paired with extremely low-latency S3 storage, they can process tasks faster and release these computing resources as soon as possible, thereby reducing costs.
That's right, it's not the cost of storage that is saved here, it's the cost of cloud hosting.
When faced with data-intensive workloads, the overall system uptime can be dramatically reduced, especially when hundreds or thousands of machines are required to work together, which can lead to significant cost reductions.
Adam shared a Pinterest story in which write speeds were increased by 10x while the overall cost was reduced by 40%.
Launched in 2006, Amazon S3 changed the way people consume and use storage, with the ability to access any amount of data anytime, anywhere.
Convenience is really convenient, expensive is really expensive, in order to reduce costs, Amazon Web Services has a lot of operations before and after.
For example, Amazon S3 Deep Archive, a lower-cost archive storage that was later released, reduced the cost to less than 1 cent per gigabyte.
Later, as the number of Amazon S3 layers increased, Amazon S3 Intelligent Tiling was later introduced, which automatically helps users migrate data between layers.
Adam said that since the release of Smart Tiering, users have saved more than $2 billion, which is a significant number.
Something special about Amazon S3 Express One Zone
Amazon S3 Express One Zone supports common S3 API functions, as well as a range of S3 features.
But unlike other S3s, it has a new bucket type (directory bucket), a new authentication model, and a new bucket naming convention.
The image above is a diagram shared by Jeff Bar in his blog about putting data from an existing S3 bucket into a newly created Amazon S3 Express One Zone directory bucket.
After that, it will perform its own batch processing operations and copy the data to a high-performance bucket.
Currently, Amazon S3 Express One Zone is supported by Amazon SageMaker Model Training, Amazon Athena, Amazon EMR, and AWS Glue Data Catalog.
Currently, US East (n.Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Stockholm) are available, but not in China.
Although the data is only placed in one Availability Zone, it is much more expensive than the standard S3 (0.16 vs 0.023), after all, the performance is high, and the expensive point is okay, isn't it? What's more, the cost of data access is 50% lower than that of the standard S3!
Finally, it is important to note that it only exists in a single Availability Zone, and if a single Availability Zone fails, data may be lost.
Jeff Bar blog introduction to the original article: