Data Transformation

I'm not very happy with this being in the library and it may be phase out/removed in future versions. I believe FluentStorage should be focussed on being the best polycloud storage library and it should not go into tangental directions. I feel that encryption is way out of scope, and the way sinks/transformations are implemented causes a lot of headache and frustration because there are too many APIs.

Are we saying that ALL polycloud API will work seamlessly and magically encrypt/decrypt data across ALL providers? What a monumental task to support. And for what? Data transformation is an endless sea of possibilities, and I don't like how few of them are implemented here, putting additional pressure on an already strained implementation. This is part of a legacy set of features inherited from Storage.NET and I don't like the way Ivan made it a part of a storage library.

If compression should be implemented, then it should be an addon package called FluentStorage.Compression which will use a strong library like SharpCompress to support a ton of formats, plus supporting native 7-zip which is the fastest possible compression/decompression engine to use for large files since it works with native C++ code. The current implementation is wrong, sub-par and is not part of my vision.

• GZip Compression
• AES Encryption
• Rijndael Encryption
• Custom Sinks

Transform sinks is another awesome feature of FluentStorage that works across all the storage providers. Transform sinks allow you to transform data stream for both upload and download to somehow transform the underlying stream of data. Examples of transform sinks would be gzipping data transparently, encrypting it, and so on.

Let's say you would like to gzip all of the files that you upload/download to a storage. You can do that in the following way:

IStore myGzippedStorage = {StorageFactory}
   .FromXXX()
   .WithGzipCompression();

Then use the storage as you would before - all the data is compressed as you write it (with any WriteXXX method) and decompressed as you read it (with any ReadXXX method).

Due to the nature of the transforms, they can change both the underlying data, and stream size, therefore there is an issue with storage providers, as they need to know beforehand the size of the blob you are uploading. The matter becomes more complicated when some implementations need to calculate other statistics of the data before uploading i.e. hash, CRC and so on. Therefore the only reliable way to stream transformed data is to actually perform all of the transofrms, and then upload it. In this implementation, FluentStorage uses in-memory transforms to achieve this, however does it extremely efficiently by using Microsoft.IO.RecyclableMemoryStream package that performs memory pooling and reclaiming for you so that you don't need to worry about software slowdows. You can read more about this technique here.

This also means that today a transform sink can upload a stream only as large as the amount of RAM available on your machine. I am, however, thinking of ways to go further than that, and there are some beta implementations available that might see the light soon.

To create the sink, call extension method WithGzipCompression and optionally pass a compression level which defaults to Optimal:

IStore store = {StorageFactory}
   .FromXXX()
   .WithGzipCompression(CompressionLevel compressionLevel = CompressionLevel.Optimal)

This sink implements symmetric encryption for upload/download data. I.e. uploaded data is encrypted with a key, and decrypted after download.

It uses AES encryption with default settings. You control which Key and IV are used.

To add:

IStore store = {StorageFactory}
   .FromXXX()
   .WithAesSymmetricEncryption(string encryptionKey, string encryptionSecret)

Note: Rijndael is obsolete in .NET 6 and beyond!

This sink implements symmetric encryption for upload/download data. I.e. uploaded data is encrypted with a key, and decrypted after download.

It uses Rijndael encryption with default settings, which is a superset of AES encryption algorithm (read about differences). You control which Key and IV are used.

To add:

IStore store = {StorageFactory}
   .FromXXX()
   .WithSymmetricEncryption(string encryptionKey, string encryptionSecret)

The encryption key is a baase64 encoded binary key. To generate it, you can use the following snippet:

void Main()
{
	var cs = new RijndaelManaged();
	cs.GenerateKey();
	string keyBase64 = Convert.ToBase64String(cs.Key);
	
	Console.WriteLine("new encryption key:" + keyBase64);
}

Note that it's your own responsibility to store the key securely, make sure it's not put in plaintext anywhere it can be stoken from!