ZipFile OpenEncrypted for read not working

[kbar]

I have written out an encrypted zip file using ZipFile->OpenEncrypted. But that function is not able to read it.

AutoAlloc<ZipFile> zip;
if (zip->OpenEncrypted(libFile, false, key, keyLength, blockSize, 0, ZIP_APPEND_CREATE))
{
zip->Close();
}

if (zip->OpenEncrypted(libFile, true, key, keyLength, blockSize, 0, ZIP_APPEND_CREATE))
{
        //Never gets here.
	zip->Close();
}

The second parameter is supposed to tell it to open for reading. But OpenEncrypted always returns false.

Note that I am passing in the correct key, keylength and blocksize since I am using them with AESFile and it works fine there. But with ZipFile I am unable to open them for reading.

Is this a bug?

I am using this in R21 but would like to use it in R20 upwards.

I also see zip file abilities in the new Maxon API. I would be happy to use that instead if it supports encryption. But there is no documentation examples that show how to use the Maxon API.

[kbar]

Actually upon further investigation it doesn't look like OpenEncrypted is able to create an encrypted zip file at all. It just creates a regular zip file that can be extracted using any tool. IE no encryption.

Any insights into how this method should work are appreciated.

[ferdinand]

Hello @kbar,

thank you for reaching out to us. The second call you do is in principle correct. The lib_zipfile.h library is part of the old/classic API and the method OpenEncrypted was only used by one feature of Cinema which now dormant, the updater. The internal call was similar to the call I am doing here:

const char* password = "9USGxEPo0Tx6d8ZRRLbpEc4D88xdU2bb";
int keyLength = 256;
int blockLength = 256; // That is indeed the block size that is defined internally
if(!zipFile->OpenEncrypted(fileName, true, password, keyLength, blockLength, 0)) 
{
    ApplicationOutput("Could not decrpyt file"_s);
}

But just as for you, this did not work for me, i.e., OpenEncrypted is always returning false. Noteworthy is also that OpenEncrypted is only supporting zip-files with AES encryption, not the older and zip-native ZipCrypto form. I have created 128bit and 256bit AES encrypted zip files with both 7zip and WinZip (which both claim to be ISO-conformant) and could not open them. What also irritates me a bit is that the method is asking for a block size while AES has a fixed block size of 128 bit. The internal call does pass 256 bit for the block size in its call, but I also tried 128 with no luck.

As you already also did point out, the ZipFile type has no method to write such AES encrypted files. There is the AESFile type with which we can encrypt a file, but there is no public or private interface to merge that 'data' with a ZipFile. Which in sum leads me to the conclusion that OpenEncrypted should never have been public, as it was likely neither designed nor tested for dealing with anything else than Cinema 4D's old updater files. The method(s) for writing these update files are not in the Cinema 4D codebase as far as I can tell. But I would assume that they did deviate from the zip ISO-standard, since I can read AES encrypted zip files created with 7zip with Winzip (but not vice versa). This ties into the old zip topic of yours and the realization that the zip standard is a bit of a mess, and everyone is doing whatever he or she wants. All this combined makes the encryption feature inaccessible, i.e., private, since one can neither write such files nor can Cinema read externally provided files.

However, writing and reading encrypted zip files should be possible with maxon::StreamConversions (link), but it will take me some time to provide an example. This feature will however not be accessible in R21, i.e., might be useless to you.

Cheers,
zipit

[kbar]

This post is deleted!

[ferdinand]

Hey @kbar,

I have not forgotten this thread; I was just on vacation and this got a bit reduced in priority since you said that you have found a solution. Since then, you have hidden this answer, which I am seeing just now. Example is coming up and sorry for the long wait.

Cheers,
Ferdinand

[kbar]

Don’t worry about the example. I am fine using the SDK methods. I won’t use the new MAXON API for this. So you can mark this as solved if you would like, since the initial question was answered. Thanks, Kent.

[ferdinand]

Hello @kbar,

in case you are interested in it for academic purposes or someone else is, here is a full example for how to use some stuff within the maxon::StreamConversions namespace.

I included examples for hashing, encrypting, and compressing inputs. The example also reverses the operation for compression and encryption. This might be helpful, because there is a bug in the Stream Conversions Manual AES encryption example code, which cause the code to fail due to not filling up the plain text message to the correct block size. There are also some minor bits and bobs which might not be obvious.

I have reached out to the developers about some problems of mine regarding the example code in the docs (which was provided by them, not us) and how Cinema deals with the term block size in the context of AES encryptions. My variant does run and does what I would expect it to do. I will report back here if there are any relevant updates (which I would not anticipate being the case).

Cheers,
Ferdinand

The output:

Password: wubba lubba dub-dub, Secret: d7a2f8231c17aabc885683fb4781a648

Content: Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd  guber, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum.

Encrypted content: {0,0,3,22,120,-38,-43,-110,49,110,3,49,12,4,-65,-78,15,48,-18,15,-23,-35,-27,5,60,-119,56,48,-106,-60,-117,72,30,-32,-33,-121,118,82,36,-123,-35,-91,112,-53,37,119,23,3,-98,117,114,-121,-20,22,29,85,-101,78,-104,56,-88,-77,-97,80,116,24,59,123,-28,-112,106,-18,20,25,27,-72,-119,-49,19,-116,43,-86,80,-57,-48,17,-3,10,-106,-39,-75,-62,-71,-17,105,34,-29,-112,26,-61,17,-114,70,107,-122,-128,-3,59,-128,-47,105,27,4,106,-14,25,-41,52,-32,73,-2,-53,-17,-48,22,-69,7,45,120,115,28,60,21,-84,118,-69,-90,82,-62,110,-5,-114,-113,48,87,-44,-48,31,-53,-69,-50,-124,-55,107,-12,5,-17,89,26,37,123,18,46,100,21,-40,98,-27,-20,60,52,99,8,78,23,-23,-108,-94,-47,40,30,121,108,-114,-13,67,16,-53,19,-19,15,-92,23,36,115,7,-77,77,30,-1,14,-25,-91,62,-24,11,106,-106,31,-35,0,94,-62,-56,-61,-127,58,-108,-54,89,-93,-64,10,-42}

Decrypted content: Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd  guber, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum.

The code:

// Example for a maxon API conformant implementation to encrypt and compress text data.            
// 
// This mostly showcases the maxon::StreamConversions namespace which contains operations for
// streams of data, not just text. The example showcases the three important functionalities of
// compression, encryption and hashing of StreamConversions.
// 
// This will effectively take an input string and compress and then encrypt it, and after that
// reverse the operation. I deliberately left out file operations here, to make this a bit more
// streamlined. File operations would have to be done with maxon::BaseStreamInterface types (in-
// and output streams) within the maxon API.
// 
// As discussed in:
//     https://developers.maxon.net/forum/topic/13449

#include "c4d.h"
#include "c4d_symbols.h"
#include "maxon/cryptography.h"
#include "maxon/cryptography_hash.h"
#include "maxon/datacompression.h"
#include "maxon/iostreams.h"
#include "maxon/secure_random.h"
#include "maxon/streamconversion.h"

/// -----------------------------------------------------------------------------------------------
/// Hashes a password into a 32 characters secret.
/// 
/// Could also be done with maxon::GetPasswordHash() and maxon::HashPasswordWithSalt() in a more
/// convenient form.
/// 
/// @param[in] password The password to hash.
/// @return The hashed password.
/// -----------------------------------------------------------------------------------------------
maxon::Result<maxon::String>GetAES128Secret(const maxon::String password)
{
  iferr_scope;

  // The input password and a temporary hash result as arrays.
  const maxon::BaseArray<maxon::Char> secret = password.GetCString() iferr_return;
  maxon::BaseArray<maxon::UChar> hash;

  // Use the StreamConverions MD5 hash to hash the password into a 32 characters secret. 
  const maxon::StreamConversionRef md5 = maxon::StreamConversions::HashMD5().Create() iferr_return;
  md5.ConvertAll(secret, hash) iferr_return;
  // Get the hash as a hex string.
  const maxon::String result = maxon::GetHashString(hash) iferr_return;

  return result;
}

/// -----------------------------------------------------------------------------------------------
/// Compresses and encrypts an input string.
/// 
/// @param[in] content The input text to compress and encrypt.
/// @param[in] secret The encryption secret.
/// @return The compressed and encrypted content as a char array.
/// -----------------------------------------------------------------------------------------------
maxon::Result<maxon::BaseArray<maxon::Char>>GetEncrypted(
  const maxon::String content,const maxon::String secret)
{
  iferr_scope;

  // The in- and output as char arrays.
  maxon::BaseArray<maxon::Char> plaintextContent = content.GetCString() iferr_return;
  maxon::BaseArray<maxon::Char> encryptedContent;

  // The compression settings, the compression rate can lie between 0 and 9, where 9 is the
  // strongest (and slowest) compression.
  maxon::DataDictionary settings;
  settings.Set(maxon::STREAMCONVERSION::ZIP::ENCODER::COMPRESSION, maxon::Int(9)) iferr_return;
  settings.Set(maxon::STREAMCONVERSION::ZIP::ENCODER::WRITESIZE, true) iferr_return;

  // Build a CryptoKey for an AES encryption.
  const maxon::Id encoderID = maxon::StreamConversions::AesEncoder.GetId();
  const maxon::Int blockSize = 128;
  const maxon::Int keySize = 128;
  const maxon::BaseArray<maxon::Char> key = secret.GetCString() iferr_return;
  const maxon::CryptoKey cryptoKey (encoderID, blockSize, key.GetFirst(), keySize);
  maxon::DataDictionary cryptoSettings;
  cryptoSettings.Set(maxon::CryptographyOptions::CRYPTOKEY, cryptoKey) iferr_return;

  // We can either compress first and then encrypt the data or do it the other way around. I did 
  // choose to compress first, since trying to decompress data first is relatively common attack
  // pattern. Encrypting first and then compressing might yield more favorable compression rates. 
  // This will be most impactful when the plain text is relatively small, due to us filling 
  // up the plain text message up to the AES encryption block size further down below. So, if the 
  // compressed message happens to be 17 bytes long, the code below will fill it up to 32 bytes.
  // If we compress after encrypting, we will also compress that "filling up". But since the 
  // maximum overhead due to this is 15 bytes, it can be ignored IMHO.
  const maxon::StreamConversionRef zipEncoder = 
    maxon::StreamConversions::ZipEncoder().Create(settings) iferr_return;
  zipEncoder.ConvertAll(plaintextContent, encryptedContent) iferr_return;

  // Make sure the compressed data is null terminated. We are going to have to fill up this data to
  // the fixed block size of an AES encryption. And we are going to use this null as a marker when
  // we later decrypt the data and then have to separate the message from the "fill-up"-content.
  encryptedContent.Append(0) iferr_return;

  // Setup an AES encoder.
  const maxon::StreamConversionRef aesEncoder = 
    maxon::StreamConversions::AesEncoder().Create(cryptoSettings) iferr_return;

  // Determine if the data does match the fixed block size (128 bits) of an AES encryption.
  const maxon::Int initialSize = encryptedContent.GetCount();
  const maxon::Int aesBlockSize = aesEncoder.GetBlockSize();
  const maxon::Int targetSize = (((8 * initialSize) / blockSize) + 1) * aesBlockSize;
  const maxon::Int diff = targetSize - initialSize;

  // Fill the difference of the data to the block size stride with random bits/bytes.
  if (diff > 0)
  {
    encryptedContent.Resize(targetSize) iferr_return;

    maxon::UChar* randomDataStart = (maxon::UChar*)(encryptedContent.GetFirst()) + initialSize;
    const maxon::Int randomDataSize = diff;
    const auto randomDataBlock = maxon::ToBlock<maxon::UChar>(randomDataStart, randomDataSize);
    const maxon::SecureRandomProvider provider = maxon::SecureRandom::GetDefaultProvider();
    maxon::SecureRandom::GetRandomNumber(provider, randomDataBlock);
  }

  // Check if the encoder does support in place conversions.
  if (!aesEncoder.SupportInplaceConversion())
    return maxon::UnexpectedError(MAXON_SOURCE_LOCATION);
  // Encrypt the data in place.
  aesEncoder.ConvertAllInplace(encryptedContent) iferr_return;
  
  return encryptedContent;
}

/// -----------------------------------------------------------------------------------------------
/// Decrypts and decompresses a message that has been encoded with GetEncrypted().
/// 
/// @param[in] content The input text to decompress and decrypt.
/// @param[in] secret The encryption secret.
/// @return The decompressed and decrypted content as a string.
/// -----------------------------------------------------------------------------------------------
static maxon::Result<maxon::String>GetDecrypted(
   const maxon::BaseArray<maxon::Char>& encryptedContent, const maxon::String secret)
{
  iferr_scope;

  // The compression settings, the compression rate can lie between 0 and 9, where 9 is the
  // strongest (and slowest) compression.
  maxon::DataDictionary settings;
  settings.Set(maxon::STREAMCONVERSION::ZIP::ENCODER::COMPRESSION, maxon::Int(9)) iferr_return;
  settings.Set(maxon::STREAMCONVERSION::ZIP::ENCODER::WRITESIZE, true) iferr_return;

  // Build a CryptoKey for an AES encryption.
  const maxon::Id encoderID = maxon::StreamConversions::AesEncoder.GetId();
  const maxon::Int blockSize = 128;
  const maxon::Int keySize = 128;
  maxon::BaseArray<maxon::Char> key = secret.GetCString() iferr_return;
  const maxon::CryptoKey cryptoKey(encoderID, blockSize, key.GetFirst(), keySize);

  // Setup an AES decoder.
  maxon::DataDictionary cryptoSettings;
  cryptoSettings.Set(maxon::CryptographyOptions::CRYPTOKEY, cryptoKey) iferr_return;
  const maxon::StreamConversionRef aesDecoder = 
    maxon::StreamConversions::AesDecoder().Create(cryptoSettings) iferr_return;

  // Check if decoder supports in place conversions.
  if (!aesDecoder.SupportInplaceConversion())
    return maxon::UnexpectedError(MAXON_SOURCE_LOCATION);

  // Decrypt the message in place.
  aesDecoder.ConvertAllInplace(encryptedContent) iferr_return;

  // Decompress the data up to the null termination within the decrypted message now written into
  // encryptedContent.
  maxon::BaseArray<maxon::Char> decompressed;
  const maxon::StreamConversionRef zipDecoder = 
    maxon::StreamConversions::ZipDecoder().Create(settings) iferr_return;
  zipDecoder.ConvertAll(encryptedContent, decompressed) iferr_return;

  return maxon::String (decompressed);
}

/// -----------------------------------------------------------------------------------------------
/// Runs the example.
/// 
/// @param doc Not needed, requirement of private interface.
/// -----------------------------------------------------------------------------------------------
static maxon::Result<void> RunExample(BaseDocument* doc)
{
  iferr_scope;

  //The password and some mock data to encrypt.
  const maxon::String password = "wubba lubba dub-dub"_s;
  const maxon::Char* content = "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, "
    "sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam "
    "voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd  "
    "guber, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, "
    "consetetur elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna "
    "aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. "
    "Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem "
    "ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt "
    "ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo "
    "duo dolores et ea rebum.";
  const maxon::String inputContent (content);

  // Variables for the encrypted & decrypted content and secret.
  maxon::BaseArray<maxon::Char> encryptedContent;
  maxon::String decryptedContent, secret;

  // Hash the password.
  iferr(secret = GetAES128Secret(password))
    ApplicationOutput("Failed to hash password.");

  // Compress and encrypt the content.
  iferr (encryptedContent = GetEncrypted(inputContent, secret))
    ApplicationOutput("Failed to encrypt data.");

  // Decrypt and decompress the content.
  iferr(decryptedContent = GetDecrypted(encryptedContent, secret))
    ApplicationOutput("Failed to decrypt data.");

  ApplicationOutput("Password: @, Secret: @", password, secret);
  ApplicationOutput("Content: @", inputContent);
  ApplicationOutput("Encrypted content: @", encryptedContent);
  ApplicationOutput("Decrypted content: @", decryptedContent);

  return maxon::OK;
}