Archived Forum Post
Index of archived forum posts
Question:
Hi,
I'm having real trouble with the QuickAppend method of the Chilkat Zip module. Every time I use it, I seem to end up with corrupted zips.
Here's an example of unzipping a very simple zip that I've tried to append a zip to; to show what the error I'm seeing is:
**$ unzip ../zip1.zip**
Archive: ../zip.zip
warning [../zip1.zip]: 77 extra bytes at beginning or within zipfile
(attempting to process anyway)
replace test2.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
inflating: test2.txt
file #2: bad zipfile offset (local header sig): 77
I have tried this using both NodeJs and Python versions of the Chilkat zip extension. I'm not sure if there's something I'm missing!
Here's what I'm doing (the Python version of the code). I've checked that the initial zip1.zip and zip2.zip files are okay and can be uncompressed. I then run the code below. When I try and uncompress zip1.zip after running the code, I get the error above. Here's my code:
import sys
import chilkat
zip = chilkat.CkZip()
success = zip.UnlockComponent("licensecodehere")
if (success != True):
print(zip.lastErrorText())
sys.exit()
#The zip whose contents we are going to add to the new zip
success = zip.OpenZip("zip2.zip")
if (success != True):
print(zip.lastErrorText())
sys.exit()
#Add the contents of zip2 to zip1
success = zip.QuickAppend("zip1.zip");
if (success != True):
print(zip.lastErrorText())
sys.exit()
print("Zip Appended!")
Thanks in advance for any help!
Tom
You're likely using an old version of Chilkat. I recall this problem being fixed. Test with the latest version..
Before reporting a problem, always test with the latest version of Chilkat, or at least look at the release notes at cknotes.com
Hi
Thanks for the quick reply! For node, I'm using the following command to install chilkat
npm install chilkat_linux64
(which is from: https://www.chilkatsoft.com/nodejs.asp )
(Note - I did notice that it doesn't seem to support the latest version of node module version 47 - I had to rollback to version 4.4.2 of node, node module 46)
With Python, I installed version v9.5.0.56, for Python 2.7 64 bit, which I got from here:
https://www.chilkatsoft.com/python.asp
Are there later versions that I should be trying? I noticed that cknotes.com lists version v9.5.0.57, but I couldn't see this available for download on the main chilkatsoft.com site.
Thanks in advance!
Tom
I realise that v9.5.0.57 is just for Ruby; and so I'm pretty sure I'm already on the latest version for both Python and NodeJs.
I tried researching this on Google and couldn't find much about QuickAppend. I did find it mentioned at this article:
http://www.w2lessons.com/2012/01/fast-zipping-in-amazon-s3.html
... But it's not clear what version of Chilkat he was using, and that's from 2012...
Tom
Hi Chilkat,
I'm certain I'm using the latest version of the software. Is there anything else I can try? Is it possible that it's a bug? I'm pretty sure my code is inline with the examples.
Thanks,
Tom
I'll see if I can reproduce the problem..
I tested exactly the same as what you have, but did not encounter any problems. Check to see if both .zip archives are valid, prior to running your test. Maybe you're starting with invalid .zip archives to begin with..
PS> Chilkat will catch up to the latest version of Node.js ASAP..
Hi Chilkat,
Thanks for the reply. This is really strange. I used CodeAnywhere's Linux containers to do these tests. I tried containers with both Centos 6.5 and Ubuntu 14.04. These are standalone / out-of-the-box Linux servers.
Here's how I created the zips (on the command line).
mkdir zip1/ cd zip1/ touch testfile.txt zip -r zip1.zip .
I verified that this zip was okay by unzipping it on the command line, and all uncompressed without issue.
So I don't think these are corrupt. I also tried creating zips using another Chilkat example, which worked correctly to create the zips.
But when I do the QuickAppend, I find that the zip becomes corrupt. If I append zip2 into an existing zip1, then when I uncompress the newly appended file (zip1.zip), I seem to only get the zip2 files, and don't see the original zip1 files any longer --- and I get the "77 extra bytes at beginning or within zipfile" error.
So it's as if it's added the new content, but somehow overwritten the file list or done something incorrectly at that point?
Is there anything else I can do to test? I'm using CodeAnywhere because it's an out-of-the-box system, and am trying on two different Linux OS's. I guess I could try on separate Windows or Mac systems too if it would help.
I could also try creating the zips in a different way, if it could be an issue with the built-in Linux zip utility.
Tom
Here's a good image of the zip file format:
An app would typically read a .zip by first going to the very end of it and reading the end-of-central-directory record, to find the beginning of the central directory. From that point, the location of each local file header can be found. Note that the existence of gaps between files does not indicate corruption. In fact, if one wanted to delete a single file from a .zip without re-writing the entire .zip, the way to do it is to simply remove the record from the central directory. (I may be missing some other details, but in general, you end up with a .zip with a hole in it.)
So the existence of extra bytes may not be an error technically. I'll run a simple test to examine the resulting .zip created by QuickAppend to see if I can spot the extra bytes, and if found, try to understand why it exists or what it is..
Hi Chilkat,
Thanks for this. Yes; that would be great!
Meanwhile, I have tried creating zips natively on my Mac, and then running the QuickAppend script on these. When I try to extract the resulting zip on my Mac, I get a "fatal error" when extracting the zip; so I think it's something that doesn't just affect the command line zip utility.
It turns out that the Mac does uncompress the zip, even though it gives a fatal error. But the uncompress zip only has the content of the file that was merged into the zip, and doesn't show the original zip (eg. the same thing I saw on the command line).
Great, I'll have a look more at the structure of zips and see if there's anything else I can spot about the central directory, etc.
I'm going to try on a PC as well, just so that I've tested that too. And one of my colleagues is going to try the same thing on the C# version, just in case it's something that only affects Python / NodeJs.
Thanks for your help so far,
Tom
Further to last message --- just in case it's useful, here's the output of zipinfo when I examine the file that's merged by Chilkat. This lists all the files (in both zip1 and zip2), which makes it look like everything is fine:
Archive: /Users/tomkerswill/Downloads/zip1.zip 3144 bytes 14 files
warning [/Users/tomkerswill/Downloads/zip1.zip]: 534 extra bytes at beginning or within zipfile
(attempting to process anyway)
drwxr-xr-x 2.1 unx 0 bx stor 12-Apr-16 14:30 zip2/
-rw-r--r-- 2.1 unx 23 bx defN 12-Apr-16 14:30 zip2/test3.txt
drwxrwxr-x 2.1 unx 0 bx stor 12-Apr-16 14:30 __MACOSX/
drwxrwxr-x 2.1 unx 0 bx stor 12-Apr-16 14:30 __MACOSX/zip2/
-rw-r--r-- 2.1 unx 249 bx defN 12-Apr-16 14:30 __MACOSX/zip2/._test3.txt
-rw-r--r-- 2.1 unx 22 bx defN 12-Apr-16 14:30 zip2/test4.txt
-rw-r--r-- 2.1 unx 249 bx defN 12-Apr-16 14:30 __MACOSX/zip2/._test4.txt
drwxr-xr-x 2.1 unx 0 bx stor 12-Apr-16 14:29 zip1/
-rw-r--r-- 2.1 unx 21 bX defN 12-Apr-16 14:29 zip1/test1.txt
drwxrwxr-x 2.1 unx 0 bx stor 12-Apr-16 14:30 __MACOSX/
drwxrwxr-x 2.1 unx 0 bx stor 12-Apr-16 14:30 __MACOSX/zip1/
-rw-r--r-- 2.1 unx 248 bX defN 12-Apr-16 14:29 __MACOSX/zip1/._test1.txt
-rw-r--r-- 2.1 unx 23 bX defN 12-Apr-16 14:30 zip1/test2.txt
-rw-r--r-- 2.1 unx 249 bX defN 12-Apr-16 14:30 __MACOSX/zip1/._test2.txt
14 files, 1084 bytes uncompressed, 644 bytes compressed: 40.6%
However, when I actually uncompress the zip, only the top half of these files (the zip2 ones) are extracted. Here's the message I get:
Archive: /Users/tomkerswill/Downloads/zip1.zip
warning [/Users/tomkerswill/Downloads/zip1.zip]: 534 extra bytes at beginning or within zipfile
(attempting to process anyway)
creating: zip2/
inflating: zip2/test3.txt
creating: __MACOSX/
creating: __MACOSX/zip2/
inflating: __MACOSX/zip2/._test3.txt
inflating: zip2/test4.txt
inflating: __MACOSX/zip2/._test4.txt
file #8: bad zipfile offset (local header sig): 534
file #9: bad zipfile offset (local header sig): 585
file #10: bad zipfile offset (local header sig): 682
file #11: bad zipfile offset (local header sig): 737
file #12: bad zipfile offset (local header sig): 797
file #13: bad zipfile offset (local header sig): 1022
file #14: bad zipfile offset (local header sig): 1121
I also noticed that the filesize of the newly merged zip, is more than twice these size of the two original zips; which seems weird!
zip1.zip --- 3144 bytes #the combined file, made by appending zip2 to zip1 (corrupted)
zip1.zip --- 1347 bytes #the original zip1
zip2.zip --- 1349 bytes #the original zip2
I hope this is helpful; thanks for the info so far.