Archived Forum Post

Index of archived forum posts

Question:

Problems with QuickAppend --- extra entry after central directory found

Feb 07 '17 at 04:39

Hi

I have been having problems using the QuickAppend zip function to append the contents of a new zip file to an existing one.

I've tried this with the node, python and .Net (via a colleague) versions of Chilkat, on Mac, Linux and Windows (via colleague) machines. Whichever combination I try, I seem to always end up with a zip file which has extra bytes in it. The zip file cannot be extracted using many zip tools (eg. command line zip tools, and we've also tried native Mac and PC tools to extract)

I have followed the example code exactly and am using very simple example text files and zip files to test.

I have recorded a screencast to show exactly what I'm doing and what happens. The error seems to be that there is extra data in the zip that shouldn't be there (eg. "Entry after central directory found"):

https://drive.google.com/a/tomkerswill.co.uk/file/d/0B6S93W7cxOpHdWZmUWEtRERMRXc/view

Note - I put in a forum post in April ( http://www.chilkatforum.com/questions/10132/quickappend ), but rather than resurrect that dead thread, I thought it was better to create a new one.


Answer

Here's a view of the actual zip files in a hex editor, in case it's helpful.

The one on the left came from running Chilkat quickAppend as above. The middle one is the result of zipping the two files on the command line (same two files as in the Chilkat zip). The right one is the preExisting zip before it was appended to.

I noticed that the Chilkat QuickAppend seems to simply append the zip on top, keeping the central directory from the first zip (see first patch of brown on screenshot, with blue arrow).

I also noticed some more data that's repeated in the Chilkat zip (see second blue arrow -- brown then red text).

Here's the screenshot.

https://drive.google.com/a/tomkerswill.co.uk/file/d/0B6S93W7cxOpHNG1KU055T2IyU3c/view?usp=sharing


Answer

Thanks! I'll have a look later today..

-Matt


Answer

Perfect - thanks!


Answer

I've reproduced the problem. I'm looking into it...


Answer

I found the problem and made the fix. However, technically, the .zip is perfectly OK. It just has some unused space within it that contains data from the original zip (potentially some of the central directory and end-of-central dir record).

If you look here: http://tinyurl.com/jrvgvmy

You'll see that it really doesn't matter if there is extra space in between parts -- as long as all the offsets point to the correct places.

Here's why I couldn't reproduce the problem initially: The issue only occurs if the 1st additional LFH + file data is not larger than the original central directory + EOCD. When doing a QuickAppend, we start writing new LFH's at the point where the original central directory starts. The QuickAppend was, after each new LFH + file data was written, seeking to the end of the file before writing the next. But we could've still been writing over the old central directory + EOCD. Skipping forward to the actual end of the file leaves this hole -- which is completely harmless.

Let me put it another way -- examine the zip structure here again: http://tinyurl.com/jrvgvmy

It would be perfectly OK if a program decided to leave 2K of space between each LFH. It doesn't matter -- as long as the relative file pointers are all correct. It may not be smart to do that, but it's still a valid .zip. The complaint w/ the "zip -FFv ..." is that it found a header signature -- because when scanning to fix, it's looking for those sorts of things. But again, the .zip is still 100% valid.

Hopefully you understand. I did actually fix the issue so that the hole is no longer present. However, given that it's Node.Js, I'm not able to produce a build for other unrelated reasons. The "fix" will have to wait a week or two..

Regarding the zip tools that cannot extract -- the zip tools are actually broken in this case. A zip tool SHOULD be starting at the end of the .zip. The first thing it SHOULD do is to read the end-of-central-directory record, which gives the location of the central directory, and from there you get all the locations of the individual file headers. To start at the beginning of the .zip is foolish because it involves scanning and is more time consuming...


Answer

Thanks so much for the comprehensive reply!

Ah, okay - now I understand why I was seeing part of the old central directory there; it's just leftover.

Re: it not technically being a corrupt zip and the zip tools being the ones that are broken --- okay, I see from your explanation why it's technically a valid zip and that the tools should just be able to extract it.

The issue for us is that we'll have 200k non-technies downloading zips from us. If their zip tool gives them a warning or even only shows some of the original files and ignores some (like the linux command line zip utility does), then we can't tell the user to go and try another zip tool. They'll expect to be able to just double-click the zip like they normally do and have it extract...

Thanks for the help. Looking forward to testing it out...

Tom


Answer

Hi,

Further to last message --- I may have got the wrong end of the stick on this, but I'd thought from your info that the issue should only affect things if they are small files --- eg. The issue only occurs if the 1st additional LFH + file data is not larger than the original central directory + EOCD.

I've rerun this with two files that are about 7mb. I'm still seeing the same problems --- the error is still "30 extra bytes at beginning or within zipfile ". Here's the screencast:

https://drive.google.com/a/tomkerswill.co.uk/file/d/0B6S93W7cxOpHb1ZQekRLNm5TZW8/view

I have also showed what happens when I try to unzip. It seems to just give the file that was in the original zip, and not the one that's been quick-appended.

I'm just flagging this up really, in case there's another issue at play too (and not just one that affects small files).

Also - I'd love to do a proof of concept anyway, to my boss. Can you let me know what zip tool you were using initially? Eg. one that will work with these files? Would be great if I can run the whole process with a working zip program to show how it would work. All the ones I've tried so far seem to choke on these zips!


Answer

Hi,

Sorry for the multiple messages. I asked my colleagues to try a variety of zip programs, and all of them are giving errors, including 7Zip and the default Windows zip utility, plus the zip utilities I've tried for Mac and Linux. So I think this is quite a widespread issue! Most just refuse to unzip at all. Here are some screenshots:

7Zip - https://drive.google.com/file/d/0B6S93W7cxOpHbVNLWjhxdjJDR0k/view?usp=sharing

Windows Default Zip Program - https://drive.google.com/open?id=0B6S93W7cxOpHdTJDR3dNUmJGSW8

Thanks,

Tom


Answer

Hi Chilkat

I'm just following up to see if there'd been any luck with publishing the fix to this.

Thanks in advance!

Tom


Answer

Sorry for the long delay. Here's a new Mono build you can test:

http://www.chilkatsoft.com/download/preRelease/chilkatMono-9.5.0.zip