Create deterministic ZIP files with GIT stamps

February 20, 2025 from Phillip Johnston

Some of us Courses reference .zip Archives saved on the website. The reference code for our courses is kept in one Git Repository. Whenever changes are made Ci/CD The infrastructure checks the code, packs it and uploads the files to the server.

The problem

We only want to upload archives to the server if the content has changed. We decided rsync for that, but that noticed that all Archives have been updated every time the upload step was carried out.

We have found that the file -Hashes changed every time we created an archive, even if the content of the file did not change.

md5 module_001_basics.zip
(out)MD5 (module_001_basics.zip) = 81f97561721eb9fdca96f31af79c2f90
make package # no source changes in between
md5 module_001_basics.zip
(out)MD5 (module_001_basics.zip) = e999800745e9929774e316d706be4ee4

Goals

We should be able to pack files in a deterministically in A .zip Archive so that the hash from A .zip The archive of unchanged files remains the same. In this way when we rsync The sentence of all archives to the web server, archives without changes are recognized and left alone.

Investigation

The first attempt used a naive zip -r to pack the files. After some research we learned that zip The files contain several additional file Attribute, e.g. B. temple, file owner, creation time, access time, authorizations, etc. Therefore, changes in one of these attributes can lead to different archive content without corresponding changes in the file content.

zip offers a --no-extra ((-X) MAD to remove additional file attributes. However, this is not reduced all Attributes: Permits and time stamps are still taken into account. So if file content remains unchanged, but the time stamp or the authorizations changes zip The process creates two different archives.

As soon as we added them -X Flag to our zip Commands, test sums of the archives were the same:

md5 module_001_basics.zip
(out)MD5 (module_001_basics.zip) = 1116ece88719fbe5fc07713959ddbcb3
make package # no source changes in between
md5 module_001_basics.zip
(out)MD5 (module_001_basics.zip) = 1116ece88719fbe5fc07713959ddbcb3

This worked well enough until we led the script on our build machine. After a few examinations, we learned that the build machine produced various MD5 -Hashes because the time stamps of the files were different. The files are freshly cloned for a new “nightly” build, which leads to another time temple for each file than in the previous build. Every night construction generated archives with different hashes and led to a complete upload of each file.

The solution

In order to create a deterministic ZIP file, we have to make sure that the time stamp of each file contained remains the same via machines and packaging runs.

The solution in which we (and others) have arrived git log. This script used below git ls-tree How to generate a list of persecuted files. We are using each file in a loop with use touch Update the file with your last commit temple in the git log.

cd ${MESON_SOURCE_ROOT}

rev=HEAD
if (( "$OSTYPE" == "darwin"* )); then
    git ls-tree -r -t --full-name --name-only "$rev" | while read filename ; do
        touch -t $(git log --pretty=format:%cd --date=format:%Y%m%d%H%m.%S -1 "$rev" -- "$filename") "$filename";
    done
else
    git ls-tree -r -t --full-name --name-only "$rev" | while read filename ; do
        touch -d $(git log --pretty=format:%cI -1 "$rev" -- "$filename") "$filename";
    done
fi

This can be a time -consuming process for large repository, so we have shown this to a separate goal in ours Build the system ((update_timestamps). For CI -Builds we still want to carry out a dry run of the packaging process so that we do not update_timestamps Operation. Nightly Builds, in which the “upload” operating process is carried out in the CI pipeline, updates the time stamp before packages are generated.

.PHONY: upload
upload: buildall
    $(Q)cd $(BUILDRESULTS); ninja update_timestamps
    $(Q)cd $(BUILDRESULTS); ninja package_course_001
    $(Q)cd $(BUILDRESULTS); ninja upload_course_001

This is a sufficient solution for our purposes. We transmit files using rsync --checksumIgnore the times and examined file sizes. If the file sizes are different, there is a transmission. If the sizes match, they are examined (md5), and those who have different sums are also transmitted.

If time stamps were included, you need an additional step: Setting the time stamp of the generated setting .zip Archive deterministic. We recommend the same general approach above. Use git Use touch Define the time stamp of the produced .zip Archive this value. For example:

ls -l libc-skeleton.zip
(out)-rw-r--r--   1 phillip  staff    126 Feb 20 08:56 libc-skeleton.zip
touch -t `git log --pretty=format:%cd --date=format:%Y%m%d%H%m.%S -1 HEAD -- libc-skeleton` libc-skeleton.zip
ls -l libc-skeleton.zip
(out)-rw-r--r--  1 phillip  staff  126 Dec  3 11:12 libc-skeleton.zip

References

Related