Dealing with double files in a monorepo

April 7, 2023 from Phillip Johnston • Last updated on March 13, 2024

I interrupted mine Monorepo migration effort to complete a client project. Now I’m back in action. At this point I merged all projects (Get with signed commits).

In the next work phase, the build setup will be adjusted Support:

1. A monorepo building at the highest level that integrates and builds all sub-projects
2. Building individual sub -projects alone (both within the Monorepo tree and as an independent, distributed projects)

To overcome a significant hurdle is the problem of File Duplicating. In this Monorepo, double files take on two forms.

First, we have the concept of Meson sub -projects. Each sub -project within the Monorepo has a sentence of .wrap Files that describe the dependencies you need. If you create a project, Meson downloads that in the defined dependencies .wrap Files and place them in the subprojects/ Tree. This makes sense for distributed projects. But when I build in the “Monorepo mode”, I don’t want to download an external. I would like to use the local copy of the dependencies that I have in my Monorepo source tree! And in the case of “really external” dependencies like that ETLI just want to download one Copy of this dependency in my Monorepo, no separate copy for each sub -project that requires it.

The other source of duplication is our common building infrastructure. We have two sub -projects, meson-buildsystem And cmake-buildsystemwhich are included in the various distributed repository as a sub -module. After merging, there are now 51 registered Build -Submodule in the Monorepo. If these sub -module are populated, significantly slows down Git Command line processes. We would like to use when building in Monorepo one Common copy of the Build infrastructure, not 51 different copies.

Symlinks to rescue

If you want to refer to a single source in several places in your file system, there is a simple solution: Symlinks.

Here is the basic strategy:

• External dependencies are cloned into the top level of the Monorepo subprojects/ Directory with which the project can be installed in Monorepo mode
• External dependencies are symbolized in the subprojects/ Folders for components within the Monorepo you need support the individual sub -project building
• Internal dependencies (dependencies of projects defined in our Monorepo)

The way to manage all of this is to create a script in order to fill the required symlinks in the right places. We can do this automatically by using it find Command and looking for everywhere A .wrap File is specified.

# Here's a sampling of finding wrap files for external projects and linking to the top-level subprojects copy
find src -name "gsl-lite.wrap" -exec sh -c 'ln -sf `pwd`/subprojects/gsl-lite $(dirname {})/gsl-lite' \;
find src -name "compiler-rt.wrap" -exec sh -c 'ln -sf `pwd`/stdlib/compiler-rt $(dirname {})/compiler-rt' \;
find src -name "catch2.wrap" -exec sh -c 'ln -sf `pwd`/subprojects/Catch2 $(dirname {})/catch2' \;

Another challenge is to treat transitive dependencies. For example, many of the independent repositories have dependence on the embvm-core Project. The embvm-core The project has its own dependencies, which are “promoted” when building the Meson project. When creating links, we also have to include these transitive dependencies.

# Embvm-core dependencies have transitive deps to satisfy, we'll just link them in
find src -name "embvm-core.wrap" -exec sh -c 'ln -sf `pwd`/src/core $(dirname {})/embvm-core;
    ln -sf `pwd`/stdlib/compiler-rt $(dirname {})/compiler-rt;
    ln -sf `pwd`/subprojects/gsl-lite $(dirname {})/gsl-lite;
    ln -sf `pwd`/stdlib/libmemory $(dirname {})/libmemory;
    ln -sf `pwd`/src/libraries/c-linked-list $(dirname {})/c-linked-list;' \;

When you look at the Monorepo for the first time, you can run this script to set up the Symlinks. As soon as you move the external sub -projects into the upper level subprojects List that you will create all the goals in the Monorepo without relying on further clone operations.

De-duplication of sub-modules is more difficult

In general, Symlinks are an ideal solution for our common building infrastructure. However, there are minor differences that give complexity here. The Build Submodules in particular are persecuted files. If we replace the sub -moduler with a symink, git will see not changed changes. We could commit these changes unintentionally git commit -a. Even if we notice it, this still requires the revision of our commits before we can bring our changes together. In contrast, we configured .gitignore so that the dependencies in the downloaded dependencies were downloaded subprojects/ Folder not persecuted. git Ignorates the symptoms in subprojects/ Also directories.

Here were three options that I saw for dealing with them:

1. Eliminate the submodule and only have several copies in the source tree. Updates are distributed by copying the monorepo folder content of the top level into the individual sub-projects.
   • That is fineBut it leads to our repo pursuing many other files that are otherwise not needed.
   • This is also prone to errors because it is technically possible for us that a subproject building modules deviate from the others. These changes can be blown away if an update script spreads changes from the tip of the Monorepo to the various sub -projects.
2. Find out how you get git To temporarily ignore the fact that we switch from a submodule to a Symlink.
3. Make the Build modules usable by Meson’s subproject infrastructure.
   • This has some ideal properties if possible:
     • Eliminated sub -module
     • Avoid duplicated files
     • Eliminates the need to update submodules commits
     • Enables the simple Symlink script to the entire work
   • There are some reusable modules that may not work in this setup, e.g. B. one that defines the compiler and left flags for every destination.
   • This requires rethinking the structure of each build module. It’s just too much to attack at that moment.

Option 2 seems to do best for the time being. There are Some workflows with use git --work-tree for thatBut we prefer the much easier approach to use git update-index To ignore changes to the sub -modules if we want them to act as symlinks.

Create submodule -setup steps

The basic idea is to tell git To ignore changes to the various build submodules. We do this by issuing git update-index --assume-unchanged path/to/submodule. This allows us to replace the sub -module in the top building directories without having git Think that the files have changed. Then we can create all sub -projects locally without checking the build module 51 different times.

We can create a script that finds the Build -Submodule in the subproject directories, it says in git In order to ignore changes, the submodule file deletes and creates a Symlink for the upper level’s build infrastructure.

# First, we need to disable local tracking for build submodules
find src stdlib templates -name "meson" -not -path "*/buildresults/*" \
    -exec sh -c 'git update-index --assume-unchanged {}' \;

# Then we need to remove the build submodule folders and create symlinks
# to the top-level folder
find src stdlib templates -type d -name "meson" -not -path "*/buildresults/*" -prune \
    -exec sh -c 'rm -rf {}; ln -sf `pwd`/meson {}' \;

Create sub -module recovery step

To lead the sub -module back into its “form of production” git update-index --no-assume-unchanged path/to/submodule and replace the folders with the submodule file.

# First we need to enable local tracking
find src stdlib templates -name "meson" -not -path "*/buildresults/*" -exec git update-index --no-assume-unchanged {} \;

# Then we need to restore the submodule files
find src stdlib templates -name "meson" -not -path "*/buildresults/*" -exec sh -c 'rm -rf {}; git checkout {}' \;

Update Build -Submodule Commits

If you work on site, you would like to use the Build -Submodule exclusively in your symlink form. The restoration of submodules is primarily useful if you have updated the build module and want to spread an update to the various distributed sub -projects. In the long run ours Ci The server will probably treat this process for us.

Here is the required flow for updating the build submodules in the distributed projects. So far, this is the most complicated part of the distribution process.

1. Publish the latest changes to the Top Level Build sub-module in its independent repository.
2. Get the Commit Hash from the independent repo
   • This could be automated in the future by interconnecting the commit hash during the update step, but at the moment I check it manually Github
3. Create the step for the creation of sub -module recovery
   • We only want the sub -module pointer, not the chicked content, so we never initiate the submodule.
4. Update the Submodule Commit Hash for every Build module instance
5. Make the changes
6. Provide the subprojects that now refer the latest commit in the Build -Submodul

Updating the commit -hashs also happens git update-index. This time we use the --cacheinfo Option to change the commit.

git update-index --cacheinfo 160000,20e59d92a7499c55b13e5f7299ad807f2d5ffc35,src/core/meson

Translation:

• 160000 is the file mode, here it shows a directory
• 20e59D92A749C55B13E5F7299AD807F2D5FC35 is the commit hash in the independent repository that we would like to refer to
• SRC/Core/Meson is the way to the sub -module that we want to update
  • Note that there is no subsequent slash (this would result in an error)

The above command updates the submodule pointer, but does not commit the change – we still have to do this explicitly.

Ultimately, we would like to create another script that takes over the distributed update process. We can use it to indicate which creation infrastructure we want to update (meson/cmake) and use the commit.

./tools/monorepo-management/build_module_update_commit.sh meson 20e59d92a7499c55b13e5f7299ad807f2d5ffc35

Inside this follows the same find Bang patterns than the other scripts:

find src stdlib templates -name"meson" -not -path "*/buildresults/*" \
    -exec git update-index --cacheinfo 160000,$1,'{}' \;

Now all sub -projects will refer to the new commit -hash, and the changes are ready to be committed (finally as part of this script). Then we are ready to provide the changes to the distributed repository.

git diff --cached
diff --git a/src/app/blinky/meson b/src/app/blinky/meson
index 29bf2439e..20e59d92a 160000
--- a/src/app/blinky/meson
+++ b/src/app/blinky/meson
@@ -1 +1 @@
-Subproject commit 29bf2439e21590678b3e3c8c0335335888bafe40
+Subproject commit 20e59d92a7499c55b13e5f7299ad807f2d5ffc35
diff --git a/src/app/embvm-demo/meson b/src/app/embvm-demo/meson
index 29bf2439e..20e59d92a 160000
--- a/src/app/embvm-demo/meson
+++ b/src/app/embvm-demo/meson
@@ -1 +1 @@
-Subproject commit 29bf2439e21590678b3e3c8c0335335888bafe40
+Subproject commit 20e59d92a7499c55b13e5f7299ad807f2d5ffc35
diff --git a/src/app/environment-monitor/meson b/src/app/environment-monitor/meson
index 29bf2439e..20e59d92a 160000
--- a/src/app/environment-monitor/meson
+++ b/src/app/environment-monitor/meson
@@ -1 +1 @@
-Subproject commit 29bf2439e21590678b3e3c8c0335335888bafe40
+Subproject commit 20e59d92a7499c55b13e5f7299ad807f2d5ffc35
diff --git a/src/drivers/aardvark/meson b/src/drivers/aardvark/meson
index 29bf2439e..20e59d92a 160000

(etc...)

Read more

Related