Building a simplified CLI with Invoke (for Monorepo management)

April 18, 2023 from Phillip Johnston

Our Monorepo work has included this to create several Helper scripts for repeated complex operations. This is essential, but let me tell you that it is annoying to always be in the Monorepo root and call it up on the full path.

I am a big fan to create simplified Cli interfaces to make work easier. My usual method is to create a make -film “Shim” at the highest level that usesPHONY Goals. That works well, but it has disadvantages:

• Options are annoying to do because they have to be treated by make -variables
• You must (or refer) to the top level of the directory to carry out the commands
• They are limited to the functions that you can implement

I wanted to experiment with the Python Invoke module For a while, and that seems to be a good test run. Calling off the above problems:

• Options and switches are to be defined and used trivial
• I can carry out the commands from anywhere within the work tree
• The full skills of python Are available

This post only goes through how I simplify our Monorepo tool interface with Invoke. For an overview of the Invoke module, Check out the guide for the first steps (or This interrupt contribution).

The basic idea

With this movement I would like to have a single command (e.g., e.g. inv mono) This can manage the Monorepo interactions.

Instead of being on the top level of the repository and performing a script as follows:

./tools/monorepo-management/build_module_dev_setup.sh

I want to be everywhere in the repository and carry out an command as follows:

inv mono --dev

For the first support, I focused on options that support these three common activities:

1. Use Symlinks for the Build infrastructure
2. Restore the Build infrastructure in GIT sub -modules
3. Set up the Monorepo after checking out

As soon as these skills are described, I will cover them Complete task definition and command line options.

Use Symlinks for the Build infrastructure

An ability I need is the configuration of the tested repository for local development. With our existing scripts, this can be as simple as:

def monorepo_dev_setup(ctx):
    # Have Invoke run the script
    ctx.run(f"{ROOT_DIR}/{MONOREPO_MANAGEMENT_DIR}/build_module_dev_setup.sh")

This is okay, but the script essentially uses two find Commands that can be Slightly ported to “pure python”. Here is the code that is needed to exchange them meson Submodules with symlinks to the Monorepo module at the top level:

def monorepo_dev_setup(ctx):
    '''
    Configures the monorepo for local development.

    Right now, this causes build submodules to be "ignored", allowing
    us to swap out the submodule pointers with symlinks to the top-level folder
    '''
    meson_list = find_all_submodule_instances('meson')
    disable_git_tracking(ctx, meson_list)
    replace_submodules_with_links('meson', meson_list)

def find_all_submodule_instances(submodule_name):
    '''
    This function looks for all of the various build submodule instances
    that are scattered throughout the source tree.

    Expected inputs are: 'meson', 'cmake'

    This function returns a list of POSIX paths to the submodules.
    '''
    found_list = list(ROOT_PATH.glob(f"src/**/{submodule_name}/"))
    found_list += list(ROOT_PATH.glob(f"stdlib/**/{submodule_name}/"))
    found_list += list(ROOT_PATH.glob(f"templates/**/{submodule_name}/"))
    found_list = (dir.as_posix() for dir in found_list if not 'buildresults' in dir.as_posix())
    return found_list

def disable_git_tracking(ctx, file_list):
    '''
    Tell git to ignore changes to the specified file list
    '''
    ctx.run(f"git update-index --assume-unchanged {' '.join(file_list)}")

def replace_submodules_with_links(submodule_name, submodule_list):
    '''
    Given a specific submodule target, replace the submodules in the list
    with symlinks to the top-level folder of the same name.

    Expected submodule_name inputs are: 'meson', 'cmake'
    The list should correspond to the target submodule
    '''
    for dir in submodule_list:
        rmtree(dir) #, ignore_errors = True)
        os.symlink(os.path.join(ROOT_DIR, submodule_name), dir)

Restore the Build infrastructure in GIT sub -modules

The inverse operation, which turns the Build module back into suitable sub -module pointers with activated tracking, can be a simple script call again:

def monorepo_production_setup(ctx):
    print("Restoring monorepo to production state")
    ctx.run(f"./{MONOREPO_MANAGEMENT_DIR}/build_module_restore.sh")

However, this process is also easy to port on Python:

def monorepo_production_setup(ctx):
    '''
    Restores the monorepo to a production state.
    - Build submodules will be tracked once again
    - Symlinks will be removed
    - Submodule pointers will be restored
    '''
    # This function was shown in the previous example
    meson_list = find_all_submodule_instances('meson')
    enable_git_tracking(ctx, meson_list)
    restore_submodules(ctx, meson_list)

def enable_git_tracking(ctx, file_list):
    '''
    Tell git to stop ignoring changes to the specified file list
    '''
    ctx.run(f"git update-index --no-assume-unchanged {' '.join(file_list)}")

def restore_submodules(ctx, file_list):
    for dir in file_list:
        os.unlink(dir)

    ctx.run(f"git checkout {' '.join(file_list)}")

Set up the Monorepo after checking out

Our special Monorepo setup will work well with a fresh clone, but it will do it Introduce undesirable inefficiencies (E.g. meson cloning repositories that are actually included in the tree and you have to check out 50+ submodule). We have several scripts that can be carried out according to a fresh clone that sets an optimal environment.

However, this is an additional complication Which setup steps do you want to do depends on the role.

• The Ci Order test builds must configure symlinks and submodules, but do not configure remote controls.
• The CI job that exceeds the updates of the various distributed repositories only Must populate remote controls, but do not do any work that have to create sub -modules or symlinks.
• I have to do the entire Setup steps for my local work

To provide this flexibilityWe will treat the requested setup mode as a string option and carry out certain actions in accordance with the request.

def monorepo_setup(ctx, setup_type="Default"):
    '''
    This function sets up a freshly cloned repository in monorepo mode.

    The default setup option will do full setup, e.g., for local development
    by someone who will make releases.

    Suboptions include:
    --setup=ci: This option is used by the CI server to do a minimal setup
        for CI operations. This will initialize required submodules and
        create symlinks.
    --setup=remotes: this will only populate the remotes, which is useful for a
        CI job that will distribute changes to the dependent repositories (and
        do nothing else)
    --setup=links will create symlinks for subprojects throughout the source
        tree. This is useful when building the monorepo locally, but not
        distributing changes.
    --setup=submodules will populate the required submodules, ignoring those that
        should be replaced by symlinks for local builds within the monorepo.

    '''
    print("Setting up monorepo with mode: ", setup_type)

    if setup_type not in ("Default", "ci", "remotes", "links", "submodules"):
        print(f"Error: {setup_type} is not a valid option.\n"
            "Valid options are: --setup with no argument, "
            "--setup=ci, --setup=remotes, --setup=links, --setup=submodules")
        return

    # First step: populate remotes
    if setup_type in ("Default", "remotes"):
        ctx.run(f"{ROOT_DIR}/{MONOREPO_MANAGEMENT_DIR}/populate_remotes.sh")

    # Second step: initialize required submodules
    if setup_type in ("Default", "ci", "submodules"):
        ctx.run(f"{ROOT_DIR}/{MONOREPO_MANAGEMENT_DIR}/submodule_init_required.sh")

    # Third step: create symlinks for subprojects
    if setup_type in ("Default", "ci", "links"):
        ctx.run(f"{ROOT_DIR}/{MONOREPO_MANAGEMENT_DIR}/create_subproject_symlinks.sh")

    # Fourth step: if CI mode is used, we'll just preemptively set up
    # the symlinks for the build modules, rather than requirng an extra
    # CI command
    if setup_type in ("Default", "ci"):
        monorepo_dev_setup(ctx)

The task definition

Above you will find the different technical parts based on the command. The structuring of the recording on the task itself is much easier because we are primarily call The functions defined above.

Essentially, I would like an initial Cli interface that supports:

• inv mono --setup For a full setup, or inv mono --setup=<Type> For a specific setup configuration
• inv mono --dev To transform submodules into symlinks
• inv mono --prod To restore sub -modules

When defining an Invoke task, we can create command line arguments by specifying functional parameters. Invoke creates parameters with the same name as the variable (i.e. dev → --dev). Corresponding short command line options are also created (--dev → -d). We can define default settings for these variables and the flags can overwrite.

For basic applications such as the --dev And --prod Boolesche flags, set a default settings False Value and the flag True Works well enough. But what about --setupWhat should be taken for yourself or used with an option? Invoke provides one for this optional Specificors in the task Metadata. Optional arguments follow the following rules:

• If the flag is not specified, the standard value for the function parameter is used
• If the flag is given without value, it is treated as a bool and Seta on True
• If the flag is specified with a value, the value is delivered (ie as a string)

This is what the basic command structure looks like:

@task(optional=('setup'))
def mono(ctx, setup=None, dev=False, prod=False):
    # Optional arugment
    # If setup is truth-y, we'll check to see if it's a string or not.
    # If it's a string, we'll pass the value to the monorepo_setup function
    # If it's not, it's a bool, and we'll use the default setting.
    if(setup):
        if isinstance(setup, str):
            monorepo_setup(ctx, setup)
        else:
            monorepo_setup(ctx)

    # A pure bool: we'll only call the function if the flag has been specified
    if dev:
        monorepo_dev_setup(ctx)

    # A pure bool: we'll only call the function if the flag has been specified
    if prod:
        monorepo_production_setup(ctx)

One aspect when creating command line tools is to create auxiliary text. Call up automatically helps you help from auxiliary text by having the function of the function of the function. Options can be documented via the help Metadata field.

@task(optional=('setup'),
    help={
    "setup": "Run monorepo setup steps (e.g., after a fresh clone). "
    "--setup with no args will run the full setup process. "
    "--setup=ci will run a modified setup process for the CI server. "
    "--setup=remotes will only configure the remotes, and no other steps. "
    "--setup=links will only configure symlinks for subprojects. ",
    "dev": "Configure monorepo for local development (e.g., convert build submodules to symlinks)",
    "prod": "Restore monorepo to production state (e.g., convert from symlinks back to proper submodule pointers)"
})
def mono(ctx, setup=None, dev=False, prod=False):
    '''
    Monorepo management commands.
    '''

Here is the shell edition:

inv mono -h
Usage: inv(oke) (--core-opts) mono (--options) (other tasks here ...)

Docstring:
  Monorepo management commands.

Options:
  -d, --dev                        Configure monorepo for local development
                                   (e.g., convert build submodules to symlinks)
  -p, --prod                       Restore monorepo to production state (e.g.,
                                   convert from symlinks back to proper
                                   submodule pointers)
  -s (STRING), --setup(=STRING)    Run monorepo setup steps (e.g., after a
                                   fresh clone). --setup with no args will run
                                   the full setup process. --setup=ci will run
                                   a modified setup process for the CI server.
                                   --setup=remotes will only configure the
                                   remotes, and no other steps. --setup=links
                                   will only configure symlinks for
                                   subprojects.

Future extensions

At the moment these are only a fundamental improvement in the quality of life. We will expand from here:

• Add support for copying wrap files from the upper level to sub -projects
• Add support for exporting changes to the distributed repository
• Use Invoke -Namespaces to have a more sophisticated command architecture (e.g. mono.setupPresent mono.export)))

References

Related