Paige Niedringhaus

devops

First published August 13, 2023

Copy Files from One Repo to Another Automatically with GitHub Actions

Even add a newly copied file to a feature branch for easier pull request reviews.

Assembly line making identical pink shoes

Introduction

Welcome to the third (and final) installment in my series about the many things I learned while building my first open source API library for Blues, the Internet of Things startup I work for.

It was a good challenge and a great learning opportunity for me.

I automated as many pieces of the library's maintenance and deployment to as I could with the help of GitHub Actions workflows, one of them being: copy the openapi.yaml file from the Blues cloud repo, Notehub, whenever that file was updated, and push it into a feature branch in the Notehub JS repo (the name of the API library).

Interacting with two different repos in one workflow might sound tricky, and at first it was, but it is possible and not too bad once you see how it can be done.

In this tutorial, I'll demonstrate how to create a GitHub Actions workflow that automatically copies a file from one repo to another and pushes the changes to another repo inside a feature branch.

I know the use case sounds a bit specific, but it came in really handy for me, and you never know when it might for you too.

Notehub JS

Before we get to the actual GitHub Actions workflows, I'll give you a little background on the Notehub JS project.

This section outlines the folder structure for the repo in case you want to explore it in GitHub, if you just want to get to the solutions, feel free to jump down to the next section.

Notehub JS is a JavaScript-based library for interacting with the native Notehub API, and it is actually generated from the Notehub project's own openapi.yaml file, which follows the OpenAPI specification standards.

The OpenAPI Generator CLI is a tool that uses the openapi.yaml file to create an entire API library complete with documentation, models, endpoints, and scripts to package it up for publishing as an npm module. The end library that is published to npm is a subfolder inside of the main Notehub JS repo. At the root of the project are the openapi.yaml file, the GitHub Actions workflows, and a few other config files.

Here's a simplified view of the Notehub JS repo's folder structure:

root/
├── .github/ 
| ├── workflows/
| | ├── create-pr.yml 
| ├── PULL_REQUEST_TEMPLATE.md
├── src/ <- this is the folder generated by the OpenAPI Generator CLI
| ├── dist/
| ├── docs/
| ├── src/ 
| | ├── api/ 
| | ├── model/
| | ├── index.js 
| openapi.yaml
| config.json
| package.json

Based on the diagram above, the openapi.yaml file that the library is generated from lives at the root of the repo, and the src/ folder is what holds all the Notehub API JavaScript code that powers the Notehub JS library.

This openapi.yaml file gets copied from the Notehub repo when changes are made to it. And changes are made to that file whenever the Notehub API is updated with new features and functionality, so making sure that the Notehub JS library keeps up with the new additions to the original API it's based on is important.

Now that I've explained more about the Notehub JS repo and why keeping it in sync with the Notehub API is important (and depends on the openapi.yaml file), we can get down to the business of ensuring the file gets copied to this repo whenever it's updated.

Looking for more details about Notehub JS?

Check out my first blog post about how to use GitHub Actions to automatically publish new releases to npm - I do a fairly deep dive on Notehub JS there.

Create a GitHub Actions workflow to run when the openapi.yaml file is updated

First things first: creating a new GitHub Actions workflow that's triggered when the openapi.yaml file changes in the Notehub repo.

Need a refresher on GitHub Actions?

If you want a quick primer on what GitHub Actions are, I recommend you check out a previous article I wrote about them here.

Below is the final version of the copy-openapi-file.yml file, which lives inside of the ./github/workflows/ folder. I'll walk through each step of the script below.

.github/workflows/copy-openapi-file.yml

name: Copy updated OpenAPI file

# only run this workflow when the openapi file has changed on the master branch
on: 
  push:
    branches: 
      - master
    paths:
      - 'notehub/api/openapi.yaml'

jobs:
  copy_openapi_file_to_notehub_js: 
    runs-on: ubuntu-latest
    steps:
    # check out notehub project
    - name: Check out Notehub project
      uses: actions/checkout@v3
    # check out notehub-js project using token generated in that project to successfully access it from inside the Notehub GitHub Action workflow
    - name: Check out Notehub JS project
      uses: actions/checkout@v3
      with:
        repository: blues/notehub-js
        path: ./notehub-js
        token: ${{ secrets.NOTEHUB_JS_TOKEN }}
    # make a copy the openapi file from notehub project
    - name: Copy OpenAPI file
      run: bash ./.github/scripts/copy-openapi-file.sh
      env:
        DESTINATION_PATH: ./notehub-js/
    # make a branch in notehub-js repo and push the copy of the openapi file there
    - name: Push to notehub-js repo
      run: bash ./.github/scripts/push-openapi-to-notehub-js.sh
      env:
        BRANCH: feat-openapi-update

Each GitHub Actions workflow needs a name, and I try to choose descriptive ones like this one: Copy updated OpenAPI file.

Then, I want this workflow to only run when the openapi.yaml file in the master branch of the project is changed. master is the branch that gets deployed to production, and in my case it's a safe assumption that when there's a change to this file in master, those changes are going to prod, which is when I want to copy the file over to the Notehub JS repo so it can reflect those same changes.

The code snippet below does just that.

on: 
  push:
    branches: 
      - master
    paths:
      - 'notehub/api/openapi.yaml'
  • on is how a workflow is triggered.
  • push is the event that triggers the workflow.
  • branches defines which branch names this workflow should run for.
  • paths is another filter to even more finely determine when the workflow runs.

What it boils down to is: when there's a change to the master branch in the repo for the openapi.yaml file at this particular file path, run the GitHub Actions workflow.

From there, the copy_openapi_file_to_notehub_js job runs. There's only one job in the jobs section of this script, but if there were multiple jobs, they'll run sequentially.

The create_pr_repo_sync job defines that it runs on the latest version of Ubuntu in runs-on.

And finally, I get to the steps section. Here's how it goes:

  1. Check out the Notehub code so the workflow can access it using the GitHub Action actions/checkout@v3.
  2. Check out the Notehub JS code so the workflow can access it as well using the GitHub Action actions/checkout@v3.
    • NOTE: Notice that in this second checkout action I had to include a with parameter that allows me to specify another repo to check out by providing a repository, path, and token. The token is a GitHub Personal Access Token (PAT) required when checking out other libraries beyond the current one the workflow is operating within.
  3. Run a shell script named copy-openapi-file.sh with the GitHub Actions environment variable DESTINATION_PATH passed to the script.
  4. Run another shell script named push-openapi-to-notehub-js.sh with the GitHub Actions environment variable BRANCH passed to the script.

In the following sections I'll go through what those two shell scripts do in detail.

Ok, so the workflow file is done. Time to get into the details of the two scripts doing the heavy lifting.

Write one shell script to copy the openapi.yaml file to the Notehub JS repo

Premade GitHub Actions can do many things, but something as specific as copying a file from one repo to another is a bit beyond them.

Luckily, GitHub Action workflows can run shell scripts, so if you can use a scripting language like PowerShell or Bash, you're in luck.

Here is the Bash script I wrote to copy the file from the Notehub repo to the Notehub JS repo.

copy_openapi_file.sh

#!/usr/bin/env bash
# bash boilerplate
set -euo pipefail # strict mode
readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
function l { # Log a message to the terminal.
    echo
    echo -e "[$SCRIPT_NAME] ${1:-}"
}

# File to copy from Notehub
OPENAPI_FILE=./notehub/api/openapi.yaml

# if the file exists in Notehub, copy it to Notehub-JS repo
if [ -f "$OPENAPI_FILE" ]; then
    echo "Copying $OPENAPI_FILE"
    cp -R ./notehub/api/openapi.yaml $DESTINATION_PATH
fi

echo "OpenAPI file copied to $DESTINATION_PATH"

The 9 lines at the top of the file are Bash script set up, so don't worry too much about what's happening there.

The first line to pay attention to is where the OPENAPI_FILE variable is defined with a path to the openapi.yaml file in the Notehub repo.

After identifying the file to copy from the Notehub repo, a standard Bash if statement checks if the file exists in the Notehub repo (just to be safe), and if it does, it copies the file recursively using cp -R to the $DESTINATION_PATH variable that was provided by the GitHub Actions environment variable.

In this case, the $DESTINATION_PATH is pointing to the root folder of the Notehub JS project where the file should be copied to, but you could specify it copy the file anywhere within the receiving repo that makes sense.

During the copying process and when the script finishes copying the file I added a few echo commands to print out useful messages in the logs to let me know things are going to plan.

And that's the first shell script that actually copies the file that will be taken from one repo to the other. I hope it's not too complicated once the actions in the script are broken down.

Now, it's time to add that file to the Notehub JS repo.

Use a second shell script to push the copied file to a feature branch in the other repo

When I checked the GitHub Marketplace for a ready-made action to create a new feature branch in the repo receiving the copied file, there were a few to choose from, but I wanted more control over what was happening than they offered.

I wanted an action that would:

  1. Create a new branch in the Notehub JS receiving repo if it didn't exist, add the copied file, and then push that branch to GitHub.
  2. Check out the already existing branch in the Notehub JS repo, overwrite any previous file changes in the branch, and then push the new file changes in that branch back to GitHub.

Here's the bash script that I came up with to handle those two scenarios.

push-openapi-to-notehub-js.sh

#!/usr/bin/env bash
# bash boilerplate
readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
function l { # Log a message to the terminal.
    echo
    echo -e "[$SCRIPT_NAME] ${1:-}"
}

# move to the root the notehub-js repo
cd "./notehub-js"
echo "Open root of Notehub JS repo"

# check if there's already a currently existing feature branch in notehub-js for this branch
# i.e. the altered openapi.yaml file's already been copied there at least once before
echo "Check if feature branch $BRANCH already exists in Notehub JS"
git ls-remote --exit-code --heads origin $BRANCH >/dev/null 2>&1
EXIT_CODE=$?
echo "EXIT CODE $EXIT_CODE"

if [[ $EXIT_CODE == "0" ]]; then
  echo "Git branch '$BRANCH' exists in the remote repository"
  # fetch branches from notehub-js
  git fetch
  # stash currently copied openapi.yaml
  git stash
  # check out existing branch from notehub-js
  git checkout $BRANCH 
  # overwrite any previous openapi.yaml changes with current ones
  git checkout stash -- .
else
  echo "Git branch '$BRANCH' does not exist in the remote repository"
  # create a new branch in notehub-js 
  git checkout -b $BRANCH
fi

git add -A .
git config user.name github-actions
git config user.email [email protected]
git commit -am "feat: Update OpenAPI file replicated from Notehub"
git push --set-upstream origin $BRANCH

echo "Updated OpenAPI file successfully pushed to notehub-js repo"

Once more, the first 9 lines of code are boilerplate for Bash; no comments necessary.

The first thing this script does is situate itself in the root of the Notehub JS folder just as you would when navigating around a computer via the command line: cd "./notehub-js".

Once there, it checks if the feature branch environment variable passed in by the GitHub Action $BRANCH already exists in the Notehub JS repo with this one-liner.

git ls-remote --exit-code --heads origin $BRANCH >/dev/null 2>&1

Essentially, this code is:

  • Getting a list of references in the remote repository with git ls-remote,
  • Checking if env var $BRANCH (passed from the GitHub Action) exists only in the refs/heads of the origin remote with --heads
    • (i.e. does the branch name already exist in the GitHub repo where it keeps track of the names of all of its branches),
  • And returning an --exit-code with status "2" when no matching refs are found in the remote repo and a status of "0" if a matching ref is found.

When the exit code is returned, it's set equal to the variable $EXIT_CODE.

Then, an if/else statement checks what $EXIT_CODE equals.

EXIT_CODE = "0"

If the exit code is 0, that means the branch already exists in the repository; this could happen if multiple changes are made to the openapi.yaml file before the feature branch in Notehub JS gets merged in and deleted.

When that happens the following steps take place:

  1. The branches for the Notehub JS repo are fetched via git fetch,
  2. The current changes (the freshly copied openapi.yaml file) are stashed with git stash,
  3. The existing branch of the same name as the $BRANCH env var is checked out locally,
  4. The stashed changes are applied to that branch overwriting whatever was there before with git checkout stash -- ..

EXIT_CODE = "2"

If $EXIT_CODE is equal to 2, that means the branch doesn't already exist remotely in GitHub and the script can simply checkout a new branch with the name in $BRANCH via git checkout -b $BRANCH.

Finally, after all this is completed, a standard set of git commands follows:

  1. Use git-add -A . to stage all changes in the working directory,
  2. Configure a username and email address for who's doing the commit with git config user.name and git config user.email,
  3. Add a commit message with git commit -am "Standard commit message here",
  4. Lastly, push it to GitHub with git push --set-upstream origin $BRANCH.

At this point, you should be good to go. The file should be successfully copied over to the other repository.

NOTE: Inspiration for this second Bash script was initially based on this blog post by Remarkable Mark.

Check that the workflow runs

The simplest way to test this new functionality is just to keep an eye out for changes in the main branch of the repo that would trigger the workflow to run and see if it works.

When you name the workflow a descriptive name, it's easy to find it amongst the various GitHub Action workflows the repo may have.

Here's a screenshot of the workflow runs of the Copy updated OpenAPI file job inside the GitHub repo's Actions page.

Multiple runs of a particular workflow in the GitHub Actions page of a repository

And if you want to dig in further to any of the runs to check the logs and job steps (especially handy for debugging purposes), just click on a workflow run, then the name of the job, and inside the job you can expand out each step.

Here's another screenshot of the Push to notehub-js repo step expanded to see exactly what the Bash script in that step is logging to the console via echo.

Details of a particular step inside of a GitHub Actions job expanded for review

And you're done! The workflow's running, the steps are successfully completing, and, most importantly, the file from the Notehub repo is successfully copied to the Notehub JS repo. Mission accomplished.

Conclusion

My first foray into building an open source software library on behalf of one of my company's APIs was a unique challenge that taught me a lot. Especially as I got to leverage GitHub Action workflows to automate as many parts of the process as I could to make ongoing maintenance and upkeep easier.

One such task that I automated involved copying a file from one repo into another whenever a change was made to that particular file. And while I was able to leverage a few pre-existing GitHub Actions, I also had to write a couple custom Bash scripts for some of the trickier, more-unique-to-my-use-case steps.

Fortunately, GitHub Action workflows allows for both types of steps to be combined in a workflow, making for some very powerful, turnkey solutions that did just what I needed.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

Thanks for reading. I hope learning to copy a file from one repo to another automatically with GitHub Actions workflows is as helpful for you as it has been for me. Enjoy!

References & Further Resources

Tags:
bash
devops
git
github

Want to be notified first when I publish new content? Subscribe to my newsletter.