Integrate Different Git Repositories (Symink, Submodule)

@ CI-CD

Introduction

The website is still completely based on single Git repo; all code, fonts, images, videos, and article content… Large files are managed through Git LFS.

This decision has its pros and cons, one of which is the high coupling between content and code, making it necessary for anyone who wants to maintain it to download the hundreds of articles I wrote. 😅 As the project matures, I’ve decided to separate the two.

Method 1: Git Submodule

Git Submodule can achieve embedding a repository within another repository, effectively setting a pointer to another repository. The GitHub Action checkout🔗 also has a convenient submodules option. However, this is not my final chosen method.

Simply needing additional understanding of submodules is troublesome, and you often have to run git commands to sync content, so I don’t consider it. I expect a solution that is completely separate two repositories but can be assembled together at build time.

Method 2: Symbolic Link + Auto Pull Repository

Unix-like systems have Symbolic Link methods to simply point one folder to another, as follows:

Terminal window
ln -s Path_A Path_B


ln -s ~/Desktop/my-content/post ~/Desktop/web-dong-blog/src/content/post

I can pull both repositories and link the content repository’s content to the development repository using symbolic links. Any future changes in content will also instantly reflect in the development repository. Additionally, remember to ignore the link pointer files in .gitignore:

.gitignore
# Skip Symbolic link file
src/content/shortpost
src/content/toolbox

As for the online deployment part, the content repository can set up to send a content-updated event to the target Repo when pushing to the main branch to trigger the CI/CD of the development repository.

name: Update Content


on:
  push:
    branches:
      - main


jobs:
  trigger-blog:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger Blog CI
        env:
          BLOG_TRIGGER_TOKEN: ${{ secrets.BLOG_TRIGGER_TOKEN }}
        run: |
          curl -X POST -H "Accept: application/vnd.github+json" \
            -H "Authorization: Bearer $BLOG_TRIGGER_TOKEN" \
            https://api.github.com/repos/riceball-tw/web-dong-blog/dispatches \
            -d '{"event_type": "content-updated"}'

The development repository receives the previous content-updated event trigger and uses actions/checkout🔗 to pull the content repository and manually use rsync🔗 to move and merge files:

name: Deploy Pipeline


on:
  workflow_dispatch:
  repository_dispatch:
    types: [content-updated]


jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4


      - name: Checkout external content
        uses: actions/checkout@v4
        with:
          repository: riceball-tw/obsidian-sync
          ref: main
          path: src/content-temp/
          token: ${{ secrets.CHECKOUT_GITHUB_CONTENT_PAT }}
          lfs: true


      - name: Merge local and external content folder
        run: rsync -av --ignore-existing src/content-temp/ src/content/

Conclusion

The above provides different ways to solve the issue of needing to separate multiple repositories during development but for them to be assembled together at build time. The same concept can likely be applied to different platforms with a little differences.

Some may find it strange to use Git to store non-development resources. Wouldn’t it be better to set up a dedicated CMS?

Indeed, this is a structure arrived at through trade-offs. To reduce costs and complexity, I chose not to rely on third-party software for content management, and at least for the foreseeable future, this approach still works.

Further Reading