Using Hugo and Travis CI To Deploy Blog To Github Pages Automatically

My personal static blog is hosted on Github. I used Hexo before, but as its complicated package dependence and cumbersome deployment process, I have to encapsulated the whole deploying environment into Docker image for rapid deployment. But it was still too complex. Now I switch to Hugo which is faster, simpler.

This article documents how to automatically synchronize blog contents generated by Hugo to Github through Travis CI.

Attention: The following operation process will not involve account registration, Git configuration.

Prerequisite

Registering a Github account;
Registering a Travis CI account (login with Github account);
Installing git, hugo in your system;
Personal domain (optional, here my domain is axdlog.com);

Official Docs

The relevant operations in this document are based on the official document as the operation credentials:

Hugo Installation

Github url: https://github.com/gohugoio/hugo
Downloading page: https://github.com/gohugoio/hugo/releases
Release Info API: https://api.github.com/repos/gohugoio/hugo/releases/latest

Install Hugo is the official installation document of Hugo. If you’re using GNU/Linux, you may consider using my Python script to install it.

Current release version info

1
2
3
4
5


# which hugo
/usr/local/bin/hugo

# hugo version
Hugo Static Site Generator v0.43 linux/amd64 BuildDate: 2018-07-09T10:00:08Z

Python Script

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112


#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Writer: MaxdSre
# Date: Wed 2018-07-11 12:10:23 -0400 EDT

# import requests
from urllib.request import urlopen
#from bs4 import BeautifulSoup
import json
import re
import os
import tarfile
# remove unempty dirs
from shutil import rmtree
import subprocess

utility_name = 'Hugo'
hugo_release_api='https://api.github.com/repos/gohugoio/hugo/releases/latest'
symlink_path = '/usr/local/bin/hugo'
pack_save_dir = '/tmp'
target_dir = "/opt/" + utility_name
hugo_binary_path = target_dir + "/hugo"
is_latest_version = True

# - extract release info from api
# raw_data = requests.get(hugo_release_api, timeout=0.5).text
raw_data = urlopen(hugo_release_api).read().decode()

json_data = json.loads(raw_data)

release_version = json_data['tag_name'].lstrip('v')
release_date = json_data['published_at']
release_pack_link=''
release_pack_size=''

for item in json_data['assets']:
    if re.search(r"Linux-64bit.*tar.gz$",item['name']):
        release_pack_link=item['browser_download_url']
        release_pack_size=item['size']
        break
    else:
        continue

# - if existed latest version
if os.path.exists(hugo_binary_path):
    version_info = subprocess.getoutput(hugo_binary_path + " version")
    version_num = re.search(r".*?v([\d.]+).*", version_info).group(1)

    if version_num == release_version:
        print("Latest version {} existed.".format(release_version))
    else:
        is_latest_version = False
        # remove target dir
        if os.path.exists(target_dir) and os.path.isdir(target_dir):
            rmtree(target_dir)
        print("Local version {} < latest version {}.".format(version_num, release_version))
else:
    is_latest_version = False

if is_latest_version == False:
    # remove target dir
    if os.path.exists(target_dir) and os.path.isdir(target_dir):
        rmtree(target_dir)

if not os.path.exists(target_dir):
    # - download
    pack_save_path = pack_save_dir.rstrip("/") + "/" + release_pack_link.split("/")[-1]

    if os.path.exists(pack_save_path) and os.path.getsize(pack_save_path) == release_pack_size:
        print("Find existed pack {}.".format(pack_save_path))
    else:
        if os.path.exists(pack_save_path):
            os.remove(pack_save_path)
        # https://stackoverflow.com/questions/7243750/download-file-from-web-in-python-3
        with open(pack_save_path, "wb") as file:
            # response = requests.get(release_pack_link)
            # file.write(response.content)

            response = urlopen(release_pack_link)
            file.write(response.read())

            if os.path.exists(pack_save_path) and os.path.getsize(pack_save_path) == release_pack_size:
                print("Successfully download pack {}!".format(pack_save_path))

    # - decompress
    tar = tarfile.open(pack_save_path, 'r:gz')
    # 1. extract all file
    tar.extractall(path=target_dir)

    # 2. extract needed file
    # for item in tar:
    #     if not item.name.endswith(".md"):
    #         tar.extract(item, path=target_dir)

    # - create symlink
    if os.path.exists(hugo_binary_path):
        print("Successfully install {} v{}!".format(utility_name, release_version))

        if os.path.islink(symlink_path):
            os.unlink(symlink_path)
            # os.remove(symlink_path)
        os.symlink(hugo_binary_path, symlink_path)
        print("\nSymlink info: \n{}".format(subprocess.getoutput("ls -lh " + symlink_path)))

    # - remove package
    if os.path.exists(pack_save_path):
        os.remove(pack_save_path)

if os.path.exists(hugo_binary_path):
    print("\nHugo info: \n{}".format(subprocess.getoutput(hugo_binary_path + " version")))

# Script End

operating process

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# sudo python3 ~/hugo.py
Successfully download pack /tmp/hugo_0.52_Linux-64bit.tar.gz!
Successfully install Hugo v0.52!

Symlink info:
lrwxrwxrwx 1 root staff 14 Dec  4 11:28 /usr/local/bin/hugo -> /opt/Hugo/hugo

Hugo info:
Hugo Static Site Generator v0.52 linux/amd64 BuildDate: 2018-11-28T14:06:34Z

# sudo python3 ~/hugo.py
Latest version 0.52 existed.

Hugo info:
Hugo Static Site Generator v0.52 linux/amd64 BuildDate: 2018-11-28T14:06:34Z

Create A New Site

You can reference official document Get Started、Quick Start to know how to create a new site through Hugo.

If this is your first time using Hugo and you’ve already installed Hugo on your machine, we recommend the quick start.

Here I name the site to quickstart (it is just the directory name, not the final site name).

Generated directory structure by Hugo:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# tree -L 2
.
├── archetypes
│   └── default.md
├── config.toml
├── content
│   ├── about.md
│   └── post
├── data
├── layouts
├── static
└── themes
    └── even

8 directories, 3 files

Attention: These files are not the final blog contents, they are just used to generate the blog contents.

Directory content is used to store your blog file (Markdown file).

If you wanna preview the final effect, just executing command hugo server to set up a local server which is listening port 1313. And opening the link in your browser.

1
2


Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Executing command hugo in your project directory, it will generate a directory named public. This directory contains the final blog contents. What you need to do is push all the files in this directory to the branch master of your GitHub repository.

The demonstration process is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


# hugo

                   | EN   
+------------------+-----+
  Pages            | 426  
  Paginator pages  |  24  
  Non-page files   |   0  
  Static files     |  34  
  Processed images |   0  
  Aliases          |  95  
  Sitemaps         |   1  
  Cleaned          |   0  

Total in 403 ms

# tree -L 1
.
├── archetypes
├── config.toml
├── content
├── data
├── layouts
├── public
├── static
└── themes

7 directories, 1 file

Github Repository Operation

Important: The handling ideas and operations of this section are very important. Not involving Git setting procedure.

If you use Github Page to host blog, you need to create a repository named <username>.github.io. Then putting the files generated by Hugo to the branch master of this repository.

I create 2 branches via command git checkout --orphan to store source files and blog contents in directory public separately.

branch code stores source files;
branch master stores blog contents in directory public;

Attention please, this method is different from official document Configuring a publishing source for GitHub Pages.

Create Empty Repository

My GitHub account is MaxdSre, so I need to create a repository named maxdsre.github.io.

After creating the repository, it will prompts

or create a new repository on the command line

1
2
3
4
5
6


echo "# maxdsre.github.io" >> README.md
git init
git add README.md
git commit -m "first commit"
git remote add origin [email protected]:MaxdSre/maxdsre.github.io.git
git push -u origin master

or push an existing repository from the command line

1
2

git remote add origin [email protected]:MaxdSre/maxdsre.github.io.git
git push -u origin master

Pushing Repository

There are 4 steps:

Initializing Git repository via command git init in project directory;
Creating branch code, pushing source files to remote repository;
Executing command hugo to generate public directory; Transferring all the files in this directory to another temporary directory, empty project directory, then move the files store in temporary directory back to project directory;
Creating branch master via command git checkout --orphan, then pushing all files store in current directory to remote repository.

Entering the target directory (here is /PATH/quickstart/), then executing the following commands in sequence.

Branch code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# Initialized empty Git repository in /PATH/.git/
git init

# Switched to a new branch 'code', equal to 'git branch code && git checkout code'
git checkout -b code

# exclude dir public/
echo '/public/' >> .gitignore
sed -r -i '/^\/public\/$/{$!d}' .gitignore

# Add file contents to the index
git add .

# Show the working tree status
# git status

# Record changes to the repository
git commit -m "Hugo generator code"

# Manage set of tracked repositories
git remote add origin [email protected]:MaxdSre/maxdsre.github.io.git

# Update remote refs along with associated objects to branch 'code'
git push -u origin code

Branch master

Executing command hugo in project directly to generate directly public. If you wanna check the branch info of this repository, just executing command git branch -a.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# execute command hugo
hugo

# create temporary dir
HUGO_TEMP_DIR=$(mktemp -d)
cp -R public/* "$HUGO_TEMP_DIR"

# create orphan branch 'master'
git checkout --orphan master

# empty current dir
rm .git/index
git clean -fdx

# copy back contents in dir public/
cp -R "$HUGO_TEMP_DIR"/. .

# add, commit, push
git add .
# git status
git commit -m 'initial blog content'
git push -u origin master

# remove temp dir
[[ -d "$HUGO_TEMP_DIR" ]] && rm -rf "$HUGO_TEMP_DIR"

Branch image

This section is optional. In order to store images in GitHub directly, I create the third branch image. All images used in blog will be saved in this branch.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


git checkout --orphan image
rm .git/index
git clean -fdx

# add content to this branch

git add .
git commit -m 'initial blog images'
git push -u origin image

# fatal: The current branch image has no upstream branch.
# To push the current branch and set the remote as upstream, use
# git push --set-upstream origin image

Switching branch

If you wanna switch your local branch to remote branch remotes/origin/image, you may consider executing the following command

1
2
3
4
5
6
7
8


# git checkout -b <branch> --track <remote>/<branch>
# -b <new_branch>    Create a new branch named <new_branch> and start it at <start_point>;
# -t, --track     When creating a new branch, set up "upstream" configuration.

git checkout -t remotes/origin/image

# local branch, switch from other branch to branch 'code'
git checkout code

Demonstration example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# git branch
* code

# git branch -a
* code
  remotes/origin/HEAD -> origin/code
  remotes/origin/code
  remotes/origin/image
  remotes/origin/master

# git checkout -t remotes/origin/image
Branch 'image' set up to track remote branch 'image' from 'origin'.
  Switched to a new branch 'image'

# git branch
  code
* image

# git branch -a
  code
* image
  remotes/origin/HEAD -> origin/code
  remotes/origin/code
  remotes/origin/image
  remotes/origin/master

Deleting branch

Deleting branches has three types: remote branch, local branch, local remote-tracking branch, more details in How do I delete a Git branch both locally and remotely?.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# 1 - Delete a remote branch
$ git push origin --delete <branch> # Git version 1.7.0 or newer
$ git push origin :<branch> # Git versions older than 1.7.0

# 2 - Delete a local branch
$ git branch --delete <branch>
$ git branch -d <branch> # Shorter version

$ git branch -D <branch> # Force delete un-merged branches

# error: The branch 'image' is not fully merged.
# If you are sure you want to delete it, run 'git branch -D image'.

# 3 - Delete a local remote-tracking branch
$ git branch --delete --remotes <remote>/<branch>
$ git branch -dr <remote>/<branch> # Shorter

$ git fetch <remote> --prune # Delete multiple obsolete tracking branches
$ git fetch <remote> -p # Shorter

Travis CI Configuration

GitHub Pages Deployment is the official document of Travis CI。

Github Access Token Generation

You’ll need to generate a personal access token with the public_repo or repo scope (repo is required for private repositories). Since the token should be private, you’ll want to pass it to Travis securely in your repository settings or via encrypted variables in .travis.yml.

Generating personal access tokens in GitHub page Developer settings. If it is a private repository, choose repo. Otherwise, just choose public_repo. Here I choose public_repo, repo:status, repo_deployment.

The length of the generated token is 40, please keep is private.

Environment Variables Setting

Generated token is used in Travis CI, the page url of target repository in Travis CI is https://travis-ci.org/MaxdSre/maxdsre.github.io/settings.

Don’t forget to check Build only if .travis.yml is present, Build pushed branches, Build pushed pull requests.

The default name of token defined in official document GitHub Pages Deployment is GITHUB_TOKEN, this document alse lists other directives.

Notice that the values are not escaped when your builds are executed. Special characters (for bash) should be escaped accordingly.

Defining variables, it will be used in file .travis.yml latter.

Environment Variables	Value
GITHUB_USERNAME	MaxdSre
GITHUB_EMAIL	[email protected]
GITHUB_TOKEN	75e8b72618ebf48df0b235cp4affd79e167b2489 (pseudo value)
CNAME_URL	axdlog.com

.travis.yml directives

Hugo is written by Golang. So I chose language: go in .travis.yml primitively, but the process of build Golang environment costed too much time (a few minutes). What worse is the container fail to install hugo. I find someone use Python to deploy successfully, then chose Python as operation environment. As the containers of Travis CI is based on Ubuntu, I can use command dpkg, apt-get, but it needs sudo privilege.

As Python failed to build frequently, so I decided to use Golang.

The final code is as follow

Golang

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


# https://docs.travis-ci.com/user/deployment/pages/
# https://docs.travis-ci.com/user/languages/go/
# https://docs.travis-ci.com/user/customizing-the-build/

language: go

python:
    - master

# before_install
# install - install any dependencies required
install:
    - go get github.com/gohugoio/hugo

before_script:
    - rm -rf public 2> /dev/null

# script - run the build script
script:
    - hugo
    - echo "$CNAME_URL" > public/CNAME

deploy:
  provider: pages
  skip-cleanup: true
  github-token: $GITHUB_TOKEN  # Set in travis-ci.org dashboard, marked secure
  email: $GITHUB_EMAIL
  name: $GITHUB_USERNAME
  verbose: true
  keep-history: true
  local-dir: public
  target_branch: master  # branch contains blog content
  on:
    branch: code  # branch contains Hugo generator code

Python (deprecate)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


# https://docs.travis-ci.com/user/deployment/pages/
# https://docs.travis-ci.com/user/languages/python/
language: python

python:
    - "3.6"

install:
    # install latest release version
    - wget -qO- https://api.github.com/repos/gohugoio/hugo/releases/latest | sed -r -n '/browser_download_url/{/Linux-64bit.deb/{[email protected][^:]*:[[:space:]]*"([^"]*)".*@\[email protected];p;q}}' | xargs wget
    - sudo dpkg -i hugo*.deb
    - rm -rf public 2> /dev/null

script:
    - hugo
    - echo "$CNAME_URL" > public/CNAME

deploy:
  provider: pages
  skip-cleanup: true
  github-token: $GITHUB_TOKEN  # Set in travis-ci.org dashboard, marked secure
  email: $GITHUB_EMAIL
  name: $GITHUB_USERNAME
  verbose: true
  keep-history: true
  local-dir: public
  target_branch: master  # branch contains blog content
  on:
    branch: code  # branch contains Hugo generator code

Writing it into file .travis.yml, and push file .travis.yml to the branch code of remote repository.

Continuous Integration

After all operations are finished. If you push your changed source file to branch code. Travis CI will automatically execute directives defined in file .travis.yml, then pushing the generated blog contents to branch master.

Deploying log

References

Change logs

2018.04.11 00:20 Wed America/Boston
- first draft
2018.07.11 12:20 Wed America/Boston
- add python script for hugo installation
2018.08.03 08:44 Fri Asia/Shanghai
- add switch branch
2018.12.04 11:46 Tue America/Boston
- use Golang in travis

Table of Contents