Monday, September 19, 2011

Sysadmin guide to using git and etckeeper for tracking configuration changes

When administering any Linux or Unix system over a period time, the problem of tracking what changes occurred to which files quickly arises. This becomes increasingly difficult when there are multiple system administrators configuring a single server. How do you easily track who made what changes, and when? This is where git comes in handy.

Git is a distributed revision version control system, which enables you to track changes to a file(s) by using a repository, also known as an index. Changes are tracked each commit. A commit can be thought of as a snapshot of a file's current state, similar to virtual machine snapshots in a virtual environment.

The index, which is basically the main database, can be initiated in any directory. However, wherever the Git index is initiated, it will only track files that are a child of that directory. For example, if we were to track /home/localadmin, it would track all files and sub-directories within localadmin's home directory, but nothing above it in the directory tree. A system can have many indexes initiated in different directories.

On Linux and Unix servers, changes primarily occur within the /etc directory. This is where most of the important .conf files are found. When a new package is installed, existing files may change and several new ones are added to /etc. All of these files and changes will eventually need to be automatically committed and added to the git index. This is where etckeeper comes in.

Etckeeper works with git, allowing automatic commits to the /etc directory after each package installation (shown and explained later.) It also tracks file metadeta that revision control systems do not normally support, but are important for /etc, such as permissions of /etc/shadow.

You may have heard of git heavily in the software development community, but I try to show how it can also be leveraged as a powerful tool for system administrators.

NOTE: This tutorial demonstrates instructions based on an Ubuntu 10.04 LTS 32-bit server installation. However, I have successfully configured the same setup on Ubuntu 11.04, Debian 5, Debian 6, CentOS 5.6, CentOS 6.0, and Red Hat Enterprise Linux 6.0.

Installing git and etckeeper

Install the git and etckeeper packages:

$ sudo aptitude -yvVR install git-core etckeeper

  • -y assume yes when prompted.
  • -v verbose output.
  • -V show which versions of packages will be installed.
  • -R do not treat recommended packages as dependencies.

Etckeeper supports other software version control systems (VCS) besides git, double-check the configuration file to make sure it is using git.

$ sudo vi /etc/etckeeper/etckeeper.conf
# The VCS to use.
# VCS="hg"
VCS="git"
# VCS="bzr"
# VCS="darcs"

Now initialize the /etc directory:

$ cd /etc
$ sudo etckeeper init

Now make your first commit:

$ sudo git commit

You will be brought to a screen where you can create a commit message along with the commit itself. Enter a brief description above the commented lines.

Initial commit for server build

* Initializing git repository in /etc

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# Committer: localhost root
#
# On branch master
#
# Initial commit
#
# Changes to be committed:
# (use "git rm --cached …" to unstage)
#
# new file: .etckeeper
# new file: .gitignore
# new file: adduser.conf
# new file: aliases
# new file: alternatives/README
# new file: alternatives/awk
# new file: alternatives/awk.1.gz
# new file: alternatives/builtins.7.gz

In each commit message, git will display which files will be commited to the repository.

sudo is required for running the git commands because the repository was initialized in /etc by sudo; it is readable and writable only by root.

Any git command should be run after changing your working directory to /etc.

Making commits

After the first initialization, if a file is changed, git will notice the change.

$ sudo git status

  • Show status of current repository, changes, etc.

rsabalburo@localhost:/etc$ sudo git status

# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: issue
#
no changes added to commit (use "git add" and/or "git commit -a")

Above I added the line "Add new message of the day." to the /etc/issue file.

Using the following command, we can see what changes happened to /etc/issue since the previous commit:

$ sudo git diff

rsabalburo@localhost:/etc$ sudo git diff

diff --git a/issue b/issue
index 9029f91..2100bfd 100644
--- a/issue
+++ b/issue
@@ -1,2 +1,4 @@
+A new message of the day.
+
Ubuntu 10.04 LTS n l

  • The diff output is identical to the regular diff command (using -u unified output switch).
  • The + shows the changes that were added since the previous commit.


Now that we are satisfied with the change we can git commit it:

$ sudo git commit -a

  • -a commit all changes.

Again, you will be prompted to create a description along with your commit.

Add a new message of the day for users logging in

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# Committer: localhost root
#
# On branch master
# Changes to be committed:
# (use "git reset HEAD …" to unstage)
#
# modified: issue
#

If you made an error in your commit message or would simply like to change it, you can go back and edit it with git commit --amend:

$ sudo git commit --amend

  • This will only allow you to change the commit message of your most recent commit.

NOTE: You generally cannot amend any commit messages past the previous commit, doing so requires you to rewrite the entire git repository which is a very complicated process.

Important: If you have more than one modified file showing up in the "Changes to be committed:" staging area, you can commit each single file rather than all at once (-a), which is highly recommended.

$ sudo git commit

How do I display commits I've made?

To view a log of all the commits that you have made:

$ sudo git log

rsabalburo@localhost:/etc$ sudo git log

commit d23affc7b89da8966826ab0fef6891d87dbd3ee1
Author: localhost root
Date: Thu Jun 17 12:37:55 2010 -1000

Add a new message of the day for users logging in

commit 30636b4bdfb84cdd4db756df60dcc2c49b53842b
Author: localhost root
Date: Thu Jun 17 11:43:47 2010 -1000

Initial commit for server build

* Initializing git repository in /etc

Each commit message has a unique SHA-1 hash, commit author, date of commit, and description of commit.

In the above git log, the author is currently localhost root. It is recommended that you change author and email address, especially if there are multiple administrators configuring the same server.

$ git config --global user.name "Rodolf Sabalburo"
$ git config --global user.email "rsabalburo@localhost"

Notice sudo is not needed for these commands. These commands will automatically create the .gitconfig file in the current users home directory:

hccitc@localhost:~$ cd ~/
hccitc@localhost:~$ cat .gitconfig

[core]
editor = vim
[user]
name = Rodolf Sabalburo
email = rsabalburo@localhost

From this point on, Rodolf Sabalburo will display in the commit log for any commit made by the user rsabalburo:

rsabalburo@localhost:/etc$ sudo git log

commit cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b
Author: Rodolf Sabalburo
Date: Thu Jun 17 13:57:07 2010 -1000

Add hosts entry for 192.168.1.1

* Showing different author and email in commit message.

commit d23affc7b89da8966826ab0fef6891d87dbd3ee1
Author: localhost root
Date: Thu Jun 17 12:37:55 2010 -1000

Add a new message of the day for users logging in

commit 30636b4bdfb84cdd4db756df60dcc2c49b53842b
Author: localhost root
Date: Thu Jun 17 11:43:47 2010 -1000

Initial commit for server build

* Initializing git repository in /etc

Important: Ensure that other privileged users set their git user.name and user.email, otherwise the commit author will show up as localhost root .

The git log command supports several additional switches, one of the most important being the -u switch which displays a diff unified output for each commit:

rsabalburo@localhost:/etc$ sudo git log

commit cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b
Author: Rodolf Sabalburo
Date: Thu Jun 17 13:57:07 2010 -1000

Add hosts entry for 192.168.1.1

* Showing different author and email in commit message.

diff --git a/hosts b/hosts
index 4eaa7c5..12a111b 100644
--- a/hosts
+++ b/hosts
@@ -1,5 +1,6 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
+192.168.1.1 localgateway

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback

commit d23affc7b89da8966826ab0fef6891d87dbd3ee1
Author: localhost root
Date: Thu Jun 17 12:37:55 2010 -1000

Add a new message of the day for users logging in

diff --git a/issue b/issue
index 9029f91..2100bfd 100644
--- a/issue
+++ b/issue
@@ -1,2 +1,4 @@
+A new message of the day.
+
Ubuntu 10.04 LTS n l

It also supports the --color switch, sudo git log -u --color which displays commit messages in yellow, removed text/lines in red, added text/lines in green.

To make the color behavior default use git config:

$ git config --global color.diff auto

Additional git log switches:

sudo git log --pretty=oneline
sudo git log --pretty=email
sudo git log --decorate

See 'man git-log' for more and the Additional git commands near the end of the tutorial for examples.

How do I compare different commits?

When viewing the different commits, you can compare changes with the git diff command:

$ sudo git diff -u

  • You do not have to type the entire hash, the first 4-6 hash charactes will suffice.

rsabalburo@localhost:/etc$ sudo git diff -u 30636b a208e373

diff --git a/hosts b/hosts
index 4eaa7c5..b5fe3df 100644
--- a/hosts
+++ b/hosts
@@ -1,5 +1,7 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
+192.168.1.1 localgateway
+192.168.1.2 localproxy

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
diff --git a/issue b/issue
index 9029f91..2100bfd 100644
--- a/issue
+++ b/issue
@@ -1,2 +1,4 @@
+A new message of the day.
+
Ubuntu 10.04 LTS n l

You can compare commits using git log as well.

By default when issuing git log under /etc, it will display ALL commits within the /etc git repository.

You can run git log on a single file (or directory) to see all changes made to it:

$ sudo git log

  • -u unified diff output

rsabalburo@localhost:/etc$ sudo git log -u hosts

commit a208e373fc8c5ddff86e752128bada592ccd2f60
Author: Rodolf Sabalburo
Date: Thu Jun 17 14:26:24 2010 -1000

Add new host entry for 192.168.1.2 localproxy

* Make more changes to hosts to demonstrate git log on single file.

diff --git a/hosts b/hosts
index 12a111b..b5fe3df 100644
--- a/hosts
+++ b/hosts
@@ -1,6 +1,7 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
192.168.1.1 localgateway
+192.168.1.2 localproxy

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback

commit cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b
Author: Rodolf Sabalburo
Date: Thu Jun 17 13:57:07 2010 -1000

Add hosts entry for 192.168.1.1

* Showing different author and email in commit message.

diff --git a/hosts b/hosts
index 4eaa7c5..12a111b 100644
--- a/hosts
+++ b/hosts
@@ -1,5 +1,6 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
+192.168.1.1 localgateway

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback

Tracking new files

Git will track new files added to the repository (any file under /etc because we initiated the repository here) and display a notification in it's status output.

rsabalburo@localhost:/etc/shorewall$ sudo cp /usr/share/doc/shorewall/default-config/masq .

rsabalburo@localhost:/etc/shorewall$ sudo git status
# On branch master
# Untracked files:
# (use "git add …" to include in what will be committed)
#
# masq
nothing added to nothing added to commit but untracked files present (use "git add" to track)

Add the file or directory using git add:

$ sudo git add

rsabalburo@localhost:/etc/shorewall$ sudo git add masq

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
# Changes to be committed:
# (use "git reset HEAD …" to unstage)
#
# new file: masq
#

Git will now notify you that the masq file has changes to be commited. After committing the changes, git status will display that there are none.

rsabalburo@localhost:/etc/shorewall$ sudo git commit -a
[master 23d7db8] Copy masq file to shorewall/
1 files changed, 11 insertions(+), 0 deletions(-)
create mode 100644 shorewall/masq

rsabalburo@localhost:/etc/shorewall$ sudo git status
# On branch master
nothing to commit (working directory clean)

How do I untrack files?

If there is a file that you do not want committed or tracked by git use the git rm --cached command along with the .gitignore file.

rsabalburo@localhost:/etc/shorewall$ sudo git rm --cached masq
rm 'shorewall/masq'

rsabalburo@localhost:/etc/shorewall$ sudo git status
# On branch master
# Changes to be committed:
# (use "git reset HEAD …" to unstage)
#
# deleted: masq
#
# Untracked files:
# (use "git add …" to include in what will be committed)
#
# masq

Notice that the masq file shows "deleted", this does not mean it has been deleted from the system only deleted from the git repository. The masq file still shows up under the Untracked files staging area.

rsabalburo@localhost:/etc/shorewall$ sudo git commit -a

[master bf06da0] Remove shorewall/masq from git repository
1 files changed, 0 insertions(+), 11 deletions(-)
delete mode 100644 shorewall/masq


rsabalburo@localhost:/etc/shorewall$ sudo git log

commit bf06da0f52c16002dbf1a134ffc97b8f30d22673
Author: Rodolf Sabalburo
Date: Thu Jun 17 14:56:04 2010 -1000

Remove shorewall/masq from git repository

* masq itself is not deleted from the file system, only from the
repository.
* To remove from staging area, use .gitignore file.


rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
# Untracked files:
# (use "git add …" to include in what will be committed)
#
# masq

To untrack masq completely, add the file to /etc/.gitignore. This file is responsible for ignoring any files or directories to be tracked from git.

hccitc@localhost:/etc$ sudo vi .gitignore
# Ignore masq file to demonstrate .gitignore
shorewall/masq


rsabalburo@localhost:/etc$ sudo git status
# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: .gitignore
#

Git also tracks the .gitignore file, commit the change describing which file was ignored in the commit message.

rsabalburo@localhost:/etc$ sudo git commit -a
[master 13957e9] Add shorewall/masq to .gitignore
1 files changed, 6 insertions(+), 0 deletions(-)

rsabalburo@localhost:/etc$ sudo git log
commit 13957e9dc87a5207a11081298f48ff37c1011b6b
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:01:12 2010 -1000

Add shorewall/masq to .gitignore

* This file will no longer be tracked or show up in untracked files
staging area.

commit bf06da0f52c16002dbf1a134ffc97b8f30d22673
Author: Rodolf Sabalburo
Date: Thu Jun 17 14:56:04 2010 -1000

Remove shorewall/masq from git repository

* masq itself is not deleted from the file system, only from the
repository.
* To remove from staging area, use .gitignore file.

How do I reset to an older commit?

In the instance you need to rollback to an earlier or snapshot, use git reset --hard.

$ sudo git reset --hard

WARNING!: Be very careful with this command! Not only does it abandon any uncommitted changes when rolling back to a previous commit i.e. any changes you have not committed will not be saved and wiped out, it also wipes out any changes in commits found after the one you are resetting to! ONLY perform a hard reset if you are absolutely sure you want to reset to an earlier commit.

For example if I had made changes to /etc/shorewall/params and /etc/shorewall/rules and wanted to roll back prior those changes:

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)

rsabalburo@localhost:/etc/shorewall$ sudo git log --pretty=oneline

9f3a7708d600ed44913d1f7a6cacc0919825740f Add rule to allow ssh from net zone to $FW
d8c2337ed3127a9d091a3075857ae46b6d7284a2 Add iv-rover to trusted ssh jumpserver variable
8512a7dac4565bf6ec8fb459ca4faf0f08471429 Add shorewall/masq to .gitignore
bf06da0f52c16002dbf1a134ffc97b8f30d22673 Remove shorewall/masq from git repository

The commit I would want to reset to would be "8512a… Add shorewall/masq to .gitignore" using:

$ sudo git reset --hard 8512a7d

rsabalburo@localhost:/etc/shorewall$ sudo git reset --hard 8512a7d

HEAD is now at 8512a7d Add shorewall/masq to .gitignore


rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)


rsabalburo@localhost:/etc/shorewall$ sudo git log

commit 8512a7dac4565bf6ec8fb459ca4faf0f08471429
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:11:23 2010 -1000

Add shorewall/masq to .gitignore

* This file will no longer be tracked or show up in untracked files
staging area.

Undoing a reset

In the event you do a hard reset, it is still possible to recover by finding the commit using git reflog. reflog is a history log of all actions regarding commits in git.

$ sudo git reflog

rsabalburo@localhost:/etc/shorewall$ sudo git reflog

8512a7d HEAD@{0}: 8512a7d: updating HEAD
9f3a770 HEAD@{1}: commit: Add rule to allow ssh from net zone to $FW
d8c2337 HEAD@{2}: commit: Add iv-rover to trusted ssh jumpserver variable
8512a7d HEAD@{3}: commit: Add shorewall/masq to .gitignore

  • Notice "updating HEAD" in reference to commit "8512a7d Add shorewall/masq to .gitignore", which we previously reset to.
  • We want to roll back to 9f3a770 the most recent commit before our reset.

rsabalburo@localhost:/etc/shorewall$ sudo git reset --hard 9f3a770

HEAD is now at 9f3a770 Add rule to allow ssh from net zone to $FW


rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)


rsabalburo@localhost:/etc/shorewall$ sudo git log

commit 9f3a7708d600ed44913d1f7a6cacc0919825740f
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:15:48 2010 -1000

Add rule to allow ssh from net zone to $FW

commit d8c2337ed3127a9d091a3075857ae46b6d7284a2
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:15:17 2010 -1000

Add iv-rover to trusted ssh jumpserver variable

commit 8512a7dac4565bf6ec8fb459ca4faf0f08471429
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:11:23 2010 -1000

Add shorewall/masq to .gitignore

* This file will no longer be tracked or show up in untracked files
staging area.

The git repository head is now at the commit before our initial reset.

How do I reset to the previous commit?

If you just want to reset or revert a just a SINGLE file to the previous commit use:

$ sudo git checkout

For example, lets say I add a PING rule to /etc/shorewall/rules and test it out. I did not commit the changes yet because they didn't work -- I can simply reset the single file to the most previous commit.

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: rules
#
no changes added to commit (use "git add" and/or "git commit -a")


rsabalburo@localhost:/etc/shorewall$ sudo git diff

diff --git a/shorewall/rules b/shorewall/rules
index 09dfd26..b47ea62 100644
--- a/shorewall/rules
+++ b/shorewall/rules
@@ -20,7 +20,8 @@ SSH(ACCEPT) net $FW

# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

-Ping(DROP) net $FW
+#Ping(DROP) net $FW
+Ping(ACCEPT) net $FW

# Permit all ICMP traffic FROM the firewall TO the net zone


rsabalburo@localhost:/etc/shorewall$ sudo git checkout rules

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)

If you just want to reset or revert ALL files (which were changed, but uncommited) to the previous commit use:

$ sudo git reset --hard

In this example, lets say I add a PING rule to /etc/shorewall/rules and change a policy in /etc/shorewall/policy. If I test out these changes and I'm not satisfied, I can reset everything to the most previous commit rather than having to use git checkouton each individual file.

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: policy
# modified: rules
#
no changes added to commit (use "git add" and/or "git commit -a")


rsabalburo@localhost:/etc/shorewall$ sudo git diff

diff --git a/shorewall/policy b/shorewall/policy
index dd1d7a6..f43371a 100644
--- a/shorewall/policy
+++ b/shorewall/policy
@@ -15,8 +15,8 @@

# RS: If you want open access to the Internet from the firewall, change the
# $FW to net policy to 'ACCEPT' and remove 'info' under LOG LEVEL.
-#$FW net ACCEPT
-$FW net REJECT info
+$FW net ACCEPT
+#$FW net REJECT info
net all DROP info
# The FOLLOWING POLICY MUST BE LAST
all all REJECT info
diff --git a/shorewall/rules b/shorewall/rules
index 09dfd26..b47ea62 100644
--- a/shorewall/rules
+++ b/shorewall/rules
@@ -20,7 +20,8 @@ SSH(ACCEPT) net $FW

# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

-Ping(DROP) net $FW
+#Ping(DROP) net $FW
+Ping(ACCEPT) net $FW

# Permit all ICMP traffic FROM the firewall TO the net zone



rsabalburo@localhost:/etc/shorewall$ sudo git reset --hard

HEAD is now at 9f3a770 Add rule to allow ssh from net zone to $FW


rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)

How do I reset a SINGLE file to a much older commit?

If you have a single file you want to revert to a much older commit e.g. maybe you want a revert a configuration file to it's original form in one of your first commits, use the git checkout command and specify the hash and filename as arguments.

sudo git checkout

  • Below I use sudo git checkout cae7 hosts, cae7 is the hash, hosts is the filename.

  • I found the correct hash I wanted to revert to by using sudo git log -u hosts.

rsabalburo@localhost:/etc$ sudo git status

# On branch master
nothing to commit (working directory clean)


rsabalburo@localhost:/etc$ head hosts

127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
192.168.1.1 localgateway
192.168.1.2 localproxy

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes


rsabalburo@localhost:/etc$ sudo git show cae7

commit cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b
Author: Rodolf Sabalburo
Date: Thu Jun 17 13:57:07 2010 -1000

Add hosts entry for 192.168.1.1

* Showing different author and email in commit message.

diff --git a/hosts b/hosts
index 4eaa7c5..12a111b 100644
--- a/hosts
+++ b/hosts
@@ -1,5 +1,6 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
+192.168.1.1 localgateway

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback


rsabalburo@localhost:/etc$ sudo git checkout cae7 hosts


rsabalburo@localhost:/etc$ sudo git status

# On branch master
# Changes to be committed:
# (use "git reset HEAD …" to unstage)
#
# modified: hosts
#


rsabalburo@localhost:/etc$ head hosts

127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
192.168.1.1 localgateway

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Git shows in the staging area that hosts has been modified (in this case, to our previous commit cae7…). The hosts file would still need to be commited if we were satisfied with the changes. If not satisfied, we can reset to the previous commit using git reset --hard.

How do I modify my previous commit and include additional changes without using a reset?

For example if I added the '192.168.1.3 homecomputer' host entry to /etc/hosts and commit the changes:

rsabalburo@localhost:/etc$ head /etc/hosts

127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
192.168.1.1 localgateway
192.168.1.2 localproxy
192.168.1.3 homecomputer

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix


rsabalburo@localhost:/etc$ sudo git log | head

commit c853bfcc0fd65e37be4959f208af8122e1390806
Author: Rodolf Sabalburo
Date: Thu Jun 24 12:08:04 2010 -1000

Add new host entry for 192.168.1.3 homecomputer

commit ac1d59be8222ff8f0d956434c2c2ba57e938047e
Author: Rodolf Sabalburo
Date: Wed Jun 23 13:55:15 2010 -1000

committing changes in /etc after apt run

Package changes:
+liblzo2-2 2.03-2
+libpkcs11-helper1 1.07-1build1
+openssl-blacklist 0.5-2
+openvpn 2.1.0-1ubuntu1
+openvpn-blacklist 0.4

If I felt that I could have included another host entry to the same commit I could use git commit --amend on the file after I made the changes:

$ sudo git commit --amend

rsabalburo@localhost:/etc$ sudo vi hosts

(Here I added the homeserver entry)

rsabalburo@localhost:/etc$ sudo git diff

diff --git a/hosts b/hosts
index 77fdaee..0f0ac1f 100644
--- a/hosts
+++ b/hosts
@@ -3,6 +3,7 @@
192.168.1.1 localgateway
192.168.1.2 localproxy
192.168.1.3 homecomputer
+192.168.1.4 homeserver

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback


rsabalburo@localhost:/etc$ sudo git status

# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: hosts
#
no changes added to commit (use "git add" and/or "git commit -a")


rsabalburo@localhost:/etc$ sudo git log

commit c853bfcc0fd65e37be4959f208af8122e1390806
Author: Rodolf Sabalburo
Date: Thu Jun 24 12:08:04 2010 -1000

Add new host entry for 192.168.1.3 homecomputer

commit ac1d59be8222ff8f0d956434c2c2ba57e938047e
Author: Rodolf Sabalburo
Date: Wed Jun 23 13:55:15 2010 -1000

committing changes in /etc after apt run

Package changes:
+liblzo2-2 2.03-2
+libpkcs11-helper1 1.07-1build1
+openssl-blacklist 0.5-2
+openvpn 2.1.0-1ubuntu1
+openvpn-blacklist 0.4


rsabalburo@localhost:/etc$ sudo git commit --amend /etc/hosts

(update our new change, create a commit message)

[master bfdb9ff] Add new host entry for 192.168.1.3 homecomputer and 192.168.1.4 homeserver
2 files changed, 2 insertions(+), 2 deletions(-)


rsabalburo@localhost:/etc$ sudo git log

commit bfdb9ffc7956a6123d447f0996b9f23b6858abce
Author: Rodolf Sabalburo
Date: Thu Jun 24 12:08:04 2010 -1000

Add new host entry for 192.168.1.3 homecomputer and 192.168.1.4 homeserver

commit ac1d59be8222ff8f0d956434c2c2ba57e938047e
Author: Rodolf Sabalburo
Date: Wed Jun 23 13:55:15 2010 -1000

committing changes in /etc after apt run

Package changes:
+liblzo2-2 2.03-2
+libpkcs11-helper1 1.07-1build1
+openssl-blacklist 0.5-2
+openvpn 2.1.0-1ubuntu1
+openvpn-blacklist 0.4

Notice the previous commit was overwritten after we used the git commit --amend command.

You can also include changes made to several other files if needed:

$ sudo git commit --amend

  • Redo previous commit, and include changes made to , , etc.

etckeeper

Etckeeper performs the simple role of automatically committing changes within /etc after package installations, shown below:

rsabalburo@localhost:/etc$ sudo git status

# On branch master
nothing to commit (working directory clean)


rsabalburo@localhost:/etc$ sudo aptitude -yvVR install openvpn

Reading package lists… Done
Building dependency tree
Reading state information… Done
Reading extended state information
Initializing package states… Done
Writing extended state information… Done
The following NEW packages will be installed:
liblzo2-2{a} [2.03-2] libpkcs11-helper1{a} [1.07-1build1] openssl-blacklist{a} [0.5-2] openvpn [2.1.0-1ubuntu1] openvpn-blacklist{a} [0.4]
The following packages are SUGGESTED but will NOT be installed:
resolvconf
0 packages upgraded, 5 newly installed, 0 to remove and 25 not upgraded.
Need to get 0B/7,937kB of archives. After unpacking 16.3MB will be used.
Writing extended state information… Done
Preconfiguring packages …
(Reading database … 44756 files and directories currently installed.)
Unpacking openssl-blacklist (from …/openssl-blacklist_0.5-2_all.deb) …
Unpacking liblzo2-2 (from …/liblzo2-2_2.03-2_i386.deb) …
Unpacking libpkcs11-helper1 (from …/libpkcs11-helper1_1.07-1build1_i386.deb) …
Unpacking openvpn-blacklist (from …/openvpn-blacklist_0.4_all.deb) …
Unpacking openvpn (from …/openvpn_2.1.0-1ubuntu1_i386.deb) …
Processing triggers for man-db …
Processing triggers for ureadahead …
Setting up openssl-blacklist (0.5-2) …
Setting up liblzo2-2 (2.03-2) …

Setting up libpkcs11-helper1 (1.07-1build1) …

Setting up openvpn-blacklist (0.4) …
Setting up openvpn (2.1.0-1ubuntu1) …
* Restarting virtual private network daemon(s)… * No VPN is running.

Processing triggers for libc-bin …
ldconfig deferred processing now taking place
[master ac1d59b] committing changes in /etc after apt run
Author: Rodolf Sabalburo
13 files changed, 412 insertions(+), 0 deletions(-)
create mode 100644 bash_completion.d/openvpn
create mode 100644 default/openvpn
create mode 100755 init.d/openvpn
create mode 100755 network/if-down.d/openvpn
create mode 100755 network/if-up.d/openvpn
create mode 100755 openvpn/update-resolv-conf
create mode 120000 rc0.d/K80openvpn
create mode 120000 rc1.d/K80openvpn
create mode 120000 rc2.d/S16openvpn
create mode 120000 rc3.d/S16openvpn
create mode 120000 rc4.d/S16openvpn
create mode 120000 rc5.d/S16openvpn
create mode 120000 rc6.d/K80openvpn
Reading package lists… Done
Building dependency tree
Reading state information… Done
Reading extended state information
Initializing package states… Done
Writing extended state information… Done

Current status: 0 broken [+0], 25 updates [+0], 29045 new [+0].


rsabalburo@localhost:/etc$ sudo git show

commit ac1d59be8222ff8f0d956434c2c2ba57e938047e
Author: Rodolf Sabalburo
Date: Wed Jun 23 13:55:15 2010 -1000

committing changes in /etc after apt run

Package changes:
+liblzo2-2 2.03-2
+libpkcs11-helper1 1.07-1build1
+openssl-blacklist 0.5-2
+openvpn 2.1.0-1ubuntu1
+openvpn-blacklist 0.4

  • Notice the "create mode"s on files that were added to the system.
  • Also an automatic commit message is generated show in the git log.
  • If you want to change this message and be more verbose on the packages installed you can use sudo git commit --amend.

Stanardized git commit message format

Here I have standardized my git commit messages to keep git logs uniform, tidy, and easy to read.

  • The first line should be a summary or short phrase of what change was made on which file.
  • Use present tense only, and omit period or other punctuation at end.
  • Reference the file from relative path, (everything is a child of /etc) e.g. shorewall/rules.
  • Second line should be blank.
  • Third line indented with three spaces, use an asterisk * as a bullet to describe in more detail about change.
  • Feel free to be as verbose as possible, and make notes for self and other administrators.
  • Use proper punctuation and grammar.

Example of a good commit message:

Add SNMP rule from net zone to firewall zone in shorewall/rules

* Allow snmp traffic to reach firewall.
* This will remain commented out for now as a hook, enable when needed.
* Note that Shorewall 4.0 to 4.4 have different "macro" formats.

Example of my git log from an OpenVPN server I manage:

commit 94591e98136adcbe10212e69b30964326044dad9
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:49:41 2010 -1000

Update shorewall/params add OpenVPN variables

* Add variable for NET_OPENVPN_TRUST and ROAD_OPENVPN_TRUST.
* See shorewall/params for description of variables.

commit b2bb776167537cb6cfcead81e1cca90766e5ae80
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:39:23 2010 -1000

Fix shorewall/policy

* Add explicit policy for net -> $FW drop, with logging.

commit b28940adfdfa4c2b5370831c990d84d1364c52c1
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:37:08 2010 -1000

Fix shorewall/interfaces

* Change eth0 interface to eth+.
- + denotes wildcard, or "all eth interfaces".

commit 974cd9f34c4f06ecabea414006453d610e2af0e3
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:27:16 2010 -1000

Deprecate shorewall/tunnels file

* tunnels to be deprecated, use rules-hcc.inc instead to allow related
inbound and outbound OpenVPN traffic.
* tunnels creates open-type policies in iptables ($FW -> net udp openvpn,
net -> $FW udp openvpn) that take higher priority in the net2fw chain.
* rules-hcc.inc will be used to accomplish close-type policies, only
allowing trusted hosts or networks from the net zone.
* See "Eliminating the /etc/shorewall/tunnels file"
http://www.shorewall.net/VPNBasics.html

commit 6cbab1296f351e069857a035df077fa55911a745
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:19:07 2010 -1000

Build is now a release candidate: ubuntu1004-x86-hccvpn-rc1

* Hostname change using /usr/local/sbin/hcc-ubuntu-hostname-change.

Additional git commands

Change git commit editor

For some distributions the default editor for commit messages is nano or emacs, this can be changed with the following command to vim (or vi):

$ git config --global core.editor "vim"

  • On a per user basis like user.name and user.email options; sudo not needed.

Commit verbosely

You can commit verbosely i.e. include the diff of the contents being committed in the commit message screen with:

$ sudo git commit -a -v
$ sudo git commit -av

  • You can commit a single file if instead, replace -a with filename.

git blame

git blame will show what revision and author last modified each line of a file. Example:

$ sudo git blame

rsabalburo@localhost:/etc$ sudo git blame /etc/hosts

^30636b4 (localhost root 2010-06-17 11:43:47 -1000 1) 127.0.0.1 localhost
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 2) 127.0.1.1 localhost.localdomain localhost
cae791d4 (Rodolf Sabalburo 2010-06-17 13:57:07 -1000 3) 192.168.1.1 localgateway
a208e373 (Rodolf Sabalburo 2010-06-17 14:26:24 -1000 4) 192.168.1.2 localproxy
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 5)
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 6) # The following lines are desirable for IPv6 capable hosts
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 7) ::1 localhost ip6-localhost ip6-loopback
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 8) fe00::0 ip6-localnet
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 9) ff00::0 ip6-mcastprefix
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 10) ff02::1 ip6-allnodes
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 11) ff02::2 ip6-allrouters

  • Notice lines 3 and 4 of /etc/hosts was edited by Rodolf Sabalburo, both on June 17.
  • The far left hand column shows the truncated commit hash.

More git log switches

$ sudo git log --pretty=oneline

ac1d59be8222ff8f0d956434c2c2ba57e938047e committing changes in /etc after apt run
9f3a7708d600ed44913d1f7a6cacc0919825740f Add rule to allow ssh from net zone to $FW
d8c2337ed3127a9d091a3075857ae46b6d7284a2 Add iv-rover to trusted ssh jumpserver variable
8512a7dac4565bf6ec8fb459ca4faf0f08471429 Add shorewall/masq to .gitignore
bf06da0f52c16002dbf1a134ffc97b8f30d22673 Remove shorewall/masq from git repository
23d7db8e818ff27c789b229f47d8614a99f6ce11 Copy masq file to shorewall/
a208e373fc8c5ddff86e752128bada592ccd2f60 Add new host entry for 192.168.1.2 localproxy
cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b Add hosts entry for 192.168.1.1
d23affc7b89da8966826ab0fef6891d87dbd3ee1 Add a new message of the day for users logging in
30636b4bdfb84cdd4db756df60dcc2c49b53842b Initial commit for server build.

$ sudo git log --pretty=oneline --abbrev-commit

ac1d59b committing changes in /etc after apt run
9f3a770 Add rule to allow ssh from net zone to $FW
d8c2337 Add iv-rover to trusted ssh jumpserver variable
8512a7d Add shorewall/masq to .gitignore
bf06da0 Remove shorewall/masq from git repository
23d7db8 Copy masq file to shorewall/
a208e37 Add new host entry for 192.168.1.2 localproxy
cae791d Add hosts entry for 192.168.1.1
d23affc Add a new message of the day for users logging in
30636b4 Initial commit for server build.

$ sudo git log --pretty=email

From 9f3a7708d600ed44913d1f7a6cacc0919825740f Mon Sep 17 00:00:00 2001
From: Rodolf Sabalburo
Date: Thu, 17 Jun 2010 15:15:48 -1000
Subject: [PATCH] Add rule to allow ssh from net zone to $FW


From d8c2337ed3127a9d091a3075857ae46b6d7284a2 Mon Sep 17 00:00:00 2001
From: Rodolf Sabalburo
Date: Thu, 17 Jun 2010 15:15:17 -1000
Subject: [PATCH] Add iv-rover to trusted ssh jumpserver variable


From 8512a7dac4565bf6ec8fb459ca4faf0f08471429 Mon Sep 17 00:00:00 2001
From: Rodolf Sabalburo
Date: Thu, 17 Jun 2010 15:11:23 -1000
Subject: [PATCH] Add shorewall/masq to .gitignore

* This file will no longer be tracked or show up in untracked files
staging area.

From bf06da0f52c16002dbf1a134ffc97b8f30d22673 Mon Sep 17 00:00:00 2001
From: Rodolf Sabalburo
Date: Thu, 17 Jun 2010 14:56:04 -1000
Subject: [PATCH] Remove shorewall/masq from git repository

* masq itself is not deleted from the file system, only from the
repository.
* To remove from staging area, use .gitignore file.

  • See man git-log for additional switches.

Best practices and tips

  • Git commits should be atomic i.e. make a commit anytime you make a single change to a file or similar changes across a few files.
  • For example, if I had the same IP I had to change in multiple configuration files I can commit it all as one with a short description describing why the IP was changed.
  • If I changed /etc/fstab and /etc/hosts; they should be commited separately as they are entirely unrelated.
  • Always set the git author immediately using git config user.name and user.email.
  • Use git status and git diff often to see the progress of your work, and if there were any changes you did not commit yet.
  • Anytime you make a change, whether a commit, revert, reset; git will always update the timestamp on the related file!

Reading diff output

The git log command supports the diff -u (unified format) output, which is much easier to read, and allows the log to show differences made to each commit:

rsabalburo@localhost:/etc/shorewall$ sudo git log -u

commit 9f3a7708d600ed44913d1f7a6cacc0919825740f
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:15:48 2010 -1000

Add rule to allow ssh from net zone to $FW

diff --git a/shorewall/rules b/shorewall/rules
index 0e33b66..09dfd26 100644
--- a/shorewall/rules
+++ b/shorewall/rules
@@ -14,6 +14,10 @@
#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE USER/ MARK
# PORT PORT(S) DEST LIMIT GROUP

+# Allow ssh connections to $FW
+
+SSH(ACCEPT) net $FW
+
# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

Ping(DROP) net $FW

Notice the --- a/shorewall/rules and the +++ b/shorewall/rules. These are the files that are being compared. In this case, the older rules file and the newer rules file.

  • Spaces are where the two files are the same. (no - or + )
  • - denotes changes in the previous file, but were usually removed in the newer file.
  • + denotes changes in newer file, usually additions.

Also notice the format @@ -14,6 +14,10 @@. These are called change hunks or chunks. The -14 denotes the first/older file line number, and the 6 is the line range. This means that at line 14 in the older file there were changes within 6 lines. The second part +14 denotes the newer file, and 10 is again the line range. In other words, line 14 onward for 10 lines there is a change.

By default the -u switch gives 3 lines of context, or 3 lines BEFORE and AFTER there was a change. This can be changed with the -UN format, where N is the number of lines before or after the changes that show. If you only want to see changes without surrounding context use:

$ sudo git log -U0

Resources

Very helpful Git command cheat sheet: http://cheat.errtheblog.com/s/git
Git on Wikipedia: http://en.wikipedia.org/wiki/Git_(software)
Git for the lazy: http://www.spheredev.org/wiki/Git_for_the_lazy
Starting git using just 10 commands: http://blog.xkoder.com/2008/08/13/git-tutorial-starting-with-git-using-just-10-commands/

If this is your first time reading this tutorial, I recommend reading it again and practicing it hands-on to fully understand it.

4 comments:

  1. dear rodolf, thank you. your intro of git is good and concise!

    ReplyDelete
  2. Nice Ty im using git etckeeper for my thesis this blog helps me alot

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Can we schedule it using the crontab?

    ReplyDelete