![\"\"](\"https://pic.blog.plover.com/prog/git/history-proposal/current-practice.svg\")
While I was still at ZipRecruiter, our Git monorepo was developing an\nannoying problem: it had too many refs.
\nWhenever there was a deployment of any of the applications in the\nrepo, to staging or to production, the deployment tool would create a\ntag object with a name like this:
\nrefs/release/stg.www.our-app-2019-09-06T114410.954-0700-treesha-179dab6fd11bee5195f0f8e31cea3edfb86ae23bThe tag contained the name of the target tier of the deployment (stg here, for\n“staging”), the app name and class of hosts on which it ran,\nwww.our-app in this case, the date\nand time, and the hash of the tree that was deployed. This made a\ncomplete record of what was deployed, where, and when.
After we had been doing this for a while, there were thousands of\nthese tag objects. Git is well-designed for handling thousands of\ncommits, but not for handling thousands of refs. Perhaps this has\nchanged, but at the time, Git would store refs in a file called\npacked-refs, which, for each ref, contained an entry with the ref's\nname and the ID of the object to which it referred.
(As with all things Git, the actual situation is\nsomewhat more complicated. These additional ramifications don't affect\nthe following discussion.)
\nAny operation that required resolving a ref name to its object had to\nsearch through the packed-refs file, which was taking an appreciable\namount of time, far more than was desirable for an operation that\neveryone did many times a day. This affected not just local operations\nbut operations on the Git server as well, and it was clear that it was\nonly going to get slower as time went on. Also, the outputs of\ncommands like git fetch and git branch were becoming increasingly\ncluttered with the names of tag objects whose only purpose was to\nrecord release history.
It seemed reasonable to me record the deployment history in the\nrepository, but using thousands of tag objects clearly had serious\ndrawbacks. In addition to the performance problems, the metadata\nitself was purely a convention in the tag name format, not legible to\nGit itself. There was a tree ID there, but nothing enforcing the rule\nthat the tag should refer to a commit with that tree.
\nAlso, the tag objects themselves seemed poorly-designed for the\npurpose. One good reason for storing history data in the respoitory is to\nrender it immutable. But tags aren't immutable! They can be deleted\nand new ones can be inserted at any time, with any names desired.
\nAnd more important: consider the most natural sort of question that\nsomeone might want answered by the release history:
\n\nWhen did we last deploy the
www.our-appapp to production?
The only way to answer this question was to get the full list of all tags,\nfilter it for prod.www.our-app, then sort by date and take the last item.
Or similarly: What if the most recent staging deployment had turned\nout to be bad, and we wanted to immediately revert to the version\nbefore that? Again, we (or the reversion process) would have had to\nfilter and sort a list of thousands of tags, then pick out the\nsecond-last one.
\nI proposed this alternative in 2019. It stores the deployment history\nof each app as a series of unnamed but related Git objects, using\nGit's own history mechanism, so it's simple, efficient, and meshes\nwell with Git's commands.
\nI don't think it was ever adopted, and it might turn out to have\nserious drawbacks I haven't foreseen. But it might be useful to\nsomeone in a similar situation.
\nCurrently, each time a new version of an app is deployed, we create a ref:
\nrefs/release/stg.www.our-app-2019-09-06T114410.954-0700-treesha-179dab6fd11bee5195f0f8e31cea3edfb86ae23b
As these refs proliferate, many operations in the Git repo become slower and slower.
\nWe can record this information in a way that is at least as useful and which doesn't clutter up the ref list.
\nAt present, the release procedure creates a ref something like this:
\nA new tag object is created, pointing to *H*, something like
git tag -a refs/release/stg.www.our-app-2019-09-.... H
Every new tag object has a name in the ref database. This adds a name for every release. The structure is something like this:
\nOne ref object per release, each pointing to its own head commit
\nMain point of this proposal: There is only one ref for each (tier, app) pair, no matter how many releases have been done.
\nIt will look like this:
\n![\"\"](\"https://pic.blog.plover.com/prog/git/history-proposal/alternative-proposal.svg\")
A chain of metainfo commits, each with the previous\nmetainfo commit as first parent and the actual release commit as\nsecond parent
The one ref (which might or might not be an actual ref object) is\nnamed stg.www.our-app. That points not to the\nactual release commit, but to\na commit that contains\nmetainformation, such as the treesha. The\nmetainformation could be stored in any of several ways. (See below.)\nThe exact time of the release doesn't need to be stored explicitly\nbecause it is exactly the time at which the metainfo\ncommit was created.
The metainfo commit has two\nparent commits. One is the actual release\ncommit, the one that, in the current system, is the\ntarget of the named ref object. The other parent is the\nmetainfo commit from the\nprevious release.
\nThe metainfo commits are chained\ntogether exactly the same way Git normally chains commit history, so\nthe usual Git commands will work on the release history in reasonable\nand useful ways. In particular, the command git log\nstg.www.our-app will produce a listing of\nthe release history.
Creating this sort of two-parent commit is easy. Suppose the ref\nstg.www.our-app points to the current metainfo\ncommit. And suppose we want to release\nnew commit *H*. We do:
(Write some useful commit message into /tmp/message.$$.)
git commit-tree -p refs/release/stg.www.our-app -p H \\\n H^{tree} < /tmp/message.$$This creates a new metainfo commit\nwith two parents, given by the arguments of the -p options. The\nfirst parent is the metainfo commit\nfor the current release and the second is new commit\n*H*.
The git commit-tree command creates the new\nmetainfo commit and writes its SHA on standard\noutput. Say this SHA is *N*.
git push origin N:refs/release/stg.www.our-appThe release ref now points to the metainfo commit for the new release.
\nThis will be easier to use than what we have now, because Git's\ncommand set was designed to manage a list of commits linked in this\nway; this is exactly how Git normally expects to represent\nhistory. This natural fit with Git's command set shows up in many\nplaces.
\nYou can fetch a complete release history for any project with a single fetch:
\ngit fetch origin refs/release/stg.www.our-app:HISTORYAfter this, git log HISTORY will list the entire release history in\nreverse chronological order. All the usual git-log options make sense\nto filter or search this list. For example, if we want to find the\nlast release before a certain date, we can use:
git log -1 --until=date HISTORYIf we want a forward-chronological list of release commit SHAs with\ntheir release dates, we could use something like:
\n git log --reverse --format="%P %cd" HISTORYAt all times:
\nHISTORY^2.HISTORY~n.HISTORY~n^2.Notice: if M is a metainfo commit that describes a release, M^\nis the metainfo commit for the release before that, and M^{2} is\nthe exact commit that was actually released. We don't have to record\nthe tree IDs in the ref names; the tree that was released in release\nM is M^{2}^{tree}. These notations will work as expected in all\ngit commands; see git-rev-parse.
Because Git naturally understands this way of organizing history, it\nwill be easy to locate a particular previous release for\nrollback. When we roll back, we can roll back to the exact same commit\nas before: the new metainfo commit\nand the old one can share the exact same release\ncommit object.
\nWe can see the differences between the last two releases, without\nknowing their exact dates, with something like:
\n git diff HISTORY~1^2 HISTORY^2The metainfo commit is a full\ncommit with a full set of Git commit metadata, such as a creation\ndate. It has a commit message where we can record any unstructured\ntext that we want. We could also require the commit message to be in\nTOML format, or we could use the tooling provided by Git to store\nsemi-structured information in the commit message; see\ngit-interpret-trailers.
If the built-in commit structure is not big enough for everything we\nwant to record, the metainfo commit\nalso has an associated tree where we could record any amount of\nstructured information. In the example above I had its tree be\nidentical to the release commit's tree,\nwhich might be convenient. (That way, if you wanted to see the files\nthat were released, you could look in either the release\ncommit or the metainfo\ncommit; this would make some of the commands\nsimpler.) But the tree doesn't have to be the same. We could also add\nmetainformation files to this tree, or we could commit an entirely\nunrelated tree with nothing in it other than structured\nmetainformation.
\nPloni Almoni asks about the performance issues of tracing out a\nlinked list of commits. But this is exactly the same mechanism that\nGit normally uses to record history. When we do git log, Git is\ntracing a linked list of commits in exactly the way suggested here. If\nany operation in Git is going to be fast, it will be this one.
Git currently takes a few seconds to trace the 275,000-commit history\nof our monorepo. Tracing the similarly-structured history of a few\nhundred release commits is not going to be an issue.
\nI wrote this originally, but my only copy was in the\nform of a PDF. I had Claude convert the PDF to Markdown, which is the\nnative format accepted by my blog, and the two diagrams to SVG. I\nthen substantially edited the text and the diagrams. Along the way\nClaude pointed out three minor errors, which I\ncorrected. I also took Claude's advice on how to adjust the colors from the\noriginal to make them more legible to color-blind readers.
\nEvery\nword was written by my brain using my fingers.
","headings":[{"level":1,"text":"Maintaining an audit and reversion trail without thousands of refs","id":"maintaining-an-audit-and-reversion-trail-without-thousands-of-re"},{"level":2,"text":"Current practice","id":"current-practice"},{"level":2,"text":"Alternative proposal","id":"alternative-proposal"},{"level":2,"text":"Technical details","id":"technical-details"},{"level":2,"text":"Usage","id":"usage"},{"level":2,"text":"Performance?","id":"performance"}]}}