Explore Posterous

How to use Subversion & Dropbox to create an effective work management/backup system

WARNING: The following article is written from a very high level point of view with assumptions made in areas like product knowledge and computer literacy.

After working on many coding projects I had come to realise there is one tool that is indispensable for any coder... what I'm talking about is version control. You come to realise that after making so many changes to your code you need the ability to track these changes and in case of emergencies revert to previous working versions of a particular file. I used this same concept for work and implemented my own version control.

Subversion

I installed a local copy of Subversion on my work laptop and with the help of TortoiseSVN I started my own local SVN repository. Because I'm super nice I'll even include lifehacker's instructions on how to setup your Subversion.

Setting up your Subversion server

1. Download and install Subversion 1.3.2 (the svn-1.3.2-setup.exe download). Again, just accept all of the defaults in the setup, making sure that "Install and configure Subversion modules..." is checked.
2. Download and install TortoiseSVN (TortoiseSVN-1.3.5.6804-svn-1.3.2.msi), an attractive Subversion client for Windows that makes dealing with Subversion repositories a breeze via right-click.

Create your first repository

The last thing we're going to do this week is create our repository. Go to your C:\ drive, right-click the svn folder and select TortoiseSVN -> Create repository here... command. If all goes well, TortoiseSVN should ask you what type of repository you'd like to create. Select Native filesystem (FSFS) and click OK. You will get a message telling you that "The Repository was successfully created." Congrats!

Finally, let's import the directory of files that you want to keep under version control into your newly-minted repository. Right-click on your to-be-controlled folder and select TortoiseSVN -> Import.... You need to give TortoiseSVN the directory of your repository, so browse for C:\svn. Click OK and all of the files under your chosen directory will be imported into your Subversion repository at C:\svn.

... you can verify that everything worked by right-clicking on a new folder and checking out your files from the repository (right-click -> SVN Checkout...). Just direct TortoiseSVN to the directory of your Subversion repository.

Loading mentions Retweet

Comment (0) »

Colorising svn diff

google: "config svn diff colorize"

in ~/.subversion/config set:
diff-cmd = colordiff

Install colordiff, unless already installed. In my case it was:
sudo aptitude install colordiff

Result is below:

Loading mentions Retweet

Comment (0) »

How To Manage Your Configuration Files With Version Control

On Unix-based and Unix-inspired operating systems such as Linux, one often has hidden configuration files in his home directory. In most unices, your home directory is /home/username; on Apple Macintosh, it is /Users/username. These files are created by the operating system, applications, programs, and most often, manually. Dotfiles.org is a popular website for finding user-uploaded configuration files. Creating your own configuration files is tedious, error prone, and requires many hours or even days of reading documentation or man pages. It is best to embrace and extend by copying and pasting from Dotfiles.org.

Even though this article is primarily targeted at those which are comfortable with the command line since these configuration files are mostly for command line programs, it also benefits those that are not. For example, on Mac, most applications store their preference files in /Users/username/Library/Preferences. One could put said directory under version control. Should an option be changed in an Application for the worst, and the user does not remember how to change it back, all he must do is close the application, revert the directory to the base state and relaunch the application. 

Since creating these files is a labourious process, it is best to mange and store configuration files in a repository off site. You will be able to track changes and revert if a new configuration change breaks programs. Any version control system may be used. The most popular open-source systems are Subversion, Git, Mercurial, and Bazaar.

All have advantages and disadvantages. If you are on Ubuntu, you will most likely use Bazaar. It is developed by Canonical, the makers of Ubuntu, and you will use it with Launchpad. Support should be easy to find in the Ubuntu Forums. Mercurial is independently developed and the main competitor to Git. It is faster and more popular than Bazaar. The main hosting service is BitBucket. It has a decent interface. Even though it is more popular, there is less support for it than for Bazaar. It does not have a big community such as Ubuntu. Git was created for the Linux kernel by Linux Tovarlords. As such, it is very popular, not to mention, abnormally fast. Support can be easily found.  Though, it does not have the best CLI interface, and there is no decent GUI for it, you will love GitHub. Probably 70% of the Git repositories are hosted there.

For this tutorial, we will use Subversion, which has been in the wild for almost a decade. It is also pre-installed on Mac OS X 10.5 Leopard. There are many  graphical user interface clients for it. On Mac, there are Versions, which has recently received the Apple Design Award 2009, and Cornerstone, on Windows TortoiseSVN, on Linux eSVN, RapidSVN (cross platform), KDESvn, and many others. There are also many repository hosting companies with pretty and usable interfaces. Be aware that this article is not about version control. It discusses the bare minimum required for this tutorial to work. You may read a more in depth, but by no means comprehensive, tutorial on Subversion in an article written by Christ Nagele, one of the people behind Beanstalk, Subversion for Designers.

First, we will create a directory to store the configuration files. Traditionally, it is called etc, and Linux users may want to create /home/username/etc. I chose to call it Preferences since it better matches the Mac naming conventions: /Users/sorin/Preferences.

Next we have to move the configuration files from  /Users/sorin to /Users/Sorin/Preferences. We can use the command line or we can use Finder. Using Finder also requires the temporary use of the command line because Apple do not believe that Mac users should be able to easily access hidden files. Hidden files in Unix are called dot files because they begin with a dot: .bashrc. This is the configuration file for Bash, the shell that runs inside the Terminal. 

We must enable hidden files in Finder. Launch Applications > Utilities > Terminal and paste:

defaults write com.apple.finder AppleShowAllFiles TRUE; killall Finder

You can reverse the change later by pasting:

defaults write com.apple.finder AppleShowAllFiles FALSE; killall Finder

Once you have enabled hidden files, select the configuration files and folders you wish to put under version control and drag them to the Preferences folder. Then remove the dot from in front of the file to make them unhidden. From now own, you will not have to enable hidden files to easily find and edit them. They will also be displayed in Spotlight searches.

It is much easier to move and rename the files from the command line. Most likely you will not want to move all the hidden files and folders. There can be tens of them and not all are important. Select the ones you wan to move by appending file names before the semi-colon in the first line. Then paste this in Terminal.

for entry in .bashrc .vimrc .profile
do

new_name=`echo $entry | sed -e 's/\.//g'`
mv $entry Preferences/$new_name

done

Regardless of which methods you chose, the result should be similar to this screen-shot.

Next, we must create symbolic links, or shortcuts, in the home directory to point to the new location of the configuration files. I have written a script that automatically creates the symbolic links. It is safe to use. It backs up files that already exist in a directory called .deploy_backup in your home directory instead of overwriting. Just place deploy.rb in Preferences and execute it in Terminal by typing ./deploy.rb. Do not forget to change its permissions to an executable file first with chmod u+x deploy.rb.

You will see symbolic links in your home folder.

We have not taken such effort to just be able to easily edit and search configuration files with Spotlight; we want to store them offsite on someone else's server to make sure we never lose them. Get yourself a free Subversion repository hosting account at Beanstalk. It is a very easy to use application.

Once, you have created an account, you can access it online at http://username.beanstalkapp.com. Log-in then go to the Repositories tab and create a repository. I called mine personal. You may name it anything you wish. The repository is accessible via Subversion at http://username.svn.beanstalkapp.com/repositoryname. You will be prompted for your username and password. We have to import our Preferences folder into the repository. We will use the svn import command in Terminal to import all files into the repository.  Once the import process is done, unfortunately, due to the way Subversion works, our configuration files have been imported, but are not yet tracked. We have to check them out first. Since Subversion does not overwrite existing files, we have to check them out in a temporary location.  

cd ~/Preferences;

svn import http://username.svn.beanstalkapp.com/repositoryname -m "Initial import."

cd ..

svn checkout http://username.svn.beanstalkapp.com/repositoryname/ Preferences2
rm -rf Preferences
mv Preferences2 Preferences

Alternatively, in Versions, we can go to File > Import. Select the Preferences folder and click Import. Then click the Checkout icon to checkout our files to Preferences2. Do not forget to edit the bookmark created to point to Preferences and not Preferences2.

Versions makes it easy to import and checkout files.

Once your repository is set, you are free to modify the configuration files as you see fit and never have to worry about making detrimental changes. You can always revert to the older version. In the screen-shot bellow, I have modified vimrc to change the Terminal title to 'Shell' when it exists instead of 'Thanks for flying Vim'. This is reflected by the letter 'M' at the begging of the line. Versions reflects the change by displaying a pencil icon next to the file.

Loading mentions Retweet

Comment (0) »

git-svn dcommit lost my $#&(# commits

git-svn is one of the coolest pieces of software this side of Alpha Centauri. It is also not perfect, and it’s handling of interrupted Subversion commits can do seemingly frightening things to your last few commits.

The scenario

When you run git-svn dcommit, it looks for the most recent commit in your repo which was uploaded to the Subversion server. It then pops all commits newer than that commit from the branch and commits them in sequence.

If this commit process (which can be pretty long for large sets of changes) is interrupted, you’ll be left with your branch pointing somewhere in the middle of your long sequence of commits, with no sign of the commits after that point.

Now, you panic. But stop. Take a deep breath and remember:

very few git commands remove data from your repo¹.

Finding the missing commits

Your data is still in there. The first thing you should do, is to reset your branch to where it was before everything went pear shaped. git-reflog keeps a log of operations performed on a repo. In one of my repos, running git reflog gives the following:

4bea3cc... HEAD@{0}: refs/remotes/trunk: updating HEAD
ee57a9c... HEAD@{1}: rebase: Modified xliff2odf to embed the XLIFF...
1e7dbbb... HEAD@{2}: checkout: moving from master to 1e7dbbb7f0857...
886607f... HEAD@{3}: commit: Modified xliff2odf to embed the XLIFF...
800e8da... HEAD@{4}: checkout: moving from toolkit_1_2 to master
9a81bbc... HEAD@{5}: checkout: moving from master to toolkit_1_2
800e8da... HEAD@{6}: HEAD: updating HEAD
800e8da... HEAD@{7}: checkout: moving from toolkit_1_2 to master
9a81bbc... HEAD@{8}: refs/remotes/branches/toolkit_1_2: updating H...
...

The left column contains the SHA1 values of commits. Unless you’ve been a busy bee in the meantime, the SHA1 of your branch, as it looked before should be in there (and if it’s not there, read ahead for another way to get at your data). gitk is a fine tool for a quick visual history inspection; to look at the first commit in the above list, simply execute:

gitk 4bea3cc

Recovering the commits and finishing the

Once you’ve found the SHA1 value that points to all your commits, it’s time to reset your current branch to that position. First, check out the branch on which you were working:

git checkout <my_branch_name>

Now reset the branch to its old state:

git reset --hard <the_sha1_that_I_want>

Finishing off the Subversion commit

Since some of your commits have already made it into the Subversion repository, git-svn must first figure out which of the your newest commits are already there. Fortunately, the following does the trick:

git svn rebase

Now, to finish off what you started, simply execute:

git svn dcommit

Action for commits that are missing in action

If git reflog is of no help, don’t despair. git fsck will print a list of all unreachable objects in your repo. In order to filter out everything except commit objects, simply run:

git fsck | grep commit

Now you can view each of the commits with gitk until you find what you want.

¹ The only command that I use (other than rm -rf *) that removes data from a repo, is git gc --prune.

Loading mentions Retweet

Comment (0) »

Liked "Git for Designers tutorial"

via hoth.entp.com

Loading mentions Retweet

Comment (0) »

GitHub "Permission Denied (publickey)" Error Workaround

This frustrated me for 45 minutes tonight, and hopefully this post can save someone else from similar pain.

[MyRepo (master)]$ git push origin master
Permission denied (publickey).
fatal: The remote end hung up unexpectedly

I followed all the instructions at http://github.com/guides/providing-your-ssh-key and http://github.com/guides/addressing-authentication-problems-with-ssh and verified my connection was working via the ssh git@github.com command, but kept receiving the Permission Denied (publickey) error when trying to push to github. Eventually I stumbled onto a workaround. In your local repository, remove and re-add the remote link. These commands worked for me, but YMMV:

git remote rm origin
git remote add origin <remote url>

Replace the remote url with your github clone url (looks like "git@github.com:<username>/<reponame>.git") and try your push again. If it doesn't work for you, please post a comment below and let me know why and what you did instead.

Loading mentions Retweet

Comment (1) »

Importing bzr branches into git

Here’s the brief tutorial I wish I found on Google.

The Basic Idea

Send output from the bzr fast-export tool to git fast-import, thereby importing the desired bazaar branch into git.

The Process

1. Install the bzr-fastimport plugin plugin. You need this because paradoxically, the fast-export plugin is inside the fastimport plugin.

   bzr branch lp:bzr-fastimport fastimport
   mv fastimport ~/.bazaar/plugins/

2. Create the git branch (the destination branch).

   mkdir projectname.git
   cd projectname.git
   git init

3. Run the export/import.

   bzr fast-export /path/to/bzr/project | git fast-import

4. (Optional) Get a working tree.
   At this point you’re pretty much done. If you git log you will see all of the revision history imported from bzr in your new git branch. However, you won’t have a working tree. Run this command to get your files:

   git checkout

That’s it. There’s probably an even easier way to get this done. Feel free to leave a message in the comments if you can think of one!

Loading mentions Retweet

Comment (1) »

svn:externals - using libraries with subversion

SVN is one of my absolute favorite tools. It’s hands-down easier to use than TFS or (gasp) SourceSafe, which have been the other common version control systems in my industry. If you aren’t using it, you’re missing out (quiet down all you GIT-heads in the back… keep using it for a few more years and we’ll accept that it isn’t just a fad).

In any project of significant complexity you’re going to use libraries of some kind. These libraries frequently end up being copied into the project tree under a lib subdirectory and compiled or included in a project deployment from there. Most of my projects rely on open source libraries, and as we make changes to a library those changes have to be propagated from project to project using patch files or simple copy/paste. This time-consuming process opens up the possibility of human error and versioning confusion, but there is a better way.

SVN Properties
First, some basics: using Subversion, you can specify metadata properties on any directory or file. These can be used for your own custom purposes, or more often to control how the SVN client treats certain files. One commonly used property is svn:ignore, which (unsurprisingly) tells SVN to ignore particular files when looking for changes in a working copy. My new favorite property is svn:external, which when set on a directory, allows you to define subdirectories and point them at any other SVN path, even to a particular revision. You can set an SVN property using the official command-line client, but I’ll leave the syntax for that to the official documentation and show how to do it in my favorite SVN client, TortoiseSVN.

1. Check out a working copy of your project.
2. Right-click the parent folder of where you want your library subdirectories placed. On the TortoiseSVN context menu, chose Properties.
3. In the properties dialog, click New, then select a property name from the drop-down menu.

svn:externals
In the new properties dialog, select svn:externals, and then we’ll enter a value in the multi-line edit box provided. First, if you’re using an SVN client before version 1.5, upgrade and have everyone on your team upgrade. If you’re stuck using pre-1.5 clients the syntax is different so see the official documentation for the details.

In modern clients the syntax looks like this:
External_Path@RevisionNum SubDir_Name

• External Path: This can be a fully qualified SVN url like “svn://svn.example.com/skin-maker” or you can use symbols to specify a relative URL.
  • ../ Relative to the URL of the directory on which the svn:externals property is set
  • ^/ Relative to the root of the repository in which the svn:externals property is versioned
  • // Relative to the scheme of the URL of the directory on which the svn:externals property is set
  • / Relative to the root URL of the server on which the svn:externals property is versioned
  I only use one SVN server for my project and libraries repositories, so I generally have external paths in the form of /libraries/libraryName without the protocol and server.
• Revision Number: Regular integer revision number of the libraries repository. Here’s  a great note from the SVN Book:
  You should seriously consider using explicit revision numbers in all of your externals definitions. Doing so means that you get to decide when to pull down a different snapshot of external information, and exactly which snapshot to pull. Besides avoiding the surprise of getting changes to third-party repositories that you might not have any control over, using explicit revision numbers also means that as you backdate your working copy to a previous revision, your externals definitions will also revert to the way they looked in that previous revision, which in turn means that the external working copies will be updated to match the way they looked back when your repository was at that previous revision. For software projects, this could be the difference between a successful and a failed build of an older snapshot of your complex codebase.
• Subdirectory Name: usually “lib” - the name of the directory underneath the directory with the svn:externals property set, which will be mapped to an external repository URL.

My projects follow a traditional trunk-branches-tags folder structure, with a /source directory under trunk. I set the svn:externals property on that source directory and end up with repository_root/trunk/source/lib pointing at an external library directory.

After setting the property, update and commit, and you should see the external library source downloaded during the update process. Commit will make sure all your team members get the property set in their working copies and their next update will likewise download the external library.

That’s all I have for now. Fire away in the comments if you see a mistake or something isn’t clear.

Loading mentions Retweet

Comment (0) »

Nonexistent Database Application

In software development a common problem is versioning database changes. Keeping code under version control is a necessity for every programmer, but when faced with versioning a database this becomes quite complex. The problem presents itself with a separation between schema and data, and changes to both schema and data. Schema changes can effect the entire software group and thus bring down an entire development process if not correctly applied. The same can be said for data changes. In this functional spec I want to explore the current state of ways to version the database, and also look at a non-existent piece of software to see if it is worth writing. 

Rolling Your Own
This is perhaps by far the most popular method of database versioning. I have no real statistics to back up this claim, so for all intents and purposes you should just ignore it. I have asked people what they use for database version control and next to nothing this is the top contender. I can't really speculate as to the official process that people use, but there are some general guidelines experts say you should follow. 

1. Keep a baseline schema with persistent data. (It is important to remember that there are two kinds of basic data in a database. Those that should be included in the schema and those that should not. For the sake of this discussion let's call the data that should be included with the schema "static data" and those that should not be included in the schema "dynamic data".) For a more professional discussion of this you should check out K. Scott Allen's article on Versioning Databases.

2. There has to be a way to keep schema and static data up to date. Often as development on a project is made changes are made to the database schema and the static data. Those changes should be versioned. The question then becomes how do we keep track of these changes? There are different methods, but they seem to be along the lines of a migration script that maybe stores some SQL that modifies either structure or data that will eventually get run on all databases. 

Using Rails Migrations
From the little work that I have done in Rails I was very impressed with database migrations. Essentially Rails provides some scripts that generate models and migrations for you, as well as scripts that apply those migrations. The migration scripts can then be versioned just like you would any file. This coupled with a deployment tool like Capistrano makes database migrations across multiple servers pretty easy. Now it is important to note that this has a couple drawbacks. 

1. There are a lot of people that do not work in Rails. 

2. You aren't really versioning the database, but rather the Ruby scripts. 

Of course for complaint number 2 you could use that along with a Roll Your Own approach and have both at your fingertips.
There really isn't a solution that I can see for complain 1. Maybe you could figure out a way to use only the Rails migration methods for your project. I do not know enough about Ruby/Rails to know if this is possible or 
how much work it would involve. I am going to guess that it would be quite a bit of work/hassle to get going even if it were possible. It just doesn't seem really worth it to have Rails models generated that you have to keep up to date as 
well when you are not even using Rails. 

Using Django
Django can automagically create your database schema from models. This is very nice as all that you are required to do is run django-admin.py syncdb and it will go through and create the db for you. This is all fine and nice, but the command syncdb 
doesn't do exactly what it says. It doesn't actually sync the db, but rather create something if it doesn't exist. You cannot make a model run syncdb, go and change that model around and then run syncdb again and it change everything that you want. 
It is good for initial development, but it does not generate migration scripts as Rails does. Even if it did it would still suffer from the same problem as Rails which is simply, not everyone uses Django.

Some Other Framework
I know there are solutions for .NET and probably other frameworks that I know little to nothing about. I am not even going to comment on them precisely because I know little or nothing about them.

Proposed Solution
An application that would introspect your database create a baseline schema for you. The user would of course have to provide it with what tables/columns contain static data, but it would take care of the rest. This application would also create migration scripts 
that could be versioned and would allow for easy migrations. This application would have the following requirements:

1. It would be language agnostic. It doesn't care what language your application is written in. It probably wouldn't need to look at your models(maybe a later version could for some reason), but would rather gain all knowledge about your database from your database.

2. It would be as database agnostic as possible. I would not expect that the initial versions would fit this requirement, but eventually it would not care if you were running MySQL, Postgres SQL, MS-SQL, or whatever. There is of course a limitation to this. For instance, it wouldn't be practical to say that it supports every single database in the entire universe. Database syntax can be widely different, so it would try to stick to some sort of ANSI-SQL standard as much as possible. I think the initial target should be MySQL, Postgres, and MS-SQL, as those seem to be the most popular.

3. It would not be an entire deployment framework. There are already a couple of good ones out there such as Capistrano and Fabric. This is simply for maintaining version control over databases.

I have no idea exactly how it would work. I expect that if development starts on it the approach would change very quickly anyways. How I imagine that it would work though would be like this.

1. You outline what data you want to be static. This could either be from interactive questions asked at the command line, or read in from a config file.

Loading mentions Retweet

Comment (0) »