Clone all repositories from a user (bitbucket)

Now that I have computer, the next step is to install everything I use on daily basis, and set-up my projects space. This involves cloning all my repositories. I could do this manually one by one, but what is the fun in that?

Solution 1

My rule is that “if there is a way to make some task more efficient programatically, some one else thought of it before, so google it before starting scripting”. So I did, and of course someone else came up with a solution:

#!/bin/bash
#Script to get all repositories under a user from bitbucket
#Usage: getAllRepos.sh [username]
## credit to Harold Soh

curl -u ${1}  https://api.bitbucket.org/1.0/users/${1} > repoinfo
for repo_name in `grep \"name\" repoinfo | cut -f4 -d\"`
do
    hg clone ssh://hg@bitbucket.org/${1}/$repo_name
done

This however did not work perfectly for me. When downloading the rep info, the json config comes poorly formatted as a single line. This meant that the line grep \"name\" repoinfo | cut -f4 -d\" was return scm, which obviously is not what one of my repositories.

Fix

So I hacked a liitle and came up with a not-very-elegant-but-working-solution to parse the json:

#!/bin/bash
#Script to get all repositories under a user from bitbucket
#Usage: getAllRepos.sh [username]
#source: http://haroldsoh.com/2011/10/07/clone-all-repos-from-a-bitbucket-source/

curl -u ${1} https://api.bitbucket.org/1.0/users/${1} > repoinfo
# curl -u adomingues https://api.bitbucket.org/1.0/users/adomingues
# cat repoinfo

for repo_name in `cat repoinfo | sed -r 's/("name": )/\n\1/g' | sed -r 's/"name": "(.*)"/\1/' | sed -e 's/{//' | cut -f1 -d\" | tr '\n' ' '`
do
    echo "Cloning " $repo_name
    hg clone https://${1}@bitbucket.org/${1}/$repo_name
    echo "---"
done

This basically the same script as that of Harold, but with a more complex parsing:
sed -r 's/("name": )/\n\1/g' makes sure that the repo name is not at the start of each line;
sed -r 's/"name": "(.*)"/\1/', removes the string “name” at the beginning of the line;
sed -e 's/{//' removes a funny curly bracktet ar the start of the file;
cut -f1 -d\", separates and keeps the actual repository name;
– `tr ‘\n’ ‘ ‘“ just removes the new line character creating a list of repository names to be looped.

Et voilá! There is probably some less convoluted away of going about it, probably involving regex, but in someone else’s wise words: “if you need regex to solve a problem, you now have 2 problems”.

The only minor inconvenience is that I need to input my password for each repository. There might be a solution for this, but I have not found it yet.

Advertisements
This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

8 Responses to Clone all repositories from a user (bitbucket)

  1. crunchex says:

    This would probably help the “password every time” problem: https://help.github.com/articles/caching-your-github-password-in-git/

  2. Emad Elsaid says:

    I modified it a little bit to clone git repos and use ssh instead of http

  3. Vit says:

    Use “slug” instead of “name” – in case there is a space in the name, slug will give you the real name repo name for URL.

  4. Mike says:

    Doesn’t work for me. After entering password I get this:

    sed: illegal option -- r
    usage: sed script [-Ealn] [-i extension] [file ...]
           sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
    
    • That is strange. The error indicates the issue is with sed, but the password only gets requested if sed worked in the first place (meaning inside the loop). Does mercurial asks for the password? Also, I have not used this for exactly 2 years, so it is possible that some changes to bitbucket or mercurial have rendered it obsolete.

  5. Michael says:

    does not work on Mac unfortunately. I am not shell guru so some tries did not manage to fix.
    Changes i did:
    use sed -E instead sed -r
    used ssh instead of Mercurial
    But for loop not entered because all regex’s give an empty result:

    #!/bin/bash
    #Script to get all repositories under a user from bitbucket
    #Usage: getAllRepos.sh [username]
    #source: http://haroldsoh.com/2011/10/07/clone-all-repos-from-a-bitbucket-source/

    curl -u ${1} https://api.bitbucket.org/1.0/users/${1} > repoinfo

    curl -u adomingues https://api.bitbucket.org/1.0/users/adomingues

    cat repoinfo

    for repo_name in cat repoinfo | sed -E 's/("name": )/\n\1/g' | sed -E 's/"name": "(.*)"/\1/' | sed -E 's/{//' | cut -f1 -d\" | tr '\n' ' '
    do
    echo “Cloning $repo_name”
    #hg clone https://${1}@bitbucket.org/${1}/$repo_name.git
    git clone git@bitbucket.org:${1}/$repo_name.git
    echo “—”
    done

    • Is the the file repoinfo empty? If not, then it should be an issue with sed in OSX. I did some googling and it seems that sed has a few differences between OSX and Linux and -E doesn’t always work as a direct replacement for -r. You can try to install gnu sed with “brew install gnu-sed”, and then replace sed with gsed. Hopefully that will work.
      Reference: https://github.com/pyrsmk/molt/issues/8

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s