Clone all repositories from a user (bitbucket)

Now that I have computer, the next step is to install everything I use on daily basis, and set-up my projects space. This involves cloning all my repositories. I could do this manually one by one, but what is the fun in that?

Solution 1

My rule is that “if there is a way to make some task more efficient programatically, some one else thought of it before, so google it before starting scripting”. So I did, and of course someone else came up with a solution:

#Script to get all repositories under a user from bitbucket
#Usage: [username]
## credit to Harold Soh

curl -u ${1}${1} > repoinfo
for repo_name in `grep \"name\" repoinfo | cut -f4 -d\"`
    hg clone ssh://${1}/$repo_name

This however did not work perfectly for me. When downloading the rep info, the json config comes poorly formatted as a single line. This meant that the line grep \"name\" repoinfo | cut -f4 -d\" was return scm, which obviously is not what one of my repositories.


So I hacked a liitle and came up with a not-very-elegant-but-working-solution to parse the json:

#Script to get all repositories under a user from bitbucket
#Usage: [username]

curl -u ${1}${1} > repoinfo
# curl -u adomingues
# cat repoinfo

for repo_name in `cat repoinfo | sed -r 's/("name": )/\n\1/g' | sed -r 's/"name": "(.*)"/\1/' | sed -e 's/{//' | cut -f1 -d\" | tr '\n' ' '`
    echo "Cloning " $repo_name
    hg clone https://${1}${1}/$repo_name
    echo "---"

This basically the same script as that of Harold, but with a more complex parsing:
sed -r 's/("name": )/\n\1/g' makes sure that the repo name is not at the start of each line;
sed -r 's/"name": "(.*)"/\1/', removes the string “name” at the beginning of the line;
sed -e 's/{//' removes a funny curly bracktet ar the start of the file;
cut -f1 -d\", separates and keeps the actual repository name;
– `tr ‘\n’ ‘ ‘“ just removes the new line character creating a list of repository names to be looped.

Et voilá! There is probably some less convoluted away of going about it, probably involving regex, but in someone else’s wise words: “if you need regex to solve a problem, you now have 2 problems”.

The only minor inconvenience is that I need to input my password for each repository. There might be a solution for this, but I have not found it yet.

This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

10 Responses to Clone all repositories from a user (bitbucket)

  1. crunchex says:

    This would probably help the “password every time” problem:

  2. Emad Elsaid says:

    I modified it a little bit to clone git repos and use ssh instead of http

  3. Vit says:

    Use “slug” instead of “name” – in case there is a space in the name, slug will give you the real name repo name for URL.

  4. Mike says:

    Doesn’t work for me. After entering password I get this:

    sed: illegal option -- r
    usage: sed script [-Ealn] [-i extension] [file ...]
           sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
    • That is strange. The error indicates the issue is with sed, but the password only gets requested if sed worked in the first place (meaning inside the loop). Does mercurial asks for the password? Also, I have not used this for exactly 2 years, so it is possible that some changes to bitbucket or mercurial have rendered it obsolete.

  5. Michael says:

    does not work on Mac unfortunately. I am not shell guru so some tries did not manage to fix.
    Changes i did:
    use sed -E instead sed -r
    used ssh instead of Mercurial
    But for loop not entered because all regex’s give an empty result:

    #Script to get all repositories under a user from bitbucket
    #Usage: [username]

    curl -u ${1}${1} > repoinfo

    curl -u adomingues

    cat repoinfo

    for repo_name in cat repoinfo | sed -E 's/("name": )/\n\1/g' | sed -E 's/"name": "(.*)"/\1/' | sed -E 's/{//' | cut -f1 -d\" | tr '\n' ' '
    echo “Cloning $repo_name”
    #hg clone https://${1}${1}/$repo_name.git
    git clone${1}/$repo_name.git
    echo “—”

    • Is the the file repoinfo empty? If not, then it should be an issue with sed in OSX. I did some googling and it seems that sed has a few differences between OSX and Linux and -E doesn’t always work as a direct replacement for -r. You can try to install gnu sed with “brew install gnu-sed”, and then replace sed with gsed. Hopefully that will work.

  6. Andy says:

    The list in repoinfo is empty for me. I am not sure what I am doing wrong. I looked at the url in the browser and there where no repos listed. Any suggestions?

    • Hi Andy, I get an empty file if the password is incorrect – no error message at all, only an empty file. Is the password correct?

      As for the url being empty, so is mine when I tested it in the browser, however using curl still returns a full list of repos. No idea, sorry.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s