Question or problem about Python programming:
I am trying to install a Python package from a private GitHub repository. For a public repository, I can issue the following command which works fine:
pip install git+git://github.com/django/django.git
However, if I try this for a private repository:
pip install git+git://github.com/echweb/echweb-utils.git
I get the following output:
Downloading/unpacking git+git://github.com/echweb/echweb-utils.git Cloning Git repository git://github.com/echweb/echweb-utils.git to /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build Complete output from command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build: fatal: The remote end hung up unexpectedly Cloning into /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build... ---------------------------------------- Command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build failed with error code 128
I guess this is because I am trying to access a private repository without providing any authentication. I therefore tried to use Git + ssh hoping that pip would use my SSH public key to authenticate:
pip install git+ssh://github.com/echweb/echweb-utils.git
This gives the following output:
Downloading/unpacking git+ssh://github.com/echweb/echweb-utils.git Cloning Git repository ssh://github.com/echweb/echweb-utils.git to /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build Complete output from command /usr/local/bin/git clone ssh://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build: Cloning into /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build... Permission denied (publickey). fatal: The remote end hung up unexpectedly ---------------------------------------- Command /usr/local/bin/git clone ssh://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build failed with error code 128
Is what I am trying to achieve even possible? If so, how can I do it?
How to solve the problem:
Solution 1:
You can use the git+ssh URI scheme, but you must set a username. Notice the [email protected] part in the URI:
pip install git+ssh://[email protected]/echweb/echweb-utils.git
Also read about deploy keys.
PS: In my installation, the “git+ssh” URI scheme works only with “editable” requirements:
pip install -e URI#egg=EggName
Remember: Change the : character that git remote -v prints to a / character before using the remote’s address in the pip command:
$ git remote -v origin [email protected]:echweb/echweb-utils.git (fetch) # ^ change this to a '/' character
If you forget, you will get this error:
ssh: Could not resolve hostname github.com:echweb:
nodename nor servname provided, or not known
Solution 2:
As an additional technique, if you have the private repository cloned locally, you can do:
pip install git+file://c:/repo/directory
More modernly, you can just do this (and the -e will mean you don’t have to commit changes before they’re reflected):
pip install -e C:\repo\directory
Solution 3:
You can do it directly with the HTTPS URL like this:
pip install git+https://github.com/username/repo.git
This also works just appending that line in the requirements.txt in a Django project, for instance.
Solution 4:
It also works with Bitbucket:
pip install git+ssh://[email protected]/username/projectname.git
Pip will use your SSH keys in this case.
Solution 5:
I found it much easier to use tokens than SSH keys. I couldn’t find much good documentation on this, so I came across this solution mainly through trial and error. Further, installing from pip and setuptools have some subtle differences; but this way should work for both.
GitHub don’t (currently, as of August 2016) offer an easy way to get the zip / tarball of private repositories. So you need to point setuptools to tell setuptools that you’re pointing to a Git repository:
from setuptools import setup
import os
# Get the deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']
setup(
# ...
install_requires='package',
dependency_links = [
'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
.format(github_token=github_token, package=package, version=master)
]
A couple of notes here:
- For private repositories, you need to authenticate with GitHub; the simplest way I found is to create an OAuth token, drop that into your environment, and then include it with the URL
- You need to include some version number (here is
0) at the end of the link, even if there’s isn’t any package on PyPI. This has to be a actual number, not a word. - You need to preface with
git+to tell setuptools it’s to clone the repository, rather than pointing at a zip / tarball versioncan be a branch, a tag, or a commit hash- You need to supply
--process-dependency-linksif installing from pip
