Majic

So, as I am hacking away some final bits and pieces for my Gimmecert project, I have decided to take out as much tedium as possible from my release process. Given my previous experiences, I figured I might as well as try to write a convenient script that will take care of preparing the Git repository, including some release commits, maintenance branching, etc.

One of the parts of this process is publishing the new releases to PyPI.

In order not to mess-up my main project on PyPI, I decided to use the test instance for this purpose. After some scripting, I got the publishing process to work as desired, but then I ran into a small issue - PyPI does not allow re-upload of a same package version (unless package hashes are identical). While this might be actually a very good idea for the main PyPI archive, in order to ensure better integrity of packages, the problem is that the same behaviour is applied against the Test PyPI.

This, in turn, means that you will find it rather hard/annoying to develop release scripts like the one I am working on, or you won't be able to actually test the upload process prior to pushing it to main PyPI. Why do I say this? Well, if you think about it, you try to push to the Test PyPI, and figure out you've messed something up. After that, once you make the necessary fixes, you won't be able to push that exact same version there again. You will need to make a new release, with updated version. Which kind of beats the purpose of using the Test PyPI as a pre-check for the main upload. You keep bumping the version all the time in case of mistakes.

This, of course, is something that has been known to a number of people already, and there is a somewhat lengthy discussion on Github about it.

Now... If you have the patience to read through that thread of comments, you will find out about a nice project called devpi. And this is what came to my rescue - at least for release script testing.

In essence, the devpi project allows you to run your own local PyPI-compatible server. Now, it does come with a bunch of nice features, and it can be used as caching server too, but that's not the main focus for this post. My main idea is to use it as upload test-bed.

If you are looking for a nice getting started guide, I would whole-heartedly recommend Stefan Scherfke's post on this topic. I'll just outline a very quick-and-dirty set of instructions to test out your upload.

Start off by creating a separate virtual environment and installing devpi (you are using Python 3 nowadays, right? :):

mkvirtualenv -p /usr/bin/python3 devpi
workon devpi
pip install devpi

Create a temporary directory where the devpi files will be stored at (you could also skip this and let it use the default ~/.devpi location):

mkdir /tmp/devpi

Now start and initialise the server (the --no-root-pypi option prevents it from fetching package information from https://pypi.org/ which can save some processing/bandwidth):

devpi-server --serverdir /tmp/devpi --start --init --no-root-pypi

The server will be started in the background. The server will by default be available under http://localhost:3141/ (it binds to loopback interface by default). When you want to stop it, simply run:

devpi-server --serverdir /tmp/devpi --stop

If you need to start it again, simply skip the --init option:

devpi-server --serverdir /tmp/devpi --start

Configure the devpi client (used to configure the server) to use the local instance:

devpi use http://localhost:3141/

Create a user for testing the uploads:

devpi user -c myuser

Login:

devpi login myuser

Create an index (repository) for testing uploads (the volatile option will allow you to overwrite existing package versions, which is useful for this type of testing):

devpi index -c myuser/testupload volatile=True

At this point you should have a fully working local PyPI server. To upload things to it, simply go to the root of your project (possibly switching to correct Python virtual environment):

workon myproject
python setup.py sdist bdist_wheel
twine register --username myuser --repository myuser/testupload --repository-url http://localhost:3141/myuser/testupload/ dist/*.tar.gz
twine upload --username myuser --repository-url http://localhost:3141/myuser/testupload/ dist/*

And, as a final touch, just open up your local PyPI repository URL and see if everything looks fine.

Once you are done with testing, you can simply stop the devpi server, and remove the directory /tmp/devpi.

Happy hacking.