Copr final report

Fedora Summer Coding is now over for me and I’m really glad of what I learned and coded this summer.

Our initial goal was to develop a TurboGears2 Web app and JSON API for Fedora Copr. When finished, Copr should provide everyone with a place to build Fedora packages and host custom repositories for everyone to enjoy. This is a project that should prove quite popular in the Fedora Community when it gets released and I’m glad to have played a role in its development.

At first I worked on the web app, modeling the database and the relationship between coprs and repos and packages and then developing the JSON API. When the midterm came, my mentor and I decided that I should also contribute to the other parts of Copr. The original schedule had a simple command-line client planned, but we went further than that. In the end all of the functionality of the JSON API also got implemented in a client library (based on and very similar to python-fedora) and in a command-line client. I also got to dive into python-fedora’s and repoze.who’s internals in order to get basic HTTP authentication working for TurboGears2.

My latest work has been on the func module. This is the buildsystem part of Copr. Func minions running this module will be commanded by headhunter (Copr’s scheduler) to build packages in mock and then move them into repositories. The module also creates, updates and deletes package repositories and will check the built packages for Fedora conformance (e.g. licensing) – this last part is not yet implemented. I got to play with virtual machines and func and mock and createrepo.

There is a more synthethic overview of all the different things that got implemented on the wiki.

Overall, I’m really glad of what I learned this summer. This project really got me involved in a lot of different levels of the architecture of a web service and a lot of different technologies. Some of the things I worked on looked really scary at first, but as I went nearer and read more code the mist slowly vanished.

My mentor, Toshio Kuratomi was great as always. This isn’t the first project I’ve had him as my mentor. He was always there to talk to and always had great answers to all of my questions. He had great patience in answering and explaining anything I asked about. Our discussions were mostly about the architecture of the app we were building, but he also gave me great tips on the inner workings of python-fedora or on deploying the web app. I felt I had a lot of liberty to decide the way things will get implemented. Regardless of whether we will ever work together again, Toshio will always be a great inspiration for me as a programmer and as a person.


FSC: moving to the buildsystem

I started working on the buildsystem part of copr this week. Right now, I’m still getting familiar with func. That’s what we’ll be using to communicate with the builder machines: get them running errands and get back status reports at any time. I spent a lot of time getting a virtual machine setup with libvirt; networking especially was a pain (mostly because of my pppoe connection I think).

One nice feature of func that I think we’ll be using a lot is the async mode. A mock build takes a lot of time, what with all the yumming and compiling. So starting a task via one of the user interfaces and then choosing whether or not to keep watching it and what to watch for will probably be an essential part of the buildsystem’s functionality.

In the meantime, we’re slowly getting resources for the deployment of Copr. Toshio got a running instance of the current state of the TG app on publictest1. It looks just like a quickstarted TG app, because it doesn’t have any WebUI. But it can CRUD coprs, handle dependencies between them, handle permissions and CRD packages. Most of the functions require a FAS account, but you don’t need one to see a list of all the coprs, or a list of packages in a copr.


the Copr client part II

I spent this week finishing up the copr client. It now supports all the functionality that the Copr TG API supports. It’s not much, but it’s a starting point.

I spent a lot of time trying to understand the way repoze.who works and the authentication plugins that we’re using for the python-fedora FAS authentication plugin. I finally understood it, I think… The Fedora client library didn’t support basic HTTP Authentication for TG2 apps so I had to figure out how to integrate that into our authentication plugin. It was quite fun all in all, repoze.who has a very interesting way of doing authentication and writing wsgi middleware is always exciting ;). This patch will hopefully go upstream to python-fedora now.

This next week I’ll probably start working on the buildsystem part of Copr. There are a lot of new things to learn there.


the Copr client

This last week I started working on the command line client to Copr. Luckily, the python-fedora already has a lot of code in place to make the task of writing clients for TurboGears apps a lot easier. Some of the apps in infrastructure are already using this library, which make for some good examples.

So I’m building a client library and a client um… command line client. The command line client is basically one big argparse application that calls the functions in the client library and sometimes does a bit of formatting on the output. The client library implements a fedora.client.BaseClient that mostly just calls json methods on the Copr server.

It’s all pretty simple. The hard part is deciding what the command line client’s interface will look like. In argparse parlance, which ones should be the positional arguments and which should be the optional arguments. So far I’ve been inclined to use something that looks like a VCS’s interface. Here’s what it looks like so far:

$ python client/ -h
usage: [-h] [-v] [-u USERNAME] [-p PASSWORD] [--url URL]
Command line tool for interacting with Fedora Copr
positional arguments:
    list                list all the available Coprs
    info                get information about a specific Copr
    create              create a new Copr
    edit                edit an existing copr
    delete              delete an existing copr
optional arguments:
  -h, --help            show this help message and exit
  -v, --version
  --url URL             provide an alternate url for the Copr service
  -u USERNAME, --username USERNAME
  -p PASSWORD, --password PASSWORD

Right now, all the copr functions are top-level. I wonder if I’ll have to create a deeper level of nesting when I start implementing package-related functions.

I’m also having a few problems with the BaseClient that I’ll probably have to solve this week. All of the other client libraries were written for TurboGears 1.x and it seems that authentication has changed in TurboGears 2. There’s also no support for HTTP PUT and DELETE which I would like to use since I used RestControllers in the API. I also had to write a patch for file upload support; that seems to work well so far.


Fedora Summer Coding midterm

This midterm scared me when I found out about it on Friday when I looked at the schedule I had set myself. However, I have done the work that I should have done by this point in the project. When I wrote the proposal I had assumed that the buildsystem would already be built before me starting coding on the TG app, but that is not the case. Therefore I could only code the user-facing JSON interface that interacts with the DB as it would if the buildsystem would provide it with packages and repos. Except that there are no packages and repos at this stage.

So for this midterm, we’ve got working Copr CRUD, dependency handling and release/repo editing on a Copr. I also coded the Package CRUD, which basically allows for uploads of SRPMs, stores the info in the db and also allows for information retrieval and deleting packages. Actually building packages and retrieving packagebuilds will have to wait for the buildsystem to be built.

After I finish polishing things a bit, I will probably start working on a basic client and then maybe move on to working on the buildsystem part of Copr. That should be loads of fun especially since I haven’t done anything quite like this before. So it will be hard, but fun :).

If anyone wants to check out what Copr looks like so far, you’ll just have to install TurboGears 2.0.x and then:

 $ bzr branch bzr://
 $ cd devel
 $ python develop
 $ paster setup-app development.ini

And you should have a working Copr. You can run the unit tests with the nosetests command and all 52 of them should run fine. Yay!

Congratulations to everyone who is finishing their FSC adventure today! I’ll still be coding for another month or so.


Copr design - being all things to all people

Lots of things happened this third Fedora Summer Coding week. Most people are already wrapping up, but I feel like I’m still at the beginning.

The biggest accomplishment of this week has got to be the fact that we (I and my mentor, Toshio) settled on a stable design for representing Coprs, Repos and their relationships. It was harder than it might seem, since we’ve got all these different entities in Fedora: we’ve got repos that you could look at as being either a directory with a release and an architecture or a repofile that is the same across releases and arches. When talking about releases we’ve got Fedora releases (e.g. Fedora 13, Fedora 14) and then we’ve got the packages for other distros with their own releases: EPEL and OLPC.

Now, on top of all of this we’ve got Coprs and (at least) two groups of users for the API: the end-users of the Coprs – the people that install the repos and the packages and the developers of the packages in the Coprs. The end users shouldn’t have to deal with the intricacies of the Copr/Repos/Releases model; ideally they’d just have one big button per the distribution they’re using, so they can install the repo once and have it work even after they’ve upgraded their distro three times or reinstalled five times (which is sort of how a repofile works). The package developers on the other hand could get hurt by the differences between different distro releases and their different packages – when depending on different package versions for example.

So finally we get to Coprs which should basically be a collection of packages that are available for one or more distros with each one having one or more releases. The package maintainer gets to create a Copr and choose a number of releases which they want to support with that Copr. One Copr can depend on as many other Coprs as needed. When the maintainer creates a Copr, the Copr App will automatically create repos for all of the specified releases and for each of the architectures that are supported by the buildsystem.

Everything I said until now is already implemented at the level of the TurboGears App which will provide the API for the web interface and any number of JSON clients. The schema is built and the database insteraction works fine, but repos don’t actually get created, because that’s not part of my proposal and will be handled at a different level. Oh and it’s all unit tested!

This week wasn’t just designing and building though, I spent a lot of time digging through TurboGears2 and its sub-packages’ documentation for things that should make the code simpler: raising JSON errors from nested functions, sending list arguments to JSON functions via WebTest post requests and even returning a flat list from a SQLAlchemy query on a single table column. All of these things seem to me like they should already be implemented and easy to use which makes me waste time searching for them. In fact they either are bugs or require coding them myself (at least from what I understand so far). I’ll have to investigate further, especially now since the weekend is over and I hope there’ll be more people answering questions on IRC and on issue trackers.

This next week I’ll mostly start worrying about what happens when a package maintainer submits a package to be built and that package has the right dependencies available in some releases, but not others, even though the Copr should support all of them. Will she have to submit different SRPMS for each release or should the Copr have the same version of the package in all of its releases? This will be a matter of settling upon a contract that the Copr provides its users and how uniform the Copr’s content has to be.

Fedora Summer Coding! Yay!


CRUD for Coprs and testing

This last week I worked on the first controller for the Copr TG App. There is now a JSON API to CRUD Coprs in the TG App’s database. It also supports adding/removing Copr dependencies. And everything in this first controller is (mostly?) tested with nose unit tests. The happy thing is I’m still on schedule, though I’m not ahead of it anymore, which I actually expected.

I encountered a couple of problems while setting up testing. I installed python-fedora’s FAS authorization repoze-who plugin and wasted a lot of time trying to get that working with webtest. In the process I managed to screw up something in my TurboGears installation. Since I was already too deep down the rabbit’s hole I gave up on it. (I also figured out that I don’t actually need to test anything about the FAS integration so I don’t even need to install it). So I proceeded to install TG2 inside a python virtualenv which feels a lot more hygienic and will be a lot easier to replace in case of future screw-ups. I had a few problems there aswell since the documented way to install TurboGears2 without distro-packages is broken ATM, but I now have virtualenv! Yay!

Now the next step is to figure out the right relationship between Coprs and Repos and write some code to manage Repos transparently for the user. I also have to learn to write more frequent status updates.


The late Fedora Summer coder

I started my Fedora Summer Coding last week. Although most people started almost two months ago, I chose (and was allowed to – Yay, FSC!) a different schedule because I just finished college last week.

This summer I’ll be working on a new project for Fedora – Copr. Fedora Copr will allow any Fedorian to have their own package repository with packages built and hosted by Fedora’s Infrastructure. My mentor this summer will be Toshio, I’ve always enjoyed working with him and this summer will be no different. Here is my actual FSC proposal. Although the things written in that proposal are turning out to be a bit inaccurate, it’s still a good bird’s eye view of what I’m going to do this summer.

So about the first week. Things started really slow. I did a lot of orientation, certainly more than I thought I would. I hadn’t used TurboGears2 before, though I had worked with TurboGears 1.x on Fedora’s pkgdb. When I started out I had only a TG2 automatically generated skeleton app – well it’s mostly the same now, though at least I now know a lot more about what’s in there. The fact that I had to start it up myself meant I had to learn a lot of things about TG2 that I would’ve normally just copied from other parts of a fully-functional project. And that was a great experience. In a way it’s fulfilling to be able to pioneer things in this way ;). I’m trying to only ask my mentor questions about designing the actual app and solve my “How do I … in TurboGears/Python?” questions elsewhere. My mentor has always given me a lot of independence when working on things and that feels really good, though at times I feel inexperienced. There’s the thought that the project I’m working on will be used by a lot of technical users and I’m really not sure what my decisions’ impact will be on the whole project.

I’m mostly on time with my mock-up schedule because I had set the first week for orienteering. I also wrote the DB schema for Coprs, though that was on the second week. That doesn’t mean I’m ahead of schedule however, because I’ll probably have a lot to work on the Copr controllers, and a lot of documenting and designing.

I’m proud that I setup testing after a day of wading through the scattered documentation of TurboGears2 testing. There’s mostly no documentation on testing on the TurboGears2.0 docs website. So I went to the python nose webpage. But they don’t have any info on the TurboGears2 web helpers which I needed to use. So I went to pylonshq docs about testing, but they use a slightly different syntax because they’re using paste.fixture. I finally found the TurboGears2.1 testing docs which was what I really needed. It turns out that TurboGears 2.x uses WebTest.

So now I have testing. My project is not supposed to have any web interface at this point, so writing tests is the easiest way to prove that things are actually working.

This next week I’ll probably get some work done on Copr controllers. Implementing the ability to CRUD Coprs and Repos.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.