One of the problems I keep running into with Puppet is that the packages I’m trying to install are very new and I need to run apt-get update to update the repositories. Because Puppet does not run scripts it gets rather annoying to get the update to run before you install any packages.

I found a lot of solutions around the web but this one seems to work best for me.

In the light of the recent Rubygems.org security compromise the community has been looking at ways to make Rubygems.org and Ruby gems in general more secure. The project is still ongoing and feel free to help them out on #rubygems on Freenode, but here is a highlight of what I think are some of the main issues.

Some of the issues highlighted here are taken from Ben Smith’s enlightening (but scary) talk at Aloha Ruby Conference.

Disclaimer

I am not a security expert. I am just a Ruby developer and a gem author that is worried about the current state of the Ruby gems ecosystem. I also am worried that the next negative news around Ruby will involve the problems described below.

What are Ruby gems and what is Rubygems.org?

For those not familiar, Rubygems.org is the most popular repository of “gems” for the Ruby language. Gems are libraries made up out of Ruby (and optionally C) code and can be uploaded by anyone who registered for an account. Rubygems.org currently hosts 50,685 gems which have been downloaded 1,259,533,358 times since July 2009. Ruby gems are not only hosted on Rubygems.org, anyone can run their own repository but Rubygems.org is definitely the most used one.

Current state

Some parts of the current infrastructure are worrying.

• Hard to tell if gems were changed on the repo. It took the Rubygems.org volunteers more than 24 hours to verify every gem’s checksum against external mirrors.

• Impossible to tell if gems were uploaded by gem owner. It is currently very hard to know if a gem was actually uploaded by it’s owner. Developer machines can be compromised and a users API credentials for Rubygems.org are kept unencrypted in ~/gem/credentials.

• Gem owner isn’t notified of new gem uploads. When a gem developer’s credentials are compromised a new version of the gem can be uploaded without the gem owner ever knowing.

• Impossible to notify a gem user of compromised gems. When a gem developer’s credentials are compromised it is hard to notify anyone who uses any of the gems published by the developer of the situation.

• Gems can run code on install. This is probably the most interesting attack vector in the foreseeable future. It seems this feature was relatively unintentional as it involves tying into the fact that Ruby gems can contain C code. Running code on install will mean that gems can steal the unencrypted Rubygems.org credentials which can then be used to modify the compromised user’s gems and spread the malicious code further.

Proposals for change

• Notify gem owners of newly published gems. Adding a simple email notification to the gem owner will at least allow for easier detection of compromised gems. Sadly at this point the gem is already compromised and possibly already spread over any mirrors and downloaded by users.

• Secure developer’s Rubygems.org credentials. This is pretty simple. My ssh key has a passphrase on it and so should my Rubygems.org credentials. Stealing a rubygems.org API key is easy, using one that requires a passphrase a lot harder.

• Stop running code on gem install. I totally see the need for having C extensions in a Ruby gems, and those extensions need to be compiled, but we seriously need to find a way to compile C code without allowing for the arbitrary execution of code on install of a gem.

• Automatically mirror gems and checksums. A system involving the automatic mirroring of gems and their checksums to other servers would definitely have made the verification of gems a lot easier in the last few days.

• Force signing of gems. Yes, you can sign your gems but almost nobody does (and neither do I). Additionally it’s a pain to force the usage of signed keys on the gem user’s side, not to mention the futility as most gems aren’t signed. Signing is the way to go though and work on this has started. It’s a difficult topic though and work is being done to make it as painless as possible for users and developers.

• Notify gem users of unsigned/insecure gems. The rubygem binary (together with tools like Bundler) should be updated to allow for verification of signatures which will allow it to notify gem users of unsigned or compromised gems.

How can I help?

• Code: rubygems, rubygems-trust(fork for implementing a signed approach)
• Discussion: #rubygems and #rubygems-trust on Freenode

Did I miss anything?

Please let me know and I’ll add it to the list.

With all the new years resolutions out of the way I want to share what (programming) skills I learned or revisited in 2012.

This will be more positive than looking back in a year and counting the things I promised but didn’t do.

Programming skills

Advanced Ruby

I knew Ruby already, but it always was a very “Rails” type of Ruby. Working with Caius and Baz at EmberAds has brought this to a whole new level though. Writing modules, writing gems, extending classes, applying best practices, etc, all of those can seem daunting at first but they are oh so useful.

It also has had a great effect on my proficiency in Rails.

You can spot a good Rails programmer by their number of regular Ruby objects in their codebase

I don’t remember who said this but it really has started to ring truth to me in the past year. The Rails community has lately been alight with discussions about DCI, Concerns, Service Objects, Delegation and other ways of moving more and more code out of controller and models and into regular Ruby objects.

Testing code using RSpec

Learning to write tests has always been at the top of my list of things to learn but I somehow never got to it. Luckily Caius and Baz wouldn’t allow any code to go untested and it soon became a much loved practice. It is a hard skill to master though so I don’t blame my past self.

I think it’s a skill best learned by example and from a colleague. I still feel like I’m learning more and more about the intricate aspects though as it’s relatively easy to write a test that tests too much or too little. We’ll see where I am at the end of 2013.

<TL;DR>

Who is responsible for developing a hack beyond a hack day? Is it just the developer or should the sponsors and organisers be setting a right framework? If Open Sourcing a hack is not enough, what then? What do you think? Let me know in the comments.

The problem

I was surprised to see the amount of criticism I was getting during the CharityHack 2012 (more about that and my hack soon) regarding my intensions of continuing development of my hack after the event.

See, Tom and I had made a nice hack that had gathered quite a bit of interest from people at CharityHack. Even before we finished the product quite a few people from charity related organisations had come up to me to ask/tell me:

• if I was going to continue development
• if I was going to start a company
• that they could secure some funding for me
• when they could use the product

They were baffled when I told them that I - as for that moment - had no intention of starting a company based on this hack. I already have a day job, I already have a startup, and I already am busier than I should be. I offered to put the code online, make it open source, allow anyone to work on it, but somehow that wasn’t enough.

And I can understand why: it’s hard to find developers with a passion for someone else’s idea. It also hurts to see an idea abandoned as quickly as it was created. But it made me think: who is responsible for these hacks after an event? Especially an event like CharityHack where the goal is most definitely something more noble than some of the other Hack Days.

Maybe the organisers (in this case PayPal) should be the ones to set a framework. I’ve seen quite a few Hack Days where the winners get some funding to continue the product. But often this funding is very low (funding 1 developer for about half a year max), and it fails to ask the question: does the developer even want any funding? And if so, can they actually find the time?

Solutions?

So are there any solutions to this? Last weekend at the Mozilla Festival I was talking to the people behind Hacks/Hackers and they seemed to have the same issues. I thought it might be handy to have some more regular events (every month or week) to keep ideas alive and show the progress achieved, but I was told that this was already tried without much success.

It was suggested that it would be cool to have some agency as a sponsor/host/participant that would provide X hours of development work on the product after the event to help bring the product to a real first version. This reminded me of the Think Big event I helped out with at Campus Party Berlin. Here it was Telefonica that offered a certain amount of hours to have an app concept worked out into a product, seeing as the idea creators weren’t actually developers themselves.

In the end I don’t know what the solution is though. I’ve always thought that “just open sourcing it” would be enough, but I have to admit that even I can see that that would just lead to more and more awesome but abandoned projects.

What do you think? Let me know in the comments.

I’ve been playing with Puppet recently in order to finally teach myself a bit about server management. I decided for Puppet because… well… I didn’t have time to play with Chef yet.

I can’t show any of my code because it contains some stuff I’d rather not open up, but here are some of my global notes on Puppet that I wanted to share.

The good

• It did the job. I now have a few scripts that I can use to quickly setup a server for Rails, including NGINX, PostgreSQL, Unicorn, Monit, and much more.
• Quick deployment. I can now deploy a new Rails app to a server within minutes!
• The Learning Puppet series is a good starting point and explains most of the basics
• Low tech. Running a puppet script really doesn’t involve much more than running: puppet apply path/to/puppet/file.pp

The bad

• No single server deployment solution. There doesn’t seem to be a best practice on how to use puppet with just one server. I know that the serious people will have to manage many many servers, but I think that they could make Puppet more accessible to newcomers by having a good solid solution for their own server. Many of us learn new things by trying them out for our own hobby projects before using them in big-business contexts. I have resorted to using Capistrano for deployment, but it just feels wrong somehow.
• Not many great modules. Puppet has a modules system which allows anyone to package their solutions and share them with the community. Sadly most of the modules are old, unmaintained, and often broken. Additionally the modules often don’t solve the problems in a way that I’d like them to, forcing me to write my own.
• Convoluted language. Puppet requires Ruby to run, but the DSL is not Ruby, nor is it Javascript, or JSON,or YAML, or anything else that so many developers already know. The architecture for defining classes, types, and modules is convoluted, backwards, and feels very awkward. I think one of the reasons why there aren’t many well written modules is very much related to this.
• Compiling from source. One of the biggest missing features seems to be some good architecture for installing anything that isn’t packaged up. I often want to run a different Ruby, Nginx, Apache, PHP version than is in the package repositories. I know this is a hard problem, but again I wish there was some kind of best practice.
• Ordering from hell. Puppet does not run your actions in order as specified in your .pp file. Instead you can tell it if something has a requirement. In my experience almost everything has a requirement and specifying the orders is becoming a nightmare and a real frustration.
• Missing features. There are a few features that are still missing. One of the most important ones (in my eyes) is the ability to generate a folder recursively (e.g. mkdir -p path/with/multiple/folders). Instead you are now forced to create every layer as a new statement.

Conclusion

Puppet will do for now, but I wish it was a bit more opinionated in how it thought it should be used. The language is not pretty and very verbose, and its lack of best practices for single server deployment is a real omission.

Does anyone know how Chef performs in these regards?

So last weekend I was at Over The Air 2012 - on of my favorite regular Hack days - and for the first time I decided to actually try and make a “native” mobile app.

Seeing as I don’t want to have to redo the app for every platform I decided to try out one of the more famous cross platform development frameworks: PhoneGap.

Here are some of the things I discovered during the weekend:

• PhoneGap isn’t really a framework. It’s really nothing more than a wrapper around a local HTML page with an extra JS library to access native features like the camera, local storage, etc.
• Although jQuery Mobile looks very nice, it seems to emulate iOS, making it look out of place on any Android or Windows device.
• PhoneGap in combination with jQuery Mobile allows you to very quickly make a static native app, but as soon as you want to add any dynamics it seems to become quite hard. You will need some kind of JS MVC to tie it all together.
• I had a play with KnockoutJS - which we already use at EmberAds - but it had its issues:
  • Good tutorials on how to integrate with jQuery Mobile were missing, and I was encountering rendering issues.
  • It’s not really a MVC, and tying it to template partials seemed harder than it should.
  • Automatic storage of models to local storage/server was missing.

So although I’m happy with the work we did on the first version of our app it clearly will need a rewrite in time to be viable. Here’s some of the research I’ve done so far:

• I will probably replace Knockout with BackboneJS which seems to be more flexible and actually support storing to server and/or local storage.
• Alternatively I might look at Sensa Touch 2.0 which seems to be more than just a UI framework but actually also include a full MVC framework.
• I found some tutorial describing how to integrate PhoneGap, jQuery Mobile, and Backbone, including a bootstrapped app that integrates them all in a way I might be able to re-use.

tl;dr

I want to run a new kind of event: OpenCodeCamp (working title). OpenCodeCamp should be an event that promotes knowledge sharing and skills swapping amongst developers of any age or level of experience. It should be free of commercialism, competition, sexism, racism, age-ism and accessible to developers of any seniority.

Read the full post and leave a comment if you want to get involved.

I’ve been attending HackDays since 2007 when I travelled to Alexandra Palace for the first HackDay organised by Yahoo and the BBC. I was a n00b at that time (some might say I still am, #shakesfist) and I had no idea what to build. At the time I was trying to learn how to use web frameworks and at the HackDay I got my first exposures to APIs, Django, Rails, OpenID and much more. I don’t think I made anything that weekend, but damn did I learn a lot!

Since then we’ve come a long way; HackDays are no longer just organised by big organisations. There’s pretty much one happening every weekend (sometimes more), and they are more often than not focussed around a topic. I attended at least 20 HackDays since, and I even helped organise not 1 but 2 HackCamps with GeeksOfLondon.

But lately I’ve been noticing a little trend. I was at GameHack yesterday and I had come to have a fun weekend to play with some APIs, maybe play with some WebGL libraries, and get an idea of how game development works. I am not a game developer and don’t intend to be one, but I always enjoy learning new things.

Here are just some interesting gems and other software that have caught my attention recently:

• oh-my-zsh makes it easy to use zshell as a replacement for bash on OS X. It is very easy to install and comes with a whole bunch of useful plugins. I’ve only really dived half into it but already feel like it’s been making my console better. I personally love how it shows the Git branch I’m working on at the moment.
• Bloom Filters are a very cool way of doing a quick and space efficient lookup if an item exists in a set. It’s used by databases and caches, and Google uses it for their Save Browsing service. There’s some cool ruby gems that make it easy to use them and I already have an idea where to deploy them next.
• How Github uses Github is pretty much how we use Github at EmberAds. The more we do it the more I see the benefits especially:
  • We work asynchronously
  • We use the hell out of Campfire
  • Master is always deployable
  • Pull Requests are our code reviews, requests for comments, and discussion boards.
  • Other apps have great features like prioritising stuff. But honestly, we have barely touched BaseCamp in weeks.
• Fast UUID generation is something we needed at EmberAds seeing as we use UUIDs everywhere and most of the Ruby gems were rather slow. Caius cooked up a Ruby gem that uses a native C library which in our simple benchmarks spit out UUIDs about 6x faster.
• On that note, acts_as_uuid is what we use to store UUIDs on our models in Postgres.
• Rake progressbar makes it easier for me to understand if I can go and make a brew or not when some of my rake tasks are running.
• And finally, static site generators like Nanoc, Jekyll, and Octopress (used on this site) have been really catching my eye lately. Mainly because of the security and scaling implications. I’ve been working on a blog post on this, mainly to inspire regular bloggers that you don’t need a big server and a massive database to render a Wordpress site that will load slow as hell regardless.