Narnach's blog

The thoughts of Wes "Narnach" Oldenbeuving

Devnology: a bridge between developer communities

What is Devnology?

Devnology is a foundation that organizes meetings for software developers. Their goal is to bridge the gap between the many different communities that exist for the many programming languages and platforms that a software developer can choose to use. The background of the founders is mainly in the Windows world.

In order to learn from others, Devnology was started as a way to help developers share and learn tips and tricks, insights, tools, news and insights. In this respect, it is similar to how the Ruby User Group meetings work as a way to learn from each other. The difference is that the Ruby meetings are more heavily focused on socializing, while Devnology meetings are balanced more towards learning.

Ruby meetings tend to be fun beer nights: we gather in the pub, drink beer (or ice tea) and talk all night long about topics that can include Ruby, tools, new companies or projects, freelancing, pet projects and much, much more. Sometimes someone gets a laptop on the table and shows what they are working on, usually in the form of a live demo or by showing code. The key difference between Ruby meetings and Devnology meetings is the organized format of Devnology meetings. Devnology meetings have a limit on attendance, a pre-arranged location that is either rented or sponsored and they have a speaker or theme to provide enough fuel to keep the conversations going for the whole evening.

Building bridges

Getting to know new people is a chance to learn about new ways of thinking. The more different people you meet, the more you can learn. Within a single community, there tends to be an overlap in certain ways of thinking. As an example, let’s contrast Rubyists and Windows developers.

Rubyists, in my experience, tend to be young and eager to try out new things. There are quite a few people that learn on the job and don’t have a formal education. A lot of them work at smaller companies or as freelancer and they are pretty passionate about what they do. Due to Rails, a lot of work is in web development. Open Source is very important and everyone is more or less expected to have an account on Github to share their code. A lot of people have a Mac or at least run Linux on their laptop. I don’t know anyone who is serious about Windows as a developer platform for Ruby. There are some tools for Windows, but support is poor compared to Linux and Mac. Rubyists tend to be intimate with the command line and they tend to know at least a little bit of how to operate a Linux server. From front-end HTML/CSS design, through Rails code to a MySQL database design, from automated unit testing, integration testing and performance testing, a Ruby developer tends to know at least a bit about everything. Chances are, they will fulfill multiple or all of these roles on their projects.

In contrast, the handful of Windows developers I met at the last Devnology meeting tend to be older than Rubyists and appear a bit more formal. There are quite a bit of consultants working for larger companies. I actually heard people describe themselves as Software Architect and talk about corporate ladders as the most normal thing in the world. For a Rubyist to say that would be strange.

For me, these two groups are kind of opposites. Due to this, they would normally not interact much. This causes knowledge to be discovered independently and to be spread in their own communities. What is new in one group could be discovered a year later by the other group. If the two groups interact and share knowledge, the knowledge sharing might happen earlier. This is where Devnology can add a lot of value.

The git story

A good example of knowledge that is not universal is the Git distributed source code management system. For Rubyists, it is the SCM to use. Subversion is so 2007. In the Windows world, Subversion is still the way to go and Git is largely unknown. Git was developed by Linus Torvalds, of Linux kernel fame, to replace Bitkeeper after Bitkeeper stopped being free. Git has great support on Linux and Mac, but Windows support took a while to get going.

Being distributed, Git does not depend on a central repository to store all code. When you have a checkout of the code, you have the full repository. This includes all history and branches, tags, etc. Git supports a centralized workflow, where everyone pushes their changes to a single server and pulls their updates from there. Through git-svn, it is possible to even use git to interact with Subversion repositories.

The advantages of Git becomes more interesting in the Github model: everyone on the team has a public repository on Github and a private repository on their laptop. You work locally and then push your changes to your public repository. Then you pull changes from other people’s public repositories, work offline for a bit and push all changes back to your public repository. Other people can then pull in your changes and so on. This is almost an evolutionary approach to coding, where the best patches get pulled in by a lot of people and remain with the project. For closed source projects, you can use Github’s private repositories, which you can share with people of your choice. In the Ruby world, Github is becoming a social network for developers.

My personal “I am so glad I use Git instead of Subversion” moment came when the old server we use to keep a number of private repositories on had a HDD failure. We just put in a new HDD, uploaded my working copies of all repositories and got back to work. During the week we had no central server, we committed code to local branches and ran a built-in git server to share new patches over the network. With Subversion we would have needed to do extra work to regularly backup the central repository and we could not have created new patches during the week the server was down.

Last week’s Devnology meeting

Last week wednesday I attended Devnology’s second meeting and had a great evening. There were about 18 people, mostly .NET and Java developers, but there were also two Smalltalkers, a Pythonist and me as a Rubyist. Though a lot of references and examples used the Windows platform and it’s tools and languages, the discussion went about fundamentals that apply to all platforms, languages and communities. The meeting type was Round Table discussion and it was divided into two parts, each with their own topics.

First part

We gathered a number of topics and then voted on them. The two topics we would be discussing were: Generalist vs Specialist and Learning on the job vs Computer Science.

Generalist vs Specialist was interesting because it immediately became clear that everyone had a different opinion on what the terms meant. Is it specific to a language, a business domain, a platform, a role in the team or something else entirely? What is the scope in which you define these terms? It is interesting to question this. I always thought of myself as a Generalist, because I can do almost anything required in our company: be the sysadmin, be the software architect, design the database, write back-end code, write front-end code, test the system, plan the project and lead the team. On the other hand I am a Specialist, because all my knowledge is focused within the Ruby and Rails environment. I would be lost on Windows with a .NET project to develop a GUI application.

After a bit, the discussion flowed over to Learning on the Job vs Computer Science. Arguments that were put forth for CS are that it gives you a broad knowledge of different ways to solve a problem. Learning on the Job saves you four years and immediately starts to teach you what you need to know. You might not know the theoretical background about why things work the way the do, but you will be able to apply it.

The discussion also highlighted interesting perspectives of people who initially started to work and then later got their CS degree. Going this route gives you a lot more practical context to put the theory in. This is the opposite of the CS-first approach, where you first learn a lot of ‘useless’ theory that only later on becomes relevant when it gets a context in your job. A lot of theory might never get a proper context, I can imagine.

I initially started on a CS-ish route by studying Artificial Intelligence, but I did not find it challenging enough. For this reason I quit and found myself a job, where I did find a challenge and learned a lot of things. From time to time I do find myself hungry for the knowledge I could have gained at a CS course. The problem is that the school system is, in my opinion, very fake with grades as goals instead of knowledge as supreme goal. It’s been four years since I left university and I only have vague memories of most of the things I learned, while I was always among the best scoring students. On the other hand, I also tend to forget how to use certain software libraries I knew intimately half a year ago but never used since then. I think that the details of what you know will fade, but the general concepts you learn will probably expand your way of thinking and stay.

Second part

The question of this part was “What is/are your favorite…?” A couple of suggested things to list were people’s top-3 books, blogs, podcasts, tools.

My choice was the Pragmatic Programmers. They started as simple software developers turned authors, but they went on to build a publishing house for software books. Whenever I want to learn something new, I always check in there is a PragProg book or screencast available. They introduced the concept of beta books, which are books that have a beta version published as PDF while the author is still writing the book. This is great, as the author gets a lot of feedback to make the printed book better. If the book has source code that contains a bug, you can just click the ‘Report Errata’ button at the bottom of the PDF and submit a bug report and possible fix for the code on that page. Getting your hands on an early version of a book also means that you can read it way before the paper version even ships.

A second choice is Peepcode, which sells professional screencasts on a wide range of topics. Recently they also started publishing smaller eBooks as PDF. A lot of early work is focused on Ruby, but more recent work covers a wider range of topics. Non-ruby topics include Git, Emacs, Clojure, Objective-C, Productivity, Javascript and more.

Conclusion

Devnology is a great initiative that I intend to support by means of attending meetings and generating publicity in the Ruby communities I am part of. Sharing knowledge between previously unconnected communities is a good thing and I hope it will be a huge success.

Ruby quick tip

I just discovered two different ways to create a fixed-length array filled with a default value.

Array.new(5,1) # => [1,1,1,1,1]
[1] * 5        # => [1,1,1,1,1]

SimpleGate

At yoMedia we frequently have to perform simple tasks on servers that are only reachable by hopping through a number of other servers. To deploy Rails, Merb or Ruby projects, we use Capistrano and it works great. It is easy to configure and you can set it to use any number of gateway servers if that is needed to reach a server.

When deploying ruby gems or executing arbitrary commands, Capistrano does not really work for me. Don’t get me wrong, the upload:deploy task is great if you want to send an updated project file to your deployed project. Sending a gem or configuration file to an arbitrary directory on the server is not as easy. It is hard to break out of its project-sized box.

Our strategy so far has been to try and minimize our interaction with servers that were more than one hop away from our laptops. When we did have to restart a daemon or look at log files, we would have to do the SSH hop, hop, hop ritual and do our thing. Then exit, exit, exit until we’re back on our laptop’s bash shell. Having RSA key logins to a number of servers saves the trouble of having to enter passwords, but the manual SSH-hopping does get tedious after a while.

Being the kind of programmer who rather scripts the tedious things away rather than perfecting his typing speed to speed up repetitive tasks, I figured it must be possible to automate these things. Knowing I’m not the only one with this problem, Google led me to a solution. Net-ssh-gateway (NSG). Thank you, Jamis Buck!

NSG makes it possible to establish an SSH connection through a gateway server to the next gateway server behind it using port forwarding. Repeat this multiple times until you’ve connected to the final server. The code for this looks like this:

Since this is rather bulky and non-DRY code, let’s condense it into something that involves less repeating of code:

Capistrano uses NSG internally for its gateway connection code, but it seems kinda tightly coupled with Capistrano internals. Also, when I looked at it for the first time, I did not really get how it worked or what it did. To gain a better understanding, I decided to extract the relevant code into a script and rework it to be stand-alone.

The great thing about trying to do something with code you don’t really understand is that you will understand it once you have dissected it far enough. After that, you can work with it and adapt it to suit your needs.

This resulted in an early version of SimpleGate, my attempt at creating a wrapper library around NSG to make gateway chaining easier. The first version was simply the relevant Capistrano method reworked to work in isolation. The next version improved on the gateway chaining by making at as easy as calling SimpleGate.new.through_to(%w[foo bar baz]) to connect through foo and bar to baz.

SimpleGate also has a ServerDefinition to wrap a simple YAML configuration file that stores the actual server connection information. This is useful for cronjob-activated scripts and other non-interactive code when you have a password authentication server in the chain of gateways. For command line tools using SimpleGate, it saves typing.

Capistrano has a good support for SSH, its configuration files and the various authentication schemes. Passwords and RSA keys are both not a problem. SimpleGate currently only supports passwords, as that is what NSG supports out of the box. RSA key logins are a todo item. For configuration, it does the simplest thing that can possibly work: just store the connection info as a plain YAML file in ~/.servers.yml.

After two minor version bumps, I had something that was good enough to build a script that connected through multiple gateway servers to my target and request its uptime. The next step was to execute arbitrary commands, which was a small modification.

Here I discovered another hard to reproduce feature that Capistrano executes in a really nice way: sudo passwords. For some reason I still have to discover, SimpleGate does not let me enter a password when asked for it on the remote server: it just fails the password check and quits. I guess that is another todo item.

After discovering this I wanted to check up on another server that was hidden behind a number of hops and started to change the hard-coded gateway connection sequence in my test script to connect to the new server.

Woah! Wait! Full stop.

Hard-coded connection info is not good. The server name should be a command line option and the connection sequence should be figured out by the script, not by me. Since I was not interested in manually working out all possible connection sequences, I figured it was time to add a simple path-finding Router class to the project…

After a quick read of relevant sections in Bratko, to refresh my knowledge of the topic, I decided to model the search space as a directed non-cyclical graph and search through it using a simple depth-first recursive search algorithm. Support for cyclical graphs will be another todo.

For every node in the network, all its possible connections are described in a YAML file, that is just a Hash of Arrays with server name strings. A special ‘local’ node represents my laptop or any arbitrary internet-connected system. The search algorithm comes down to:

  1. If we search from target to target, return that the route is target.
  2. If the current node has outgoing connection possibilities, try them all, keep the shortest and return it with the current node prepended to the returned list of nodes.
  3. If there are no outgoing nodes, return nil.

This results in a relatively naive path finder, but with only 15 nodes in my network, this is not much of a problem. Smarter algorithms can be added once I actually need them.

A version bump later I remembered by original goal: upload a gem, install it and restart the daemon associated with it. Restarting is all done in userland, so that is not a problem. Installation requires sudo, which is still a todo item. Uploading was still open.

NSG can open a normal net/ssh session. net/sftp can use this session to do file transfers. A quick copy-paste-adjust later I had a new executable for copying a single file to a single server, through an arbitrary number of hops.

Right now SimpleGate is at version 0.5.0 and it has the three items noted above as left to do. Then it has its core functionality and should be polished up for its 1.0 version.

The command line tools should get parameter support and a –help interface. Then the config files should both be documented and command-line editable. Once those things are in, multi-server support might be useful. In a Capistrano-like fashion connect to multiple servers (sequentially or in parallel) and execute commands on all of them or upload file(s) to them. The file uploader can get a better interface instead of mimicking the code.

Result of ACK: Future

Wikipedia’s definition of a Future in programming:

In computer science, futures and promises are closely related constructs used for synchronization in some concurrent programming languages. They both refer to an object that acts as a proxy for a result that is initially not known, usually because the computation of its value has not yet completed.

Yesterday evening a number of Rubyists from Amsterdam came together in the public library with our laptops to do cool computer stuff until the library closed.

Among the topics we discussed were new-ish programming languages such Ioke and Clojure. From here the discussion went to the languages that preceded them, like Self and IO.

I think that at some point someone asked “What can IO do that Ruby can’t do?”, so we started looking at the list of features of IO. There we discovered futures.

Futures look nifty: create an object that does not yet have a value and then move on and eventually do something with the object when you need its value. It will block if there is no value yet, which makes sense, but as long as you don’t need the value, you can just move on with your business. This can be useful to do heavy computations or to initiate asynchronous interaction with a remote API. As long as you don’t need the result immediately, a future can be useful.

Then I mentioned Ruby’s Thread#value, which was new to people. So a bit of code was created that looked similar to:

# sleeps for 3 seconds, then prints 615
t = Thread.new { sleep 3; 123}
puts t.value * 5

Explicitly creating a Thread and calling its value does the trick, but it is not very elegant. If you read it, you will probably end think, “What is that Thread doing? Oh, there’s a #value call there. Thread + #value is a future!”. Or you don’t make the connection and misunderstand what this code is doing.

How should it look then to be more elegant?

# sleeps for 3 seconds, then prints 615
f = future { sleep 3; 123}
puts f * 5

So, this is way more elegant! You see it is a future, because it says so! If you don’t know about futures, there is Wikipedia or Google and you might figure out what it does.

In order to make this happen, we need to define a future method that returns an object that when called returns Thread#value for that future. A proxy object. This leads to the next implementation:

class Proxy
 def initialize(&block)
   @thread = Thread.new(&block)
 end

 def method_missing(*args, &block)
   @thread.value.send(*args, &block)
 end
end

def future(&block)
 Proxy.new(&block)
end

f = future { sleep 3; 123}
puts f * 5

We created a Proxy class to do the threading logic and use method_missing to capture method calls, which we re-direct to Thread#value.

This works as long as you don’t call methods defined on Object or Kernel, because you get those for free in every class. You need to either avoid using them or get rid of them. There is BlankSlate, but since we were just trying to make our idea work, we just un-defined most methods on the object. Then we had the crazy idea to drop the Proxy object altogether and make Thread function as its own Proxy object. Thread, say bye-bye to your methods!

Here is the final bit of code we produced:

Then the library closed so we went home, after which I polished it a bit and created a github project for it.

This morning my colleague, Filip Slagter, asked me if there wasn’t already a library to do futures in Ruby. After a bit of searching, I found there is one. And it looks nice! Still, I like to re-invent the wheel. It is the best way to learn how something works.

Adventures in REIA land

One of my current interests is the new Reia programming language, created by Tony Arcieri. The wiki contains most of the documentation, but but besides a simple “Hello, World!” and Fibonacci implementation, there is not much working code. Reia is a work in progress, so that makes sense.

Philipp Pirozhkov created Ryan, a web framework built on top of Reia and YAWS. For reasons I have yet to figure out, it does not want to build on my machine. At least there’s more code to look at to learn the language. Ryan also has a RSpec-like syntax for writing tests, which look interesting.

Starting with web development might be a bit too ambitious for me. I actually managed to mess up a simple “Hello, world!” example, so I’m starting slow. Note to self: methods in Reia have parentheses. It’s not Ruby, where you can omit them.

Wrong hello world:

puts "Hello, world!"

Proper hello world:

puts("Hello, world!")

Sincy my brain still thinks in Ruby, let’s start with a simple bit of Ruby code and convert it to Reia.

First, the Ruby code:

[1,2,3].each { |n| puts n }

This prints out three lines with 1, 2 and 3 on them. Now the same code in Reia:

[1,2,3].each { |n| puts(n.to_s()) }

The two obvious differences:

  1. All method calls need their parentheses. So use “puts(‘String’)” instead of “puts ‘String’”.
  2. The int has to be explicitly cast to a String.

Just like Ruby, Reia has multiple ways to write this code. Another Ruby-esque way to write it is by using the ‘do’ block notation instead of curly braces:

[1,2,3].each do |n|
  puts(n.to_s())

Notice the puts() is indented and there is no “end”: that is the Python-style indentation at work. A third way to write the code is by (ab)using List Comprehensions:

[puts(x.to_s()) | x in [1,2,3]]

This is the strangest form for me, since Ruby does not have something similar. The way I interpret it is by reading from right to left: for each x in [1,2,3], do the puts thingy left of the pipe.

Hello, world!

After complaining over and over that I should really get myself a blog, I finally have one! Now, let’s be nice and let me introduce myself.

My name is Wes Oldenbeuving and I’m a programmer from Amsterdam, the Netherlands. My hobbies include reading books and blogs, playing video games, watching tv series and programming.

My interests cover a wide variety of topics like computer science, economic theory, history, (melodic) metal music, politics, philosophy, personal growth and cooking. I’m not a great chef, but I do try to improvise on recipes every once in a while.

I have been programming (in Ruby) since March 2006 for the Dutch company yoMedia. Ruby on Rails is what brought me to Ruby, but it is not the only thing keeping me. Ruby as a language is really nice to work with and works for both simple scripts, to automate mundane tasks, and to build complex applications. The people in the local Ruby communities are also nice, smart and full of crazy creative ideas.

We have a number of regular events that I attend 99% of the time: Amsterdam.rb, Utrecht.rb and the soon-to-start Amsterdam.rb screen-peeking diner thingy.

I expect my blog posts to mainly cover programming-related topics, though they will not necessarily be all about Ruby. In fact, expect the first posts to be about the new Reia programming language.