Creating a continuous integration server in a hackathon, and how to make SSH connections using ruby
Share on twitter
Share on facebook
Share on linkedin
Share on tumblr

Creating a continuous integration server in a hackathon, and how to make SSH connections using ruby

Rails Rumble wrapped up recently and with that, the end of an exciting 48 hour experience where developers can challenge themselves, prove all their programming skills and create an awesome product from an idea.
This is the third hackathon I’ve participated in since I began working as a professional programmer, and I wanted push myself and create my own version of a continuous integration server.
While creating a continuous integration server we went through a lot of challenges, from user experience design to interesting architectural decisions, but one of the most challenging and interesting situations I went through was running the build jobs from a repository in a virtualized environment. At the very beginning we realized that the safest way to run the tests for a given repository was to isolate the test environment from the web server environment due to because of security risks, so we decided to setup the architecture in the following way:
 
CI2
 
 
 
The idea was to connect to a remote (virtualized) server over the SSH protocol and run the script with a provisioned environment (ruby, rvm, rubygems, postgresql, sqlite, mysql, etc.). We spent some time researching how to connect via SSH using ruby and found a library called Net::SSH which allows you to create SSH connections easily and execute a command. We did some tests and it worked but unfortunately it was very hard to navigate through folders and request a bash environment just like a normal SSH connection from the UNIX terminal, so after a long researching, testing, and reverse engineering many open source projects that use Net::SSH we decided to create abstraction layers for each of its components (use-cases).
 
CI3
 
 
By giving single responsibility to each of the classes we were able to easily build the programming interfaces on top of the CI module (see SOLID).
The simplest case scenario, you can connect to a server just by instancing the objects from the top level class of the CI module as following:

ssh = Ci::Environment.new
ssh.exec('echo "hello world from the remote server!"')
ssh.session.buffer # and get the console output string for all the commands you executed previously.

Pretty easy, right? Let’s take a look inside the module:

# From Ci::Environment class
  def initialize(buffer = nil)
    @session = Ci::SSH.new(
      user: 'username',
      host: 'example.com',
      buffer: buffer
    )
  end
  def exec(command)
    @session.exec(command)
  end

This class is only responsible for setting up the connection parameters that Ci::SSH will handle as a connection string, so we have encapsulated the Ci::SSH work in a lower level of the Ci namespace. You can actually use it outside the Ci::Environment class but you have to customize it as seen in thedef initialize method above. Now let’s take a look at how the Ci::SSH works.

# From Ci::SSH class
  # The class constructor will raise an error if the argument list received was
  # incorrect so we can ensure the persistence of the object
  def initialize(options={})
    raise_errors if options.empty?
    options.keys.each { |k| raise_errors unless argument_whitelist.include?(k) }
    @user     = options[:user]
    @host     = options[:host]
    @port     = options[:port]
    @password = options[:password]
    # We can receive an object that handles the communication or just read the output string.
    @buffer   = options[:buffer] || ""
  end
  # We generate here the connection instance with Net::SSH that will handle the connection.
  def connect
    hash = {}
    hash.merge!(password: @password) if @password
    hash.merge!(port: @port)         if @port
    @session ||= Net::SSH.start(@host, @user, hash)
  end
  # Here's where the magic happens
  def exec(command)
    # We're generating the connection once we execute the first command
    # not when the object gets instanced
    # and we do it just once
    connect unless open?
    exit_code = nil
    # We're using the Net::SSH#open_channel to actually execute the commands in a shell session
    @session.open_channel do |ch|
      # We're requesting for an interactive terminal that will allow us to
      # perform as many commands as we want and persists while the connection still open
      ch.request_pty do |channel, success|
        raise StandardError, "could not obtain pty" unless success
        # As the time we actually execute the commands we login to the bash session
        # because we need the bash environment to make sure that everything is loaded
        # rubies, gems, environment variables, etc.
        channel.exec("/bin/bash --login -c #{Shellwords.escape(command)}") do |ch, success|
          raise StandardError, "could not execute command" unless success
          # The channel execution is event based so we define what are we going to do with the
          # information received by the server with the following callbacks, in this case we're
          # going to just save all output in the buffer string/object
          ch.on_data do |ch, data|
            @buffer << data
          end
          ch.on_extended_data do |ch, data|
            @buffer << data
          end
          ch.on_request("exit-status") do |ch, data|
            exit_code = data.read_long
          end
        end
      end
    end
    # The number of seconds to wait for the event loop until the next command
    @session.loop(1)
  end
  def close
    @session.close if open?
  end

By defining these two classes the usage of Net::SSH becomes pretty straight forward:

ssh = Ci::SSH.new(host: 'example.com', user: 'user1')
ssh.exec('echo "hello world!"')
ssh.buffer # Shell output

By the end of the hackathon, all this made communication possible between the continuous integration environment and the UI. We connected the shell output to a websocket using the pusherservice, so we could push notifications from the server in real time to the user; you can see it live by visiting the actual project from RailsRumble Simple CI.

Questions?

Let me know via comments on this post or via email antonio@tangosource.com.

Share on twitter
Share on facebook
Share on linkedin
Share on tumblr