Posts Tagged rabbitmq

Installing RabbitMQ + Stomp in Ubuntu/Mac

Posted by jcastaneyra in OpenSource on May 30, 2010

En español

In the project I’m working now it was needed to have RabbitMQ running with Stomp as a message queue for Orbited, we know that Orbited comes with MorbidQ, but in our system we already had running RabbitMQ and used by other processes, so, two queue systems at the same time were not needed.

But, searching for Stomp over RabbitMQ info or some kind of tutorial was difficult and painful, a friend of mine would say “a real pain in the ass”, there was info but not too clear. For that reason here is another note to myself, just in case.

  • RabbitMQ 1.6.0 installed, here there are some steps to install it.
  • Mercurial is needed to download Stomp code
  • Download Stomp code
    • hg clone -r rabbitmq_v1_6_0 http://hg.rabbitmq.com/rabbitmq-stomp
  • Compile Stomp (use your RabbitMQ include directory, maybe in Linux /usr/lib/erlang/lib/rabbitmq_server-1.6.0/include , in Mac I have it here /usr/local/lib/erlang/lib/rabbitmq_server-1.6.0/include)
    • cd rabbitmq-stomp
make RABBIT_SERVER_INCLUDE_DIR=/usr/lib/erlang/lib/rabbitmq_server-1.6.0/include
  • Copy your compiled Stomp to the Erlang libraries
    • sudo mkdir -p /usr/lib/erlang/lib/rabbitmq-stomp
sudo cp -R * /usr/lib/erlang/lib/rabbitmq-stomp
  • Add the Stomp configuration for RabbitMQ (also use your correct directory here, in Linux /usr/lib/erlang/lib/rabbitmq-stomp/ebin and in my Mac /usr/local/lib/erlang/lib/rabbitmq-stomp/ebin)
sudo vim /etc/rabbitmq/rabbitmq.conf

#Add the lines below
NODENAME=rabbit
NODE_IP_ADDRESS=0.0.0.0
NODE_PORT=5672

LOG_BASE=/var/log/rabbitmq
MNESIA_BASE=/var/lib/rabbitmq/mnesia

SERVER_START_ARGS='
  -pa /usr/lib/erlang/lib/rabbitmq-stomp/ebin
  -rabbit
    stomp_listeners [{"0.0.0.0",61613}]
    extra_startup_steps [{"STOMP-listeners",rabbit_stomp,kickstart,[]}]'

And that’s it, if you start RabbitMQ will see how STOMP-Listeners are starting (In Linux you maybe will need to stop RabbitMQ with the following command sudo /etc/init.d/rabbitmq-server stop before running the next step, in my Mac I use this command to start RabbitMQ sudo /usr/local/lib/erlang/lib/rabbitmq_server-1.6.0/sbin/rabbitmq-server)

deploy@localhost:~$ sudo rabbitmq-server
RabbitMQ 1.6.0 (AMQP 8-0)
Copyright (C) 2007-2009 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
Licensed under the MPL.  See http://www.rabbitmq.com/

node        : rabbit@localhost
log         : /var/log/rabbitmq/rabbit.log
sasl log    : /var/log/rabbitmq/rabbit-sasl.log
database dir: /var/lib/rabbitmq/mnesia/rabbit

starting database             ...done
starting core processes       ...done
starting recovery             ...done
starting persister            ...done
starting guid generator       ...done
starting builtin applications ...done
starting TCP listeners        ...done
starting STOMP-listeners      ...done

broker running

In english

En el proyecto en el que estoy trabajando era necesario tener corriendo RabbitMQ con Stomp para poder hacer conexiones con Orbited, Orbited ya incluye MorbidQ, pero en este caso ya teníamos corriendo RabbitMQ para otros procesos y no era necesario tener un segundo sistema de queues.

Sorprendentemente, encontrar información de cómo agregar Stomp a RabbitMQ resultó algo doloroso, si había información pero como que no indicaba bien que pasos seguir. Entonces como parte de mis notas personales estoy agregando esta receta por si se llega a ofrecer de nueva cuenta.

  • Tener instalado RabbitMQ 1.6.0, aquí hay unos pasos para instalarlo.
  • Tener instalado Mercurial para bajar el código de Stomp
  • Bajar el código de Stomp
    • hg clone -r rabbitmq_v1_6_0 http://hg.rabbitmq.com/rabbitmq-stomp
  • Compilar Stomp (apuntando al directorio de include donde se encuentra RabbitMQ, en Linux /usr/lib/erlang/lib/rabbitmq_server-1.6.0/include y en Mac yo lo tengo en /usr/local/lib/erlang/lib/rabbitmq_server-1.6.0/include)
    • cd rabbitmq-stomp
make RABBIT_SERVER_INCLUDE_DIR=/usr/lib/erlang/lib/rabbitmq_server-1.6.0/include
  • Copiar Stomp compilado al directorio de librerías de Erlang
    • sudo mkdir -p /usr/lib/erlang/lib/rabbitmq-stomp
sudo cp -R * /usr/lib/erlang/lib/rabbitmq-stomp
  • Agregar la configuración de Stomp a RabbitMQ (también apuntar al directorio correcto en la configuración, en Linux /usr/lib/erlang/lib/rabbitmq-stomp/ebin y en mi Mac /usr/local/lib/erlang/lib/rabbitmq-stomp/ebin)
sudo vim /etc/rabbitmq/rabbitmq.conf

#Add the lines below
NODENAME=rabbit
NODE_IP_ADDRESS=0.0.0.0
NODE_PORT=5672

LOG_BASE=/var/log/rabbitmq
MNESIA_BASE=/var/lib/rabbitmq/mnesia

SERVER_START_ARGS='
  -pa /usr/lib/erlang/lib/rabbitmq-stomp/ebin
  -rabbit
    stomp_listeners [{"0.0.0.0",61613}]
    extra_startup_steps [{"STOMP-listeners",rabbit_stomp,kickstart,[]}]'

Listo, si intentan arrancar RabbitMQ se verá que dice que está levantando los STOMP-Listeners (En Linux talvez necesiten terminar el proceso con sudo /etc/init.d/rabbitmq-server stop antes de correr lo siguiente, y en Mac para levantar el servidor yo ejecuto sudo /usr/local/lib/erlang/lib/rabbitmq_server-1.6.0/sbin/rabbitmq-server)

deploy@localhost:~$ sudo rabbitmq-server
RabbitMQ 1.6.0 (AMQP 8-0)
Copyright (C) 2007-2009 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
Licensed under the MPL.  See http://www.rabbitmq.com/

node        : rabbit@localhost
log         : /var/log/rabbitmq/rabbit.log
sasl log    : /var/log/rabbitmq/rabbit-sasl.log
database dir: /var/lib/rabbitmq/mnesia/rabbit

starting database             ...done
starting core processes       ...done
starting recovery             ...done
starting persister            ...done
starting guid generator       ...done
starting builtin applications ...done
starting TCP listeners        ...done
starting STOMP-listeners      ...done

broker running

, , , , ,

View Comments

Enviando correo de manera asíncrona en Rails usando Workling, Workling-Mailer y RabbitMQ

Posted by jcastaneyra in Howto/tutorial, OpenSource, Ruby on June 10, 2009

Enviando correo de manera asíncrona en Rails usando Workling, Workling-Mailer y RabbitMQ (English)

El trabajar con message queues es bastante interesante, ya que podemos mandar procesos al background y que estos sean procesados de manera asíncrona, un ejemplo podría ser el envío de correos, aunque también podría servir para realizar otras tareas, por ejemplo, como el envío de mensajes sms, generación de reportes, generación de pdf’s, etc.

En esta ocasión les quiero presentar como enviar correos de manera asíncrona haciendo una aplicación sencilla haciendo uso de los puglins Workling y workling-mailer y del sistema RabbitMQ, en teoría, con esto se podría ajustar esta solución fácilmente a cualquier otro proceso que se quiera realizar de manera asíncrona.

Aplicación base

Se crea una aplicación básica con restful_authentication, y con la opción de activación.

Lo primero que hacemos es crear la aplicación en rails

rails mail_async_example -d mysql
 
rm public/index.html
 
script/plugin install git://github.com/technoweenie/restful-authentication.git
 
script/generate authenticated user sessions --include-activation

Creamos nuestra base de datos y migramos

rake db:create
 
rake db:migrate

Y hacemos unos pequeños cambios al código de la aplicación para enviar correo según las instrucciones de restful_authentication, así que agregamos lo suguiente:

config/routes.rb

map.activate 'activate/:activation_code', :controller => 'users', :action => 'activate', :activation_code => nil

config/environment.rb

config.active_record.observers = :user_observer

Con esto ya tendríamos nuestra aplicación básica la cuál podríamos probar de inmediato apuntando en el browser a http://localhost:3000/signup

Y agregando algunas cosillas extras, a app/controller/application_controller.rb agregamos la línea:

include AuthenticatedSystem

Esta misma línea que agregamos al Application controller la removemos de app/controller/sessions_controller.rb y de app/controllers/users_controller.rb

Y creamos un nuevo controlador

script/generate controller portal index

Modificamos config/routes.rb para agregar al último como página root

map.root :controller => "portal"

Por último agregamos un layout para nuestra aplicación:

app/views/layouts/application.html.erb

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
       "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
  <title>Contracts: <%= controller.action_name %></title>
  <%= stylesheet_link_tag 'scaffold' %>
</head>
<body>
<% if current_user %>
   <%= link_to 'Logout', logout_path %>
<% else %>
   <%= link_to 'Login', login_path %>
   <%= link_to 'Sign up', signup_path %>
<% end %>
 
<% [:error, :warning, :notice].each do |level|
    if flash[level] -%>
      <div class="flash <%= level.to_s -%>"><%= flash[level] -%></div>
  <% end -%>
<% end -%>
 
<%= yield %>
 
</body>
</html>

Instalando amqp, workling y workling_mailer

Ahora bien, antes de instalar Workling, para poder trabajar con RabbitMQ es necesario instalar la gema de amqp, esta gema requiere también de la gema de EventMachine, por lo que al instalar amqp también instalará EventMachine.

gem sources -a http://gems.github.com/
sudo gem install tmm1-amqp --no-rdoc --no-ri

Vemos nuestro listado de gemas

sudo gem list

Ahora si, a instalar workling y workling_mailer

script/plugin install git://github.com/purzelrakete/workling.git
 
script/plugin install git://github.com/langalex/workling_mailer.git

Para poder utilizar workling con RabbitMQ agregamos al final de config/environment.rb

config.after_initialize do
Workling::Remote.invoker = Workling::Remote::Invokers::EventmachineSubscriber
Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new
Workling::Remote.dispatcher.client = Workling::Clients::AmqpClient.new
end

Posteriormente en app/models/user_mailer.rb añadimos

include AsynchMail

A la fecha de hoy el plugin de workling tiene un pequeño issue, para resolverlo hacemos lo siguiente en el archivo lib/workling/clients/memcache_queue_client.rb y verificar que contiene:

require 'memcache'

Al principio del archivo, si no lo tienes agrégalo.

Como el amqp corre sobre un ciclo de EventMachine es necesario correr nuestra aplicación con Thin, ya que este servidor soporta EventMachine sin ningún problema. Para instalar Thin corremos

sudo gem install thin --no-rdoc --no-ri

Instalando RabbitMQ

Checar la instalación aquí http://www.jcastaneyra.com/2009/06/07/instalando-rabbitmq/

A correr todos

Nos aseguramos de que el RabbitMQ esté corriendo.

Iniciamos el cliente de workling

script/workling_client start

E iniciamos el servidor thin

thin start

Apuntamos nuestro browser a http://localhost:3000 y nos registramos con un usuario, al verificar el log de desarrollo veremos que el correo ahora es enviado después de todas las últimas operaciones registradas en el log. Podemos probar quitando la línea de include AsynchMail del UserMailer y ver lo que hace, o bien haciendo nuevo proceso que sea pesado y que se realice en background.

Referencias:

http://github.com/christospappas/workling/commit/063b9849b32e0c6140a5ecdb254357770d9bd28f
http://github.com/purzelrakete/workling/
http://github.com/langalex/workling_mailer/
http://github.com/tmm1/amqp/
http://github.com/technoweenie/restful-authentication/
http://www.rabbitmq.com
http://code.macournoyer.com/thin/
http://www.jcastaneyra.com/2009/06/07/instalando-rabbitmq/

Sending mails in asynchronous way in Rails using Workling, Workling-Mailer and RabbitMQ (Español)

Working with message queues is so interesting, since we can put processes in the background in order to be processed in asynchronous way, imagine this to send several mails, or maybe for heavy tasks as sms messages, report generation, pdf generation, etc.

So, with this post I want you to show how to send mails in asynchronous way, we are going to make a small application using Workling, Workling-mailer and as message queue RabbitMQ, actually we could adjust this solution to any other problem with async process requirement.

Base application

We create a simple application with restful_authentication and activation option.

First of all we create the rails app

rails mail_async_example -d mysql
 
rm public/index.html
 
script/plugin install git://github.com/technoweenie/restful-authentication.git
 
script/generate authenticated user sessions --include-activation

Now with rake we create and migrate DB

rake db:create
 
rake db:migrate

Following the restful_authentication instructions to send mail we add this:

config/routes.rb

map.activate 'activate/:activation_code', :controller => 'users', :action => 'activate', :activation_code => nil

config/environment.rb

config.active_record.observers = :user_observer

Our base app is working now, so to test it we get the url http://localhost:3000/signup in the browser.

Add this in app/controller/application_controller.rb:

include AuthenticatedSystem

The same line added to Application controller is removed from app/controller/sessions_controller.rb and app/controllers/users_controller.rb

We create a new controller for portal index

script/generate controller portal index

And modify config/routes.rb to add portal as root

map.root :controller => "portal"

And last we create a layout for our application:

app/views/layouts/application.html.erb

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
       "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
  <title>Contracts: <%= controller.action_name %></title>
  <%= stylesheet_link_tag 'scaffold' %>
</head>
<body>
<% if current_user %>
   <%= link_to 'Logout', logout_path %>
<% else %>
   <%= link_to 'Login', login_path %>
   <%= link_to 'Sign up', signup_path %>
<% end %>
 
<% [:error, :warning, :notice].each do |level|
    if flash[level] -%>
      <div class="flash <%= level.to_s -%>"><%= flash[level] -%></div>
  <% end -%>
<% end -%>
 
<%= yield %>
 
</body>
</html>

Installing amqp, workling and workling_mailer

Now, before using Workling, we need RabbitMQ and amqp is needed, this gem also requires EventMachine gem, so, both gems are installed when amqp is installed.

gem sources -a http://gems.github.com/
sudo gem install tmm1-amqp --no-rdoc --no-ri

List your installed gems

sudo gem list

Now, install workling and workling_mailer

script/plugin install git://github.com/purzelrakete/workling.git
 
script/plugin install git://github.com/langalex/workling_mailer.git

Add this to config/environment.rb to allow workling to use RabbitMQ

config.after_initialize do
Workling::Remote.invoker = Workling::Remote::Invokers::EventmachineSubscriber
Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new
Workling::Remote.dispatcher.client = Workling::Clients::AmqpClient.new
end

To app/models/user_mailer.rb add this

include AsynchMail

Until today I’ve seen a little issue with workling, to solve it we do the folloging in this file lib/workling/clients/memcache_queue_client.rb; verify that you have:

require 'memcache'

Add it if you don’t have it.

Amqp runs in an EventMachine cycle, so it is needed to run our app with Thin, since Thin supports EventMachine without any problem. To install Thin run

sudo gem install thin --no-rdoc --no-ri

Installing RabbitMQ

Follow the instructions here http://www.jcastaneyra.com/2009/06/07/instalando-rabbitmq/

Everybody runs

Cool, naw we ensure that RabbitMQ is running.

Start workling client

script/workling_client start

And start Thin server

thin start

Point with your browser to http://localhost:3000 and sign up a new user, then verify your log and will see that email is sent and registered to the end of log after all operations. We can test removing include AsynchMail from UserMailer and see what happens, it would be great if you test some application with a heavy process working in the background.

Links:

http://github.com/christospappas/workling/commit/063b9849b32e0c6140a5ecdb254357770d9bd28f
http://github.com/purzelrakete/workling/
http://github.com/langalex/workling_mailer/
http://github.com/tmm1/amqp/
http://github.com/technoweenie/restful-authentication/
http://www.rabbitmq.com
http://code.macournoyer.com/thin/
http://www.jcastaneyra.com/2009/06/07/instalando-rabbitmq/

, , , , , ,

View Comments

Instalando RabbitMQ

Posted by jcastaneyra in Howto/tutorial, OpenSource on June 7, 2009

Instalando RabbitMQ (English)

RabbitMQ es un sistema de message queue (MQ), el cual provee comunicaciones asíncronas, es decir que el productor y consumidor no tienen la necesidad de interactuar con los mensajes al mismo tiempo, además de que es una implementación del protocolo AMQP (Advanced Message Queuing Protocol), un protocolo para mensajeo de alto rendimiento, y por último decir que RabbitMQ está desarrollado con Erlang, Erlang es un lenguaje de programación funcional.

En Ubuntu 9.04

La instalación de RabbitMQ en Ubuntu 9.04 es tan simple como correr el comando

sudo aptitude install rabbitmq-server

Y con el cual si no has instalado Erlang te mostrará que paquetes tendrá que instalar para que lo tengas también.

jcastaneyra@ubuntu:~/sources$ sudo aptitude install rabbitmq-server
[sudo] password for jcastaneyra:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done
The following NEW packages will be installed:
erlang-base{a} erlang-nox{a} libltdl7{a} odbcinst1debian1{a} unixodbc{a}
The following partially installed packages will be configured:
rabbitmq-server
0 packages upgraded, 5 newly installed, 0 to remove and 8 not upgraded.
Need to get 28.6MB of archives. After unpacking 46.9MB will be used.
Do you want to continue? [Y/n/?]

Una vez instalado ya está corriendo.

En Mac leopard

UPDATE (27/05/2010): Desde hace un rato que ya estoy usando Homebrew para instalar cosas en Mac, ya existe una fórmula para instalar también RabbitMQ y por supuesto Erlang, para más información visitar http://github.com/mxcl/homebrew y seguir las instrucciones para instalar Homebrew. Con RabbitMQ, incluso si se quiere una versión específica de RabbitMQ se puede modificar la fórmula :)

La instalación es a través de los MacPorts, hay que bajar el instalador de la página www.macports.org/install.php, pero para poder compilar es necesario tener XCode instalado. Los MacPorts son instalados en /opt, por lo que hay que asegurarnos que se tengamos agregados los paths necesarios en el profile para ejecutar los comandos de MacPorts.

En mi caso en .profile (pero también puede ser el .bash_login) debe estar algo así (si no está lo agrego):

# MacPorts Installer addition on 2009-02-25_at_15:37:48: adding an appropriate PATH variable for use with MacPorts.
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
# Finished adapting your PATH environment variable for use with MacPorts.
 
# MacPorts Installer addition on 2009-02-25_at_15:37:48: adding an appropriate MANPATH variable for use with MacPorts.
export MANPATH=/opt/local/share/man:$MANPATH
# Finished adapting your MANPATH environment variable for use with MacPorts.

Una vez que están los MacPorts hay que instalar primero erlang, así que ejecutamos

sudo port install erlang

Y esperamos un buen rato a que se instale.

Una vez instalado erlang, bajamos la última versión de RabbitMQ

mkdir /tmp/rabbitmq &amp;&amp; cd /tmp/rabbitmq
curl -O http://www.rabbitmq.com/releases/rabbitmq-server/v1.5.5/rabbitmq-server-generic-unix-1.5.5.tar.gz
tar xvzf rabbitmq-server-generic-unix-1.5.5.tar.gz
sudo mv rabbitmq_server-1.5.5/ /opt/local/lib/erlang/lib

Ahora ya lo podemos ejecutar.

sudo /opt/local/lib/erlang/lib/rabbitmq_server-1.5.5/sbin/rabbitmq-server

Recursos

http://www.rabbitmq.com
http://playtype.net/past/2008/10/9/installing_rabbitmq_on_osx/

Installing RabbitMQ (Español)

RabbitMQ is a complete and highly reliable Enterprise Messaging system, it provides asynchronous communications,  meaning that the sender and receiver of the message do not need to interact with the message queue at the same time, also it is a AMQP implementation (Advanced Message Queuing Protocol) a protocol for high performance messaging, and last RabbitMQ is developed with Erlang, a functional programming language.

In Ubuntu 9.04

RabbitMQ instalation in Ubuntu 9.04 is so simple, it is just needed to run a command

sudo aptitude install rabbitmq-server

With this also install all packages needed by rabbitmq, Erlang is installed also.

jcastaneyra@ubuntu:~/sources$ sudo aptitude install rabbitmq-server
[sudo] password for jcastaneyra:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
Writing extended state information... Done
The following NEW packages will be installed:
erlang-base{a} erlang-nox{a} libltdl7{a} odbcinst1debian1{a} unixodbc{a}
The following partially installed packages will be configured:
rabbitmq-server
0 packages upgraded, 5 newly installed, 0 to remove and 8 not upgraded.
Need to get 28.6MB of archives. After unpacking 46.9MB will be used.
Do you want to continue? [Y/n/?]

Once the installation is finished RabbitMQ will be running.

In Mac leopard

UPDATE (27/05/2010): I am using Homebrew to install some stuff in Mac from some months ago, Homebrew has a formula to install RabbitMQ and also Erlang, for more info please visit http://github.com/mxcl/homebrew for install instructions. Also if you want to install a different version of RabbitMQ you can do it just editing the formula :)

RabbitMQ installation is done through MacPorts, you need to download the MacPorts installer from www.macports.org/install.php, also XCode is needed. MacPorts are installed in /opt, for that reason we put these paths in the .profile file (could be .bash_login), if you don’t have these paths add them to your profile file:

# MacPorts Installer addition on 2009-02-25_at_15:37:48: adding an appropriate PATH variable for use with MacPorts.
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
# Finished adapting your PATH environment variable for use with MacPorts.
 
# MacPorts Installer addition on 2009-02-25_at_15:37:48: adding an appropriate MANPATH variable for use with MacPorts.
export MANPATH=/opt/local/share/man:$MANPATH
# Finished adapting your MANPATH environment variable for use with MacPorts.

In order to install erlang execute the following command:

sudo port install erlang

You need to wait some time, maybe several minutes :)

Once you have erlang installed, we download the last RabbitMQ version

mkdir /tmp/rabbitmq &amp;&amp; cd /tmp/rabbitmq
curl -O http://www.rabbitmq.com/releases/rabbitmq-server/v1.5.5/rabbitmq-server-generic-unix-1.5.5.tar.gz
tar xvzf rabbitmq-server-generic-unix-1.5.5.tar.gz
sudo mv rabbitmq_server-1.5.5/ /opt/local/lib/erlang/lib

Now we execute rabbitmq-server with this

sudo /opt/local/lib/erlang/lib/rabbitmq_server-1.5.5/sbin/rabbitmq-server

Links

http://www.rabbitmq.com
http://playtype.net/past/2008/10/9/installing_rabbitmq_on_osx/

, , , ,

View Comments