Saturday, March 27, 2010

Clustering A Grails App with Terracotta

When I was writing Grails: A Quick-Start Guide, I thought that scalability issues were too complicated and “out of scope” for a “getting started” book. Well, the folks at Terracotta have been busy at work making me wrong. Integrating Terracotta's Web Sessions clustering product is so easy that I could have covered it in a page or two. So, we'll do that now.

In this post, we'll install and configure Web Sessions Express. In subsequent posts we'll look at some of Terracotta's other products: EhCache, Quartz, and EhCache for Hibernate.

We'll start with installing Terracotta, which can be downloaded at http://www.terracotta.org/dl/oss-download-catalog. We'll grab the tar/zip file (terracotta-3.2.1.tar.gz) and expand it to an appropriate location (like /opt/terracotta-3.2.1/). Then we'll set that directory as our TERRACOTTA_HOME.

Included with the install is a JAR file that will be used by our server. The exact location where we copy this file varies by server. We'll be using Tomcat 6.0.24, so we'll follow the instructions for Tomcat 6.0x. Let's copy TERRACOTTA_HOME/sessions/terracotta-session-1.0.0.jar to CATALINA_HOME/lib. (For the correct location for other servers, see the Terracotta documentation.)

Now for our application, we'll use the version of TekDays from the final chapter (Deployment and Beyond) of GQuick. If you worked your way through the book, you know that this application is going to be so popular that we're going to have to deploy it on a whole bunch of servers. :-) If you didn't build the application, you can download it from http://pragpub.com/titles/dkgrails/code. Also, check out this post on upgrading and deploying TekDays for some helpful tips.

Next we need to add a very simple configuration file to our application. We'll create the file web-app/META-INF/context.xml and include the following bit of XML:

<Context>

<Valve className="
org.terracotta.session.TerracottaTomcat60x
SessionValve
"
tcConfigUrl="localhost:9510"/>

</Context>

I've suggested that the Terracotta team come up with a Groovy DSL for configuring their tools in Grails applications, and it seems that they're considering it. But in the meantime, take the necessary precautions when you're adding the XML, and be sure to take a break and bandage any wounds you received through handling those sharp edges.

Note that these instructions are for deploying to a Tomcat server in the 6.0x family. If you are using an older version of Tomcat or if for some strange reason <grin> you are using some other server, you can find the appropriate configuration code at the terracotta.org documentation page.

We're almost done. Right now we could start up the Terracotta server, start up our Tomcat server, and deploy a TekDays.war. And it would almost work. But in order for Terracotta to cluster an application, all of the artifacts that we want to be clustered need to implement serializable. So, we'll just open up some of our classes and add implements serializable to their class declaration. Here's a list of the files that we need to modify:

grails-app/domain/TekEvent.groovy

grails-app/domain/TekUser.groovy

grails-app/domain/Task.groovy

grails-app/domain/Message.groovy

grails-app/domain/Sponsor.groovy

grails-app/domain/Sponsorship.groovy

grails-app/conf/SecurityFilters.groovy


For example, we'll modify the TekEvent class like so:


class TekEvent implements Serializable {
//...

We'll do the same for each of those other files, and then we'll be ready to try it out. To run our newly terracottafied TekDays application, take the following steps:

1. Build a WAR file: from the TekDays application directory, run grails war tekdays.war

2. Copy tekdays.war to Tomcat's webapps folder: /webapps

3. Start the Terracotta server: /bin/start-tc-server.sh (or .bat if you're on Windows)

4. Start Tomcat: /bin/startup.sh (or .bat if you're on Windows)

5. Quick sanity check: navigate to http://localhost:8080/tekdays (or the port that your Tomcat instance is running on.)

6. Start the Terracotta Developer Console (this step isn't necessary to run the app, but it gives us a nice way to see that it's working): /bin/dev-console.sh (again .bat if you're on Windows)

The Developer Console will give us a screen like this:




When we click "Connect", we should get something like this:



Now this only shows one connected client right now, since that's all that we're running. To really see the magic, we need to fire up another Tomcat instance on a different port. But we'll leave that as an exercise for the reader.

There is one simple thing we can do to catch a glimpse of what Terracotta is doing for us. With TekDays running, navigate to a page – maybe create a new event and start adding some tasks. OK, now leave that page up, go to the console, and shut down the Tomcat server: /bin/shutdown.sh

Let the shutdown finish, and then restart Tomcat. Give that a minute to wake up, then go back to the page and continue what you were doing. Everything is just as you left it. The logged in user is even in the session! Pretty slick!

There are other steps for setting up multiple Terracotta servers and such, but as far as our application goes, that's all there is to it. If I ever get a chance to write a second edition of Grails: A Quick-Start Guide, there will definitely be a section on scalability. Because, as we all know, Grails Scales!

11 comments:

  1. Great post, Dave! Thank you.

    Can you provide a little more detail about which classes need to be made Serializable? All domain classes and Filters? Anything else?

    Thanks.

    ReplyDelete
  2. I think you should serialize the classes you put in your session object, that is all.

    ReplyDelete
  3. Yes, any objects that might end up in the session should have classes that implement Serializble. This will generally be domain classes and all filters plus any non-domain objects that you might place in the session. If you know certain domain classes will never be placed in the session then you can skip those. It doesn't appear that UrlMapping needs to be serializable.

    ReplyDelete
  4. Feeling rather frustrated - nothing is ever as simple as the blog posts for me :(

    I feel I've followed the instructions here but I'm getting hundreds of lines of errors on tomcat startup. the relevant lines seem to be

    Caused by: java.lang.NoClassDefFoundError: org/terracotta/express/ClientFactory
    at org.terracotta.session.BootStrap.bootstrap(BootStrap.java:27)
    at org.terracotta.session.ModernTomcatSessionValve.createRealValve(ModernTomcatSessionValve.java:33)
    at org.terracotta.session.BaseExpressSessionValve.setContainer(BaseExpressSessionValve.java:137)
    at org.apache.catalina.core.StandardPipeline.addValve(StandardPipeline.java:436)
    at org.apache.catalina.core.ContainerBase.addValve(ContainerBase.java:1217)
    ... 49 more
    Caused by: java.lang.ClassNotFoundException: org.terracotta.express.ClientFactory
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    ... 54 more
    Jul 26, 2010 9:31:02 AM org.apache.catalina.startup.ContextConfig processContextConfig
    SEVERE: Occurred at line 3 column 36

    and

    Jul 26, 2010 9:31:02 AM org.apache.catalina.startup.ContextConfig processContextConfig
    SEVERE: Parse error in context.xml for
    java.lang.reflect.InvocationTargetException
    at org.apache.tomcat.util.digester.Digester.createSAXException(Digester.java:2806)
    at org.apache.tomcat.util.digester.Digester.createSAXException(Digester.java:2832)
    at org.apache.tomcat.util.digester.Digester.endElement(Digester.java:1141)
    ....


    The _project_/web-app/META-INF/context.xml looks like this





    this is all with tomcat 6 (tomcat 7 didn't even pretend to work).

    ReplyDelete
  5. Hi,

    Thanks for the post. I am pretty new to the clustering thing.

    Just wonder when using terracotta, do you still need to create clusters in tomcat according to the Tomcat clustering doc: http://tomcat.apache.org/tomcat-6.0-doc/cluster-howto.html

    Many thanks

    ReplyDelete
  6. @Jianfeng Tian, No, you don't need to create Tomcat clusters when using Terracotta sessions clustering.

    ReplyDelete
  7. Hi Dave,

    Thanks for the reply.

    But following your post, I got the same exceptions as Michael Kimsal's.

    I am using grails 1.3.4.

    ReplyDelete
  8. Thank you Dave, great topic!

    I'm a little confused with grails-terracotta integration:
    I'm using distributed cache, as well as quartz.

    for the cache, I've just installed distributed-cache plugin, used the default configuration, and was able to run same app on 2 machines and have a communication between them (beautiful). I guess there is some kind of "embedded" cache, since I'm not running anything else.
    for quartz, on the other hand, I see that I need to run an external terracotta server.
    is it tru that the dist cache runs inside the container? (I don't see how else it could work)
    and if so, is there a way to use this cache for quartz, rather than start a different instance?

    ReplyDelete
  9. Michael Kimsal & Jianfeng Tian - To get rid of the execption (i.e. java.lang.ClassNotFoundException: org.terracotta.express.ClientFactory), you need to also copy terracotta-toolkit JAR from TERRACOTA_HOME/common to CATALINA_HOME/lib.

    See http://forums.terracotta.org/forums/posts/list/4543.page

    ReplyDelete
  10. I spent a bit of time with the exception and I have a solution. To get rid of the exception, remove the space between TerracottaTomcat60x and SessionValve in the context.xml. It should read: TerracottaTomcat60xSessionValve

    ReplyDelete
  11. There is a problem with copy+paste the context.xml code and throws an Exception. To solve this problem, get rid of the space between TerracottaTomcat60x and
    SessionValve. It should read: TerracottaTomcat60xSessionValve

    ReplyDelete