Thursday, November 11, 2010

JBoss AS7 OSGi Integration

Now that JBossAS-7.0.0.Alpha1 has come out I'd like to talk about how JBoss OSGi integrates with it and what benefits our community will gain from the OSGi integration. In this post I'll show how non-OSGi components like EJB3, Servlet, CDI Beans, ManagedBeans, MBeans, etc. can access OSGi services and vice versa.

OSGi in the EE Landscape

Generally I believe that OSGi is very much an all-or-nothing technology. If your application is an OSGi bundle (or many of them) you must run within an OSGi Framework and can only have access to services that are provided by other OSGi bundles. In todays EE appplication landscape where folks take (standard) functionality like JPA, Servlet3, CDI, EJB3, JCA, JMS, etc for granted they would need a container that offers these services not only to EE applications but also to OSGi bundles.

In Mar-2010 the OSGi Enterprise Expert Group, has come up with its first EE OSGi specification, which IMHO offers a fraction of what we currently see in modern EE applications. There is no JSP, EJB3, JCA, JMS, CDI not even Annotation processing. The EEG is working on completing the picture, but this is unlikely to become generally available in the near future. David and I are members of this group, so its partially also our call.

Of course, if your application is such that it only requires what is available in OSGi Core, Compendium and Enterprise already (and many of todays larger OSGi applications do) OSGi is a good candidate to solve general modularity and service integration issues.

The JBoss approach to OSGi

At JBoss we take a different approach. With the upcoming AS7 the basic building block is a "module" which can have dependencies on other modules. There are services with basic lifecycle that can also have dependencies on each other. Our OSGi Framework implementation builds on top of this basic infrastructure like any other JBoss subsystem. We integrate at the lowest possible level. Such it becomes possible to access an OSGi service from a non-OSGi component and vice versa.

The questions about the suitability of OSGi is at the very heart of these integration issues IMHO. If EJB3, CDI, etc. offer value-add then it must be possible to leverage that value from an OSGi application. Otherwise you have an either/or situation where you gain benefits from one technology but at the same time loose benefits from another technology that cannot be accessed any more.

You can think of this in terms of two intersecting sets of technology (i.e. EE6, OSGi). I'm interested in the superset of the two and not so much in the smaller individual sets. The aim is that JBoss users who already write great and non trivial EE applications can now "also" use OSGi to address modularity issues in these very applications.

The Theory behind JBoss OSGi

Any deployment, and in-fact any jar you see in AS7, is a Module. A Module may have dependencies and a number of attached ResourceLoaders. A ResourceLoader is typically backed by a VirtualFile. A Module can choose to export a subset of the resource paths from its attached ResourceLoaders. This is equivalent to the Export-Package notion in OSGi. The dependencies on other Modules are equivalent to Require-Bundle. A module can choose which paths to import from a given dependency. A Module can choose to re-export the paths that it imports from a dependency.

An OSGi Bundle is a jar with a set of metadata in it's Manifest. To name a few, we have
Generally speaking however, these are Requirements and Capabilities that a Bundle may have. In JBoss OSGi we translate the OSGi metadata to these general Requirements and Capabilities and feed the Standalone Resolver with it. At bundle resolution time, the Resolver tries to find a consistent set of Wires, such that every non-optional Requirement is wired to a Capability. This is a non-trivial process that leads to consistent class spaces. If successful, the Bundle is said to be RESOLVED.

At the end of the day, every OSGi Framework conceptually creates dependencies between the installed bundles and limits the set of paths at the exporting and importing side of the wires.

So we get the set of Requirements and Capabilities from somewhere, feed them to our resolver abstraction and finally use the resulting Wires to setup the ModuleSpec. The source for our Requirements and Capabilities is not limited to the OSGi manifest. Any AS7 deployment can generate resolver metadata and make itself known to the OSGi layer.

At deploy time every Bundle becomes an AS7 Module and every AS7 Module may become an OSGi Bundle when its provides it's set of resolver metadata and registers with the OSGi layer. OSGi bundles are therefore a subset of the modules that exist in the AS7 module layer.

Lets look at some code ...

Bundle accessing a Non-OSGi Module

In a test case we construct and deploy a module that provides a simple Echo service to the running AS7.

JavaArchive archive = ShrinkWrap.create(JavaArchive.class, "example-xservice-target-module");
archive.addClasses(Echo.class, EchoService.class, TargetModuleActivator.class);
String activatorPath = "META-INF/services/" + ServiceActivator.class.getName();
archive.addResource(getResourceFile("xservice/target-module/" + activatorPath), activatorPath);

The module contains a few classes and a ServiceActivator. The ServiceActivator is the equivalent of a BundleActivator. When the module gets activated we register a simple EchoService.

public static void addService(BatchBuilder batchBuilder)
{
  serviceBuilder = batchBuilder.addService(SERVICE_NAME, new EchoService());
  serviceBuilder.addAliases(ServiceName.of(Constants.JBOSGI_PREFIX, Echo.class.getName()));
  serviceBuilder.setInitialMode(Mode.ACTIVE);
  log.infof("Service added: %s", SERVICE_NAME);
}

Please note, that the service is also registered with an alias. The JBoss OSGi service registry uses the ServiceContainer that contains all services that exist in AS7. For service lookup the OSGi API uses the FQN of an interface that the requested service implements. The alias is needed for this OSGi API call to succeed

ServiceReference sref = context.getServiceReference(Echo.class.getName());
Echo service = (Echo)context.getService(sref);
service.echo("hello world");

When the module is deployed, it is not automatically registered with the OSGi layer. A bundle that imports the package of the Echo interface would not resolve.

Any module can however be registered with the OSGi layer.

ModuleIdentifier moduleId = ModuleIdentifier.create("deployment.example-xservice-target-module");
registerModuleWithBundleManager(moduleId);

Currently this must be done explicitly - there is no automatic registration. In our case the module does not provide resolver metadata explicitly, so it will be generated. We add a PackageCapability for every exported path.

ResolverPlugin resolverPlugin = getPlugin(ResolverPlugin.class);
XModuleBuilder builder = resolverPlugin.getModuleBuilder();
builder.createModule(symbolicName, version, 0);
builder.addBundleCapability(symbolicName, version);
for (String path : module.getExportedPaths())
{
  if (path.startsWith("META-INF"))
    continue;

  String packageName = path.replace('/', '.');
  builder.addPackageCapability(packageName, null, null);
}
XModule resModule = builder.getModule();

Now that the module is known to the OSGi layer. The resolver can use the module's capabilities for bundle resolution. We can now install and start a real OSGi bundle that imports the echo package and calls the Echo service in its BundleActivator.

mvn -Dtarget.container=jboss70x -Dtest=BundleAccessesModuleServiceTestCase test

Running org.jboss.test.osgi.example.xservice.BundleAccessesModuleServiceTestCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.009 sec

17:21:16,469 INFO  Activating deployment: example-xservice-target-module
17:21:16,732 INFO  Service added: service jboss.osgi.xservice.target
17:21:16,733 INFO  ModuleIdentifier: module:deployment.example-xservice-target-module:main
17:21:16,854 INFO  Install bundle: deployment.example-xservice-target-module:0.0.0
17:21:18,135 INFO  Install bundle: example-xservice-client-bundle:1.0.0
17:21:18,196 INFO  Echo: hello world
17:21:18,199 INFO  Bundle started: example-xservice-client-bundle:1.0.0
17:21:18,222 INFO  Bundle uninstalled: example-xservice-client-bundle:1.0.0
17:21:18,235 INFO  Bundle uninstalled: deployment.example-xservice-target-module:0.0.0
17:21:18,292 INFO  Undeployed example-xservice-target-module 

Going forward, we can now define a set of rules for every possible AS7 deployment on whether or not it gets registered with the OSGi layer and how it generates it's caps/reqs.

A possible question could be:
"Why isn't the target module a real OSGi bundle in the first place? In other words, why does it not define its capabilities in a standard way in the OSGi manifest"
The target module may have dependencies on other services that are not available in the OSGi layer. Or simply because the folks that provide the target module do not feel sufficiently familiar with OSGi. The important thing here is that with AS7 we have a migration path to and from OSGi.

Module accessing an OSGi Service

This scenario requires a little more thought. Let's first explore the seemingly obvious approach.
  • The target bundle contains the Echo interface
  • The client module has a dependency on the target bundle
Every Module has a ModuleIdentifier, which in case of an OSGi bundle is constructed according to the following pattern
jbosgi.[bundle-symbolic-name]:[bundle-version]
In case of the target bundle that is used in this test case, we would have
jbosgi.example-xservice-target-bundle:1.0.0
This is the Require-Bundle semantic, which is generally considered bad practise in OSGi. In the OSGi core spec we find the following chapter

3.12.3 Issues With Requiring Bundles

The preferred way of wiring bundles is to use the Import-Package and
Export-Package headers because they couple the importer and exporter to a
much lesser extent. Bundles can be refactored to have a different package
composition without causing other bundles to fail.

The Require-Bundle header provides a way for a bundle to bind to all the
exports of another bundle, regardless of what those exports are. Though this
can seem convenient at first, it has a number of drawbacks:

Split Packages – Classes from the same package can come from different bundles with Require bundle, such a package is called a split package. Split packages have the following drawbacks:
  • Completeness – Split packages are open ended, it is difficult to guarantee that all the intended pieces of a split package have actually been included.
  • Ordering – If the same classes are present in more than one required bundle, then the ordering of Require-Bundle is significant. A wrong ordering can cause hard to trace errors, similar to the traditional class path model of Java.
  • Performance – A class must be searched in all providers when packages are split. This potentially increases the number of times that a ClassNotFoundException must be thrown which can potentially introduce a significant overhead.
  • Confusing – It is easy to find a setup where there is lots of potential for confusion. For example, the following setup is non-intuitive.

    A: Export-Package: p;uses:=q
       Import-Package: q
    B: Export-Package: q
    C: Export-Package: q
    D: Require-Bundle: B, C
       Import-Package: p
    


    In this example, bundle D merges the split package q from bundles B and
    bundle C, however, importing package p from bundle A puts a uses constraint on package p for package q. This implies that bundle D can see the valid package q from bundle B but also the invalid package q from bundle C. This wiring is allowed because in almost all cases there will be no problem. However, the consistency can be violated in the rare case when package C.q contains classes that are also in package B.q.

Mutable Exports – The feature of visibility:=reexport that the export signature of the requiring bundle can unexpectedly change depending on the export signature of the required bundle.

Shadowing – The classes in the requiring bundle that are shadowed by those in a required bundle depend on the export signature of the required bundle and the classes the required bundle contains. (By contrast, Import-Package, except with resolution:=optional, shadows whole packages regardless of the exporter.)

Good practise is to use Export-Package to declare a PackageCapability and Import-Package to declare a PackageRequirement. In this way importer does not need to care where a package comes from and the provider of the exported package is free to move the package around from one bundle to another without breaking the importer.

OSGi supports the notion of bundle revisions. This does not show up in the public API. However, every time a bundle is updated (i.e. the bytes change) the framework creates a new revision with potentially the same Bundle-SymbolicName and Bundle-Version. Internally we append the revision suffix to the ModuleIdentifier. As a consequence, binding to a particular ModuleIdentifier identifier ultimately means binding to a particular revision, which negates the purpose update.


Having said all this, we decide against  putting the Echo interface (i.e. the API)  in the target bundle. Instead we put it in a Module which can be registered with the OSGi layer as described above.

JavaArchive getAPIModuleArchive() throws Exception
{
  JavaArchive archive = ShrinkWrap.create(JavaArchive.class, "example-xservice-api-module");
  archive.addClasses(Echo.class);
  archive.addDirectory("META-INF");
  return archive;
}

// Deploy the module that contains the API
String apiDeploymentName = getRemoteRuntime().deploy(getAPIModuleArchive());

// Register the API module with the OSGi layer
ModuleIdentifier apiModuleId = ModuleIdentifier.create("deployment." + apiDeploymentName);
registerModuleWithBundleManager(apiModuleId);

The target bundle is a plain OSGi bundle that imports the package of the Echo service and registers an instance of that service with the OSGi service registry.

@Override
   public void start(final BundleContext context) throws Exception
   {
      context.registerService(Echo.class.getName(), new EchoImpl(), null);
   }

   static class EchoImpl implements Echo
   {
      @Override
      public String echo(String message)
      {
         log.infof("Echo: %s", message);
         return message;
      }
   }

The client module needs to define a dependency on the API module. In JBoss AS module dependencies can be defined in the manifest like this

Manifest-Version: 1.0
Dependencies: org.osgi.core,deployment.example-xservice-api-module

Ouch, this uses an AS7 proprietary manifest header as well as Require-Bundle semantics. For AS7 internal modules (i.e. the ones that we control) this direct and hard coded approach is key to bootstrap performance. It should however be carefully considered before putting in user deployments.

Never mind, for sake of this exercise we continue. When the client module activates it registers a Service.

public class EchoInvokerService implements Service<Void>
{
   InjectedValue<BundleContext> injectedBundleContext = new InjectedValue<BundleContext>();

   static void addService(BatchBuilder batchBuilder)
   {
      EchoInvokerService service = new EchoInvokerService();
      BatchServiceBuilder serviceBuilder = batchBuilder.addService(SERVICE_NAME, service);
      serviceBuilder.addDependency(ServiceName.parse("jboss.osgi.context"), BundleContext.class, service.injectedBundleContext);
      serviceBuilder.setInitialMode(Mode.ACTIVE);
      log.infof("Service added: %s", SERVICE_NAME);
   }

   @Override
   public void start(StartContext context) throws StartException
   {
      BundleContext systemContext = injectedBundleContext.getValue();
      ServiceReference sref = systemContext.getServiceReference(Echo.class.getName());
      Echo service = (Echo)systemContext.getService(sref);
      service.echo("hello world");
   }

   @Override
   public void stop(StopContext context)
   {
   }

   @Override
   public Void getValue() throws IllegalStateException
   {
      return null;
   }
}

This is a native AS7 service that gets the OSGi system BundleContext injected before it is started. Currently, when you download and try jboss-7.0.0.Alpha1 you will see that it bootstraps in very little time (i.e. less than 3sec). This is also because the OSGi subsystem is activated lazily. Installing a service like the one above with initial mode ACTIVE would cause all services that it depends on also to become ACTIVE. The OSGi subsystem would activate and the framework would start.

In the start method of the EchoInvokerService you see the usage of OSGi API that calls the target service.

mvn -Dtarget.container=jboss70x -Dtest=ModuleAccessesBundleServiceTestCase test

Running org.jboss.test.osgi.example.xservice.ModuleAccessesBundleServiceTestCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.009 sec

11:10:22,040 INFO  Activating deployment: example-xservice-api-module
11:10:22,228 INFO  Install bundle: deployment.example-xservice-api-module:0.0.0
11:10:23,140 INFO  Install bundle: example-xservice-target-bundle:1.0.0
11:10:23,251 INFO  ModuleIdentifier: module:jbosgi.example-xservice-target-bundle:1.0.0
11:10:23,252 INFO  Bundle started: example-xservice-target-bundle:1.0.0
11:10:23,413 INFO  Activating deployment: example-xservice-client-module
11:10:23,516 INFO  Service added: service jboss.osgi.xservice.invoker
11:10:23,518 INFO  ModuleIdentifier: module:deployment.example-xservice-client-module:main
11:10:23,520 INFO  Echo: hello world
11:10:23,602 INFO  Undeployed example-xservice-client-module
11:10:23,661 INFO  Bundle uninstalled: example-xservice-target-bundle:1.0.0
11:10:23,687 INFO  Bundle uninstalled: deployment.example-xservice-api-module:0.0.0
11:10:23,739 INFO  Undeployed example-xservice-api-module

Conclusion

I have shown how in principle an OSGi bundle can access an arbitrary AS7 service and vice versa. With this distribution we have reached about 80% OSGi Core TCK compliance, which we hope to complete by Q2/2011. The final version of AS7 is currently scheduled for May/2011.

I am hopeful that by the time AS7 goes final we have examples of various EE components interacting with OSGi and vice versa. The aim is that you can use the benefits of OSGi modularity and its service layer in non-trivial applications that make use of AS7 middle ware.

You may also want to check out David's post on "Using OSGi in JBoss AS7".

May this be useful
-thomas

Tuesday, October 26, 2010

JBossOSGi Hudson on Amazon Elastic Cloud

Following Amazons announcement for an AWS Free Usage Tier, I thought I would give it a try. Here my account.

Creating the Instance

If you are new to Amazon EC2 you need to create an account and sign in. Once you are in the AWS Management Console, you can create a key pair like this




I like to have control over the connection ports that are accessible on my instance, so I create a new Security Group, which I can later use with my instance.


Next, you navigate to Instances and launch an new Amazon Linux Micro instance. Basic 32-bit Amazon Linux is a good choice because it can be upgraded to small. A 64-bit instance can only be upgraded to large and above.


Select the key and security group that you created earlier, hit launch and your Linux Micro instance should be up and running.

Accessing you Linux Instance

When you created your keypair you were offered to download and save the private key to a secure location. You can now ssh into you newly created instance like this

chmod 600 .ssh/tdiesler-ec2.pem
ssh -i .ssh/tdiesler-ec2.pem ec2-user@ec2-50-16-7-151.compute-1.amazonaws.com
Last login: Mon Oct 25 13:32:07 2010 from p5b04e10a.dip.t-dialin.net

       __|  __|_  )  Amazon Linux AMI
       _|  (     /     Beta
      ___|\___|___|

See /etc/image-release-notes for latest release notes. :-)
[ec2-user@ip-10-112-99-96 ~]$ 

Installing Hudson on Tomcat

Amazon Liunx supports the Yum Package Manager, which I happen to know quite well because I run Fedora. Lets first check if Tomcat is there

$ yum list installed | grep tomcat
$ 

Ok, that's not the case, so see what's available.

$ yum list available | grep tomcat
...
tomcat6.noarch                               6.0.24-8.12.amzn1 
tomcat6-admin-webapps.noarch                 6.0.24-8.12.amzn1 
tomcat6-docs-webapp.noarch                   6.0.24-8.12.amzn1 
tomcat6-javadoc.noarch                       6.0.24-8.12.amzn1 
tomcat6-log4j.noarch                         6.0.24-8.12.amzn1 
tomcat6-webapps.noarch                       6.0.24-8.12.amzn1 

Let's install tomcat6

[ec2-user@ip-10-112-99-96 ~]$ 
$ sudo yum install tomcat6
...
Installed:
  tomcat6.noarch 0:6.0.24-8.12.amzn1
...
Complete!

Furthermore, we need Git, Ant and Maven. Git and Ant we can install with yum like above. Maven however does not seem to be available in yum. So lets get it from the Apache Maven download page.

$ wget http://apache.linux-mirror.org/maven/binaries/apache-maven-3.0-bin.zip
...
2010-10-25 14:40:23 (44.4 KB/s) - “apache-maven-3.0-bin.zip” saved [2919967/2919967]
$ unzip apache-maven-3.0-bin.zip
$ sudo mkdir /usr/java
$ sudo mv apache-maven-3.0 /usr/java
$ sudo ln -s /usr/java/apache-maven-3.0 /usr/java/apache-maven
$ mkdir bin
$ ln -s /usr/java/apache-maven/bin/mvn bin/mvn
$ mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.108s
[INFO] Finished at: Mon Oct 25 14:44:26 UTC 2010
[INFO] Final Memory: 2M/7M
[INFO] ------------------------------------------------------------------------

Amazon Linux comes with openjdk, which is fine for most purposes.  At the time of this writing however, I saw issues with the maven build and Hudson Captcha. So lets install the Oracle JDK 

$ cd /usr/java
$ sudo wget http://cds.sun.com/is-bi/...jdk-6u22-linux-i586.bin
$ sudo chmod +x jdk-6u22-linux-i586.bin 
$ sudo ./jdk-6u22-linux-i586.bin 
$ sudo ln -s jdk1.6.0_22 jdk1.6
$ sudo rm jdk-6u22-linux-i586.bin

$ vi ~/.bash_profile
...
export JAVA_HOME="/usr/java/jdk1.6"
export PATH=$HOME/bin:$JAVA_HOME/bin:$PATH

$ java -version
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Java HotSpot(TM) Client VM (build 17.1-b03, mixed mode, sharing)

 
Building JBoss OSGi

The JBoss OSGi umbrella project that holds the Hudson setup as an integral part of the code base is hosted on GitHub.

$ mkdir git; cd git
$ git clone git://github.com/jbosgi/jbosgi.git; cd jbosgi
$ git submodule init
$ git submodule update

JBoss community projects often do not contain the JBoss Nexus Configuration. Follow the instructions on Maven Getting Started - Users to setup your ~/.m2/settings.xml

Now, lets build the umbrella for the first time

$ mvn install
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] JBossOSGi JMX ..................................... SUCCESS [1:00.822s]
[INFO] JBossOSGi JMX API ................................. SUCCESS [1:44.154s]
[INFO] JBossOSGi VFS ..................................... SUCCESS [0.008s]
[INFO] JBossOSGi VFS API ................................. SUCCESS [5.442s]
[INFO] JBossOSGi SPI ..................................... SUCCESS [24.504s]
[INFO] JBossOSGi JMX Bundle .............................. SUCCESS [15.142s]
[INFO] JBossOSGi JMX iTests .............................. SUCCESS [2:37.160s]
[INFO] JBossOSGi VFS VFS30 ............................... SUCCESS [9.390s]
[INFO] JBossOSGi Deployment .............................. SUCCESS [12.598s]
[INFO] JBossOSGi Framework ............................... SUCCESS [0.047s]
[INFO] JBossOSGi Framework Core .......................... SUCCESS [32.110s]
[INFO] JBossOSGi Framework iTest ......................... SUCCESS [2:33.963s]
[INFO] JBossOSGi ......................................... SUCCESS [0.026s]
[INFO] JBossOSGi Reactor ................................. SUCCESS [0.006s]
[INFO] JBossOSGi Testsuite ............................... SUCCESS [0.017s]
[INFO] JBossOSGi Testsuite Examples ...................... SUCCESS [3:39.319s]
[INFO] JBossOSGi Testsuite Functional .................... SUCCESS [1:40.045s]
[INFO] JBossOSGi Testsuite Performance ................... SUCCESS [19.179s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:48.812s
[INFO] Finished at: Tue Oct 26 12:30:18 UTC 2010
[INFO] Final Memory: 22M/68M
[INFO] ------------------------------------------------------------------------

This will take a while, so lets continue with Tomcat and Hudson.

Installing Hudson

In our security group we opened port 8280. Tomcat however runs on 8080 by default. Lets change that.

$ sudo vi /etc/tomcat6/server.xml 
...
    <connector port="8280" connectiontimeout="20000" 
        protocol="HTTP/1.1" redirectport="8443">

Now we configure tomcat to run as the ec2-user, which allows Hudson to reuse the ec2-user's maven repository. HUDSON-HOME can also be set in the same config file.

$ sudo vi /etc/tomcat6/tomcat6.conf
...
# Where your java installation lives
JAVA_HOME="/usr/java/jdk1.6"

# Where HUDSON has its home
HUDSON_HOME=/usr/share/tomcat6/workspace/hudson-home

# What user should run tomcat
TOMCAT_USER=ec2-user

Make sure the ec2-user is in the 'tomcat' group so it can write files to HUDSON_HOME during setup.

$ sudo usermod -g tomcat ec2-user
$ mkdir -p /usr/share/tomcat6/workspace/hudson-home

You might need to re-login for the initial login group change to take effect.

JBossOSGi simplifies Hudson setup a great deal, by including configuration and jobs in it codebase. This also allows us to recreate the QA environment for any given version by running a simple Ant task.

$ cd git/jbosgi/hudson
$ cp ant.properties.example ant.properties
$ ant hudson-setup
...
init-hudson:
     [echo] hudson.home = /usr/share/tomcat6/workspace/hudson-home
...
get-hudson:
      [get] Getting: http://hudson-ci.org/download/war/1.382/hudson.war
      [get] Getting: http://hudson-ci.org/download/plugins/github/0.2/github.hpi
      [get] Getting: http://hudson-ci.org/download/plugins/git/1.1/git.hpi
...
hudson-tomcat-setup:
     [copy] Copying 1 file to /usr/share/tomcat6/workspace/hudson-home/plugins
     [copy] Copying 1 file to /usr/share/tomcat6/workspace/hudson-home/plugins

hudson-setup:
     [copy] Copying 2 files to /usr/share/tomcat6/workspace/hudson-home
     [copy] Copying 9 files to /usr/share/tomcat6/workspace/hudson-home/jobs
     [echo] 
     [echo] *************************************
     [echo] * Hudson setup successfully          
     [echo] * sudo service tomcat6 restart       
     [echo] *************************************

$ sudo service tomcat6 restart
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]

Now we should be able to connect to the running Hudson instance.


Hudson Security


By default, Hudson is setup with strict project based security. Only the 'ec2-user' or whoever you configured in ${hudson.admin} can modify the configuration and run the jobs. Lets sign-up as ec2-user


You should now be able to run the jobs.

Upgrading the Instance Type

The Amazon free offer is limited to Micro instances. With it's 613MB of RAM not surprisingly it turns out that this is not sufficient to run the JBossOSGi QA in an acceptable time.

This is however not a problem, because the instance can be stopped and it's type can be changed to m1.small or m1.large depending on what you are willing to pay.

$ ec2-stop-instances i-10429a7d
$ ec2-modify-instance-attribute --instance-type=m1.large i-10429a7d
$ ec2-start-instances i-10429a7d

Note, that this can only be done with instances that are backed by the Elastic Block Store.

Creating an Elastic IP

Every time you stop/start an EC2 instance it's public DNS entry changes. Amazon's solution to this is the Elastic IP. You pay for the IP when your instance is not connected.



This IP address can be associated with your favorite domain name. In our case its a dyndns.org forward. So what was formerly known as


now points to the EC2 instance that we've just setup.


May this be useful
-thomas

Monday, September 27, 2010

JBossOSGi 1.0.0.Beta9 released

I am happy to announce the release of JBossOSGi-1.0.0.Beta9.
 
You can download the binary here: jboss-osgi-installer-1.0.0.Beta9.jar

The release comes with improvements in the following areas
  • Uses the new JBoss Modules modularity layer, which is also the basis of AS7
  • Performance increase by approximately 2000%
  • Integration with upcoming JBoss Application Server 7
  • Support for update bundles and refresh packages
  • Improved resolver abstraction/integration
  • Migration to the Arquillian test framework
For details please have a look at the latest version of our User Guide.

Here are the change log details

Bug
  • [JBOSGI-331] - Activator cannot change bundle start level
  • [JBOSGI-339] - Fragment failures in functional runtime tests
  • [JBOSGI-346] - Attached fragment hides private package in host
  • [JBOSGI-354] - Initial Bundle Start Level initialization is done in the wrong place.
  • [JBOSGI-356] - JBoss OSGi has implicit imports
  • [JBOSGI-367] - JTA BundleActivator cannot find DocumentBuilderFactory
  • [JBOSGI-373] - Cannot load service through java.util.ServiceLoader
  • [JBOSGI-383] - Cannot refresh uninstalled bundle through FrameworkMBean
  • [JBOSGI-387] - Potential problem with refreshing uninstalled bundles via JMX

Feature Request

  • [JBOSGI-209] - Implement resolver preferences
  • [JBOSGI-376] - Initial OSGi / Modules dependency implementation
  • [JBOSGI-380] - Initial Arquillian test integration
  • [JBOSGI-386] - Invoking Refresh Packages on the framework from Web Console causes issues

Task

  • [JBOSGI-258] - [TCK] Start Level
  • [JBOSGI-336] - Implement PackageAdmin.refreshPackages(Bundle[])
  • [JBOSGI-343] - Comprehensive PackageAdmin test coverage
  • [JBOSGI-359] - Comprehensive Bundle API test coverage
  • [JBOSGI-360] - Integrate jbosgi-container into jbosgi umbrella
  • [JBOSGI-361] - TCK setup for MSC Core Framework
  • [JBOSGI-362] - Integrate the MSC Framework with the Installer
  • [JBOSGI-363] - Update resolver to Apache Felix-3.0.x
  • [JBOSGI-369] - Add support for fragments to Package Admin
  • [JBOSGI-374] - Release 1.0.0.Beta9
  • [JBOSGI-375] - Restore fragment, native code, service hook support
  • [JBOSGI-378] - OSGi Performance Analysis
  • [JBOSGI-379] - Implement Bundle.update()
  • [JBOSGI-388] - Migrate all remaining Husky tests to Arquillian

Sub-task

Enjoy 

    Tuesday, July 27, 2010

    Standalone OSGi Resolver

    When designing complex software system, modularity in one of the key aspects that folks have in mind. When I say "complex" I mean more than a handful of subsystems that evolve with individual life cycles and are developed by possibly geographically disconnected teams. Key to success are well defined integration points that also have the flexibility to evolve.

    Naturally you separate API from implementation and have some way or another to discover/load the implementation for a given API. If your chosen infrastructure is worth telling your friends about, it supports multiple versions of any API and can dynamically start/stop/update the services that your subsystems use to collaborate with each other. The installed pieces form consistent class spaces with the same class possibly loaded multiple times in different versions and wired such that shared types are loaded from the same source. Any container environment that supports user defined plugins would have requirements similar to these. Take Eclipse, or a J2EE application server for example. 

    As my colleague Hal puts it
    "OSGi is like toilet paper, sooner or later you are going to want to use it."

    Capabilities and Requirements 

    An OSGi bundle is an ordinary jar with a couple of standard headers in its Manifest. Here I list a few important ones as a reference. The exhaustive set can be found in the OSGi Core Specification.

    Bundle-SymbolicName: com.acme.daffy
    The Bundle-SymbolicName header specifies a non-localizable name for this bundle. The bundle symbolic name together with a version must identify a unique bundle.
    Bundle-Version: 1.1
    The Bundle-Version header specifies the version of this bundle
    Export-Package: org.osgi.util.tracker;version=1.3
    The Export-Package header contains a declaration of exported packages.
    Import-Package: org.osgi.util.tracker,org.osgi.service.io;version=1.4
    The Import-Package header declares the imported packages for this bundle.
    Require-Bundle: com.acme.chess
    The Require-Bundle header specifies that all exported packages from another bundle must be imported, effectively requiring the public interface of another bundle.
    In an abstract way, the bundle headers form a set of Capabilities that a bundle provides and another set of Requirements that a bundle has on the environment or other bundles. In most cases Requirements can be mandatory or optional and in some cases Requirements can be satisfied by Capabilities from the same bundle.

    When the Framework makes a connection between a Requirement and a Capability it is said to have established a Wire. This happens at the RESOLVE phase of a bundle. The details of how an OSGi Framework works out the wires between the bundles is not our subject. Instead it is sufficient to know that every OSGi Framework implements this non-trivial, non-linear OSGi resolution algorithm in one way or another. Some Frameworks do this more successful than others and a distinguishing metric would be the success/error response time and ability to form consistent class spaces that include if not all but most of the installed bundles.

    Why use a standalone Resolver

    For most use cases the OSGi resolution algorithm is an implementation detail at the Framework level that you don't need to worry about. You install a few Bundles, start them or trigger a class load on an installed bundle and the Framework resolves the installed bundles transparently. Although there is an API method on the PackageAdmin to resolve bundles explicitly it is not what you would do normally.

    The need for a resolver comes into the game when you want to do impact analysis or provide a smart repository. Impact analysis is when you have a running system and want to know if a set of bundles would resolve on that system without actually modifying the system. With a smart repository you can say, if have these requirements, give me the bundles that I need to install to provide the needed capabilities.

    The Apache Felix Resolver

    Every other month, when we both can make it, I meet Richard at the Enterprise Expert Group (EEG) meetings. Richard rewrote Felix resolver to work on the general notion of capabilities and requirements. This is part of felix-framework-3.0.1 and forms the foundation of what comes next.

    Standalone Resolver Requirements

    We had a number of requirements on an OSGi Resolver that I'd like to share with you here:
    • Independent of Core Framework implementation
    • Not bound to the Manifest as the meta data provider
    • Works on abstract notion of Module, Capability, Requirement, Wire
    • Explicit wiring for all mandatory requirements
    • Easy and intuitive user API
    • Extensible SPI for Resolver providers
    • Sensible default implementation stubs for Resolver providers
    • Callbacks for framework integration and resolved modules
    • Attachment support on various resolver atrifacts
    Every framework parses/validates and caches the Manifest. So we did not want the resolver to do this again. Instead we provide a resolver meta data model that can be constructed in memory (i.e. mapped to from whatever the framework uses internally). Additionally, we provide an OSGi meta data model that can also be constructed in memory or from the Manifest.

    In the JBoss Application Server we provide an OSGi view for non-OSGi deployments and the ability to access/inject OSGi services in other component models. For example CDI components can have OSGi services injected and vice versa. You will see this functionality in AS7. For that reason the resolver needs to work on the abstract notion of Module, Capability, Requirement which may come from non-OSGi deployments that nevertheless take part in OSGi resolution.

    If a module is resolved, it is guaranteed that every mandatory Requirement has a Wire to a Capability.

    Resolver clients can register callbacks to get informed when the Resolver changes the wiring for a Module.

    The Resolver object model can have arbitrary state attached to it, which removes the need to map client state to resolver state and vice versa.

    Working with the Resolver

    The jbosgi-resolver project is hosted on GitHub together with all the other JBoss OSGi projects that we currently work on.  When you use Maven, you can declare a dependency on the resolver like this:

    <dependency>
      <groupId>org.jboss.osgi.resolver</groupId>
      <artifactId>jbosgi-resolver-api</artifactId>
      <version>1.0.0</version>
    </dependency>
    <dependency>
      <groupId>org.jboss.osgi.resolver</groupId>
      <artifactId>jbosgi-resolver-felix</artifactId>
      <version>3.0.1</version>
    </dependency>
    
    <repository>
      <id>jboss-public-repository-group</id>
      <name>JBoss Public Maven Repository Group</name>
      <url>https://repository.jboss.org/nexus/content/groups/public/</url>
      <layout>default</layout>
      <releases>
        <enabled>true</enabled>
        <updatePolicy>never</updatePolicy>
      </releases>
      <snapshots>
        <enabled>true</enabled>
        <updatePolicy>never</updatePolicy>
      </snapshots>
    </repository>
    

    Lets start by creating a Resolver instance and add a Module to it that has its Capabilities and Requirements populated from Manifest.

    // Create a resolver instance
    XResolver resolver = XResolverFactory.getResolver();
    
    // Create the resolver module
    XModuleBuilder builder = XResolverFactory.getModuleBuilder();
    XModule module = builder.createModule(1, manifest);
    
    // Add the module to the resolver
    resolver.addModule(module); 
    

    Now you would create and add a few more modules until you actually trigger the resolve process

    // Resolve the modules and report resolver errors
    Set resolved = resolver.resolveAll(unresolved);
    for (XModule resModule : unresolved)
    {
       if (resModule.isResolved() == false)
       {
          Exception ex = resModule.getAttachment(XResolverException.class);
          log.error("Cannot resolve: " + resModule, ex);
       }
    } 
    

    The resolver uses the above mentioned Attachement API to feed back the individual resolve exceptions. You can also register a ResolverCallback when you need to do locking or want to get informed about resolver progress.

    A few changes were necessary to felix-framework-3.0.1 to abstract resolver dependencies on actual framework implementation. These can be picked up here.

    May this be useful

    Monday, June 28, 2010

    JBossOSGi 1.0.0.Beta8 released

    I am happy to announce the release of JBossOSGi-1.0.0.Beta8.

    You can download the binary here: jboss-osgi-installer-1.0.0.Beta8.jar

    The release comes with improvements in the following areas
    • Integrate Apache Felix resolver
    • Added support for DynamicImport-Package
    • Implementation of the StartLevel Service
    • Remove Felix and Equinox from the Installer (JBoss Framework reached acceptance threshold)
    For details please have a look at the latest version of our User Guide.

    Here are the change log details

    Bug

    • [JBOSGI-151] - Cannot resolve circular dependencies
    • [JBOSGI-213] - Unexpected dependee state changes
    • [JBOSGI-317] - Context is already registered in domain DefaultDomain
    • [JBOSGI-318] - Linkage error for javax.servlet.Servlet in AS600
    • [JBOSGI-319] - LinkageError for the type javax/servlet/Servlet
    • [JBOSGI-323] - DynamicImport-Package takes presendence over embedded classes
    • [JBOSGI-324] - Cannot start EventAdmin service in AS
    • [JBOSGI-326] - MC service does not maintain properties
    • [JBOSGI-334] - Fix Start Level Support from the webconsole
    • [JBOSGI-338] - Cannot obtain EventAdmin service
    • [JBOSGI-341] - Endless loop at AS server startup
    • [JBOSGI-342] - Bundle resolution depends on install order
    • [JBOSGI-344] - When stopping the framework, bundles need to be taken down in reverse startlevel order
    • [JBOSGI-348] - EventAdmin unexpectedly unregistered in Runtime
    Feature Request
    • [JBOSGI-138] - Proper system BundleContext implementation
    • [JBOSGI-143] - Add support for DynamicImport-Package
    • [JBOSGI-184] - Allow for overlay of jboss-osgi-bootstrap.xml
    • [JBOSGI-311] - Provide access to org.osgi.jmx API in OSGiFrameworkTest
    • [JBOSGI-320] - Integrate Apache Felix resolver
    • [JBOSGI-332] - Enhance the StartLevel plugin to support specification of start level
    • [JBOSGI-333] - Make it possible to specify the initial start level of the system
    Task
    • [JBOSGI-94] - Remove DeploymentProvidedDeploymentUnitFilter
    • [JBOSGI-235] - Verify system classpath access
    • [JBOSGI-301] - Enable VFS3 leak detection in runtime
    • [JBOSGI-310] - Link generated javadoc to org.osgi API
    • [JBOSGI-314] - Change Hudson QA setup to work with Git
    • [JBOSGI-315] - Integrate jboss-osgi-http in distribution
    • [JBOSGI-316] - Verify integrity of the jboss-osgi-framework-all.jar
    • [JBOSGI-327] - Add support for JBossAS-6.0.0.M3
    • [JBOSGI-328] - Merge framework vfs30 support with core
    • [JBOSGI-335] - Ensure that start levels are supported via JMX
    • [JBOSGI-349] - Remove Felix and Equinox from the Installer
    • [JBOSGI-350] - Add OSGi Alliance trade mark to images in userguide
    • [JBOSGI-351] - Release jboss-osgi-1.0.0.Beta8
    • [JBOSGI-352] - Remove dependency on osgi deployers project
    • [JBOSGI-139] - Implement system BundleContext.loadClass()
    • [JBOSGI-278] - Revisit ControllerContext handling

    Enjoy

    Thursday, April 15, 2010

    JBossOSGi 1.0.0.Beta7 released

    I am happy to announce the release of JBossOSGi-1.0.0.Beta7.

    You can download the binary here: jboss-osgi-installer-1.0.0.Beta7.jar

    The release comes with improvements in the following areas

    • Enterprise OSGi bundle management (JMX)
    • JSR160 compliant JMXConnector
    • Integrate JBoss OSGi with AS 6.0.0.M3
    • Framework service hooks
    • OSGi core TCK compliance 40%

    For details please have a look at the latest version of our User Guide.

    Here are the change log details

    Bug

    • [JBOSGI-292] - PackageAdmin cannot obtain classloading metadata
    • [JBOSGI-295] - NullPointerException in ExportedPackageHelper constructor
    • [JBOSGI-297] - OSGiBootstrapBean spinning on System.in read
    • [JBOSGI-299] - WebConsole does not display on jbossas/trunk
    • [JBOSGI-300] - Felix integration breaks jbossas/trunk bootstrap

    Feature Request

    • [JBOSGI-126] - Replace the SPI JMX functionality by RFC-139 API
    • [JBOSGI-307] - JSR160 compliant JMXConnector
    • [JBOSGI-267] - Allow multiple bundles to change state at once

    Task

    Errata

    Due to the recent migration of the JBoss Maven Repositories to Nexus, you might need to make adjustments to your ~/.m2/settings.xml

    Enjoy



    Monday, February 8, 2010

    JBossOSGi 1.0.0.Beta6 Released

    I am happy to announce the release of JBossOSGi-1.0.0.Beta6.

    You can download the binary here: jboss-osgi-installer-1.0.0.Beta6.jar

    The release comes with improvements in the following areas
    • Initial support for fragments.
    • Initial support for Bundle.update().
    • Initial support for native code libraries.
    • Support for R3 bundles.
    • Support for bundle header localization.
    • Improved Blueprint support, using Apache Aries Blueprint.
    For details please have a look at the latest version of our User Guide.

    Here are the change log details

    Bug

    • [JBOSGI-210] - Bundle installed but not started with hot deploy
    • [JBOSGI-283] - WebApp extender not installed by default
    • [JBOSGI-145] - JAXB classes are loaded from the system classpath
    • [JBOSGI-204] - Failure in Bundle.start() uninstalls the bundle
    • [JBOSGI-286] - Investigate classloading of javax.* classes
    • [JBOSGI-287] - Optional import loaded from system classloader
    Feature Request
    • [JBOSGI-269] - Add initial support for Bundle.update()
    • [JBOSGI-271] - Initial support for R3 bundles
    • [JBOSGI-272] - Implement bundle header localization
    • [JBOSGI-273] - Differenciate R3/R4 bundle validation
    • [JBOSGI-274] - Cache bundle header for default locale
    • [JBOSGI-275] - Add support for Bundle.update() from original location
    • [JBOSGI-276] - Add initial support for native code libraries
    • [JBOSGI-277] - Add initial bundle fragment support
    Task
    • [JBOSGI-263] - Integrate Aries Blueprint implementation
    • [JBOSGI-265] - Update to jboss-osgi-framework-1.0.0.Alpha3
    • [JBOSGI-281] - Release JBoss OSGi 1.0.0.Beta6
    • [JBOSGI-264] - Update to jboss-kernel-2.2.x stable release
    • [JBOSGI-279] - Update to jboss-cl-2.2.x stable release
    • [JBOSGI-284] - Investigate version numbering scheme

    Enjoy

    Wednesday, January 27, 2010

    ServiceLoader and how it relates to OSGi

    An important aspect of good software architecture is that clients of your components can interact with well defined and stable APIs. The concept of interfaces is an inherent part of the Java programming language and also central to the OSGi Service Layer.

    Lets have a look at a simple service that I want to migrate from a plain Java programming model to OSGi.

    public interface AccountService
    {   
       /** Get the current balance */   
       long getBalance();   
    
       /** Credit the given amount and return the current balance */   
       long credit(long amount);   
    
       /** Withdraw the given amount and return the current balance */   
       long withdraw(long amount);
    }
    

    This interface would be part of the API jar (i.e. serviceloader-api.jar) that I package and distribute to the clients of my AccountService implementation. I have another jar that contains the implementation (i.e. serviceloader-impl.jar) and also a file in META-INF/services that names the implementation class



    The implementation of my AccountService would be in a different package than the interface, for example something like '...serviceloader.service.internal'. Clients would be encouraged never to import anything from '*.internal'

    A Java client could obtain an instance of the AccountService by using the well known ServiceLoader that is part of JDK1.6.

    ServiceLoader loader = ServiceLoader.load(AccountService.class);
    AccountService service = loader.iterator().next();
    

    So far so good, but there are a number of issues with this approach. The most obvious one is that of the potential availability of more than one implementation of the same interface. The ServiceLoader uses the current thread's context class loader to discover all available service implementations. It then uses the same class loader to load the service instances.

    The client gets presented with an iterator of available service implemenations, but what are the criteria does the client use to choose the specific implementation it really needs? We could add something like getVersion(), getVendor(), getSomeOtherProp() to our AccountService, which the client could then use to pick the implementation it needs, but this approach would pollute our AccountService with concerns that are not about accounting.

    Another potential problem is that the client could directly instantiate or cast the service instance to gain access to internal functionality, which bypasses the fundamentals of abstraction and modularity that motivated us to use a service interface in the first place.

    How does OSGi solve this problem?

    In OSGi, a bundles can define the transitive closure of the API that it wants to make publicly available. In case of our API bundle the OSGi manifest could look like this

    Bundle-ManifestVersion: 2
    Bundle-SymbolicName: example-serviceloader-api
    Export-Package: example.serviceloader.service;version="1.0"
    

    Please note, that we associate a specific version with the package that contains the service interfaces. This is the version of our public contract. The package is versioned, not the API bundle. This allows us to rethink and change our packaging strategy in the future.

    The bundle that contains the implementation could have a manifest like this

    Bundle-ManifestVersion: 2
    Bundle-SymbolicName: example-serviceloader-impl
    Bundle-Vendor: JBoss, a division of RedHat
    Bundle-Version: 1.0.0
    Import-Package: example.serviceloader.service;version="[1.0,1.1)"
    Private-Package: example.serviceloader.service.internal
    

    The implementation explicitly states that it is valid for API version greater and equal to 1.0 up until but not including 1.1. Please also note, that the '*.internal' package is declared private, which prevents any other bundle from seeing the contents of that package even though the individual classes may be public. A client running in an OSGi environment cannot instantiate the implementation directly, nor can it cast to it.

    The implementation bundle could register the AccountService with the framework in BundleActivator.start() like this

    Hashtable props = new Hashtable();
    props.put("service.vendor", "JBoss, a division of RedHat");
    AccountService service = new AccountServiceImpl();
    context.registerService(AccountService.class.getName(), service, props);
    

    Please note, that we associate some meta data with the service registration. In this case the service vendor, which could be used by a client to obtain a specific vendor implementation.

    The client bundle could statically obtain a service instance like this

    String filter = "(service.vendor=JBoss*)";
    String sname = AccountService.class.getName();
    ServiceReference[] srefs = context.getServiceReferences(sname, filter);
    AccountService service = (AccountService)context.getService(srefs[0]);
    

    Please note that the client explicitly defines a filter criteria for available services. In this case the specific service vendor.

    To account for the dynamic nature of OSGi services, the client would not want to make the assumption that the AccountService is already available when its BundleActivator.start() method is invoked by the framework. This assumption would create an implicit start order dependency of the implementation and client bundle (i.e. the implementationbundle must be started before the client bundle).

    If the the client can dynamically react to the AccountService coming and going it should use code like this

    String sname = AccountService.class.getName();
    String filterstr = "(&(objectClass=" + sname + ")(service.vendor=JBoss*))";
    Filter filter = FrameworkUtil.createFilter(filterstr);
    
    // Track the service
    ServiceTracker tracker = new ServiceTracker(context, filter, null)
    {   
       public Object addingService(ServiceReference reference)   
       {
          service = (AccountService)super.addingService(reference);
          return service;
       }
    
       public void removedService(ServiceReference reference, Object tracked)
       {
          super.removedService(reference, tracked);
          service = null;
       }
    };
    tracker.open();
    

    The dynamic nature of OSGi services is in fact its real beauty. The client gets informed when the service gets modified (i.e. a new version of the service from the same vendor becomes available)

    Auto registration of META-INF/services

    David Bosschaert, a friend and colleague recently brought the topic of META-INF/services and its prevalent nature in Java component architecture to my attention. Because it is such a common pattern, I investigated how an OSGi framework could reuse the information that is already present in META-INF/services when a non OSGi component library is wrapped or migrated to be used in an OSGi framework.

    Starting with JBoss OSGi 1.0.0.Beta6, which should come out early Feb-2010, we provide a jboss-osgi-serviceloader bundle. That registers a LifecycleInterceptor to introspect every installed bundle. Should a bundle contain META-INF/services, these services are automatically registered as ServiceFactories.

    The interceptor associates the following properties with every service it registers
    Have a look at the service loader example that demonstrates this functionality

    Running org.jboss.test.osgi.example.serviceloader.ServiceLoaderTestCase
    Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.415 sec
    

    Conclusion

    The OSGi service model has much to offer over the traditional Java model of interfaces and service factories. The ServiceLoader being a particular and well known implementation of the more general service factory pattern.

    In OSGi you can have multiple implementations of the same service available in the framework. Services can be associated with arbitrary meta data, which means you can version your public contract. Clients can use a meta data filter to select a particular implementation. Implementation, can be strictly separated from API with no back doors left open.

    Lastly a few items from my OSGi best practises catalogue
    • Export the public contract, not the implementation
    • Declare implementation details in Private-Package
    • Associate specific versions with your public API packages
    • Define dependencies on package versions, not bundle versions
    • Define which version range of the API a you implement
    • Separate the API bundle from the implementation bundle
    • Keep you public contract concise, well documented and simple
    • Components should interact with services, not arbitrary stuff you export
    • Only service interfaces and their associated types should be exported
    • Track services dynamically, instead of getting the service from the framework

    May this be useful