Open.ControlTier > ModuleForge
 
 
PDF -icon
PDF

atg

Welcome to the "Elements" Java Server Module Library

The aim of this library is to provide a framework that automates the build from source code and reference data, and promotion of updates from one environment to another for server-side Java applications (principally J2EE compliant application server based applications).

The library contains the means to build applications from source code, package build artifacts such as modules and J2EE enterprise archives, and deploy these updates to target environments consisting of both web and application server tiers. This automation also exports data from a source database schema, packages it, deploys and imports it to target database schemas using one standardized end-to-end process.

Over-arching both of these goals the library provides process automation for the coordination of both code and data deployments including integrating continuous integration.

What is it?

This is a library of command modules that build on the ControlTier automation base types and establish an end-to-end data build and deployment process. This process is defined in terms of command workflows. The figure below describes three primary workflows:

  1. Coordinated Build and Deploy: This workflow defines the combined end-to-end code module and reference data management process and a single command to execute it called "BuildAndDeploy".
  2. Code and Data Build: This workflow is responsible for code module builds and exporting data from the master database, packaging them registering them with the repository.
  3. Code and Data Deploy: This workflow manages the distribution and installation of the application code and reference data packages, installing modules importing the export files.

Why use it?

This library builds on the ControlTier service provisioning platform to offer a number of notable features for organizations where data migration and promotion is an important part of the application lifecycle and is a frequent and critical process:

  • End-to-end automation: This framework ties together many individual procedures that are often run independently. Building on the ControlTier base types, this Library offers a single command that: builds code, exports data, packages and versions them, manages package dependencies and deploys to multiple hosts, installing the application and importing data from environment to environment.
  • Rollback: All distributions are packaged and versioned. You can rollback to any previous version.
  • Task delegation and self-service: You end up with schedulable "jobs" letting a less knowledgeable person run the process on demand or at defined periods.
  • Standardized operational interface: End users can rely on simple and predictable commands no matter what environment, database platform, schema or application.
  • Reporting: Every deployment is logged. Reports show who deployed what where and when. The full output of the job is saved can can be used for auditing later.
  • Graphical interfaces: All operational tasks can be done via a web-based graphical interface. ControlTier's JobCenter is used to run commands while Workbench can be used to review the current deployment environment dependencies.
  • Security: Using the ControlTier access control infrastructure, you can control who updates what when and where.
  • Extendability: The library offers a working end to end process but it is not monolithic. You can override various parts of the process via sub-typing.

Getting started

You can start using the this library by following the steps of the documentation pages listed below:

  1. Install: Describes how to download and install the library
  2. Configure: Describes how to configure the library to define new "BuildAndDeploy" jobs
  3. Run: Explains how to run the job either via the graphical JobCenter application or by command line.

Next: Install →

Install

Install

Overview

This document describes the installation steps necessary to use the library.

Step #0: Prepare for installation

This module library has been tested to support a broad range of platform technologies and application topologies, far beyond the scope of this documentation to cover comprehensively. In general, you can assume that the library supports all major release of the Apache Tomcat servlet container and the JBoss application server deployed both Linux/Unix and Windows servers. Additionally, a number of common database servers (e.g. MsSQL and Hypersonic SQL) are supported.

In order to document the journey to a working solution from a "standing start", the following assumptions are made:

  • The goal is to deploy Sun's Duke's Bank J2EE sample application (from their J2EE 1.4 Tutorial) to a single box.
  • Duke's Bank is to be built and deployed to the JBoss application server running on either a Windows or Linux system.
  • The Duke's Bank data will be stored in an stand-alone Hypersonic SQL database system (external to JBoss).

Specifically, then, for this demonstration of the module library the following software versions are required:

  • JBoss 4.0.3sp1
  • Hypersonic SQL Database 1.8.0.9
  • Ant 1.7.0
  • Cruise Control 2.7.1
  • CentOS release 4.5 (Final), or Redhat Enterprise Linux 4 update 2 or better, or ...
  • Windows XP Professional SP2, or Windows 2003 Server R2 (and possibly other Windows versions though they haven't been tested)

In addition, the availability of the following tools in your general environment is assumed:

  • JDK 1.5.0
  • Subversion 1.1.4, or later
Note
Assuming you wish to configure the Duke's Bank demonstration, it is assumed that the machine(s) you choose to install on have access to the Internet to checkout demonstration source using Subversion from Sourceforge.
Note
In order to run the entire set of services of the Duke's Bank demonstration on a single box (or virtual machine instance), you'll need around 2GB of disk space and 1024MB of memory. A three to six box configuration with separate server, build, and development and staging application server and database server systems can be run with virtual machines assigned 256-512MB of memory each and at 500MB of free disk space (1.5GB on the ControlTier server).

Step #1: Install the ControlTier platform software

This library assumes you have installed the latest stable release of the ControlTier 3.1 platform software on a a designated server host and one or more client hosts. Refer to the general installation procedures for more info.

You will need to know the URL to the Workbench and JobCenter applications.

Note
The module library has been tested using JDK 1.5.0 with the ControlTier framework
Note
If you plan a multi-box installation be sure up update the "server.tomcat.hostname" and "client.hostname" properties in the "default.properties" command-line installer configuration file to appropriate network host names (these are also settable options in the GUI installer).

Step #2: Choose or create a project

All work is done within the context of a "project". You may already have a project in mind, or you may wish to create a new one just for the use of this library.

Note
If you are new to ControlTier and are not sure about projects see the Projects section in the ProjectBuilder tutorial.
  1. Navigate to the Admin page. (eg., go to the URL: http://localhost:8080/itnav/do/menu/Admin
  2. Press the "Create Project" button. (eg., go to the URL: http://localhost:8080/itnav/do/projects/Input
    Note
    If you are following this documentation verbatim in order to configure the Duke's Bank demonstration, create a project called "DukesBank" (JBoss 4.0 "Duke's Bank" sample application project).
  3. Fill out the form and press "Create" button. It takes a few minutes for the new project to be created.
Note
If you are building a multi-box enviromnment, this is the moment to register your client nodes with the new project using "depot-setup"

Step #3: Download the library archive

Binary distributions of the module library can be found in the "File Releases" section of the ModuleForge Download page on Sourceforge. The package is called "Elements Module Library" and will be named something like: elements-x.y-seed.jar where "x.y" denotes the version.

You must download at least version 2.0 of the module library.

Step #4: Load the library archive

Once you have chosen the desired project, you can load the library into that project.

Be sure you have already logged into Workbench and selected the desired project where you want the library loaded. If you just created a new project, you are all ready.

  1. Navigate to the Admin page. (eg., go to the URL: http://localhost:8080/itnav/do/menu/Admin
  2. Press the "Import Seed" button. (eg., go to the URL: http://localhost:8080/itnav/do/projects/ImportSeedInput
  3. Locate and select the content-seed-x.y.jar file in the file chooser. This is the same file you downloaded in step #2.
  4. Press the "Import" button.
Note
It takes a few minutes for Workbench to load the library.

Once the library has been installed into your chosen project the next step is to configure these modules for use.

Next: Configure →

Configure

Configure

Overview

This document describes how to configure a project to use the content library.

The diagram below illustrates the configuration is driven by a defaults.xml file. This file is used as input by a command generate-objects which in conjunction with template files, produces two output files: object.xml and job.xml.

Note
Be sure you have already installed the ControlTier software, chosen a project and loaded the library archive. See the Install page for instructions.

Step #1: Edit defaults.xml

The defaults.xml file contains all the essential environment-specific information needed by the library. It answers questions like: What box is the ControlTier server? Which box is used for the development environment? Which for staging?

Open a text editor or better yet an XML editor. Cut and paste the contents of the (Linux specific) XML shown below (Windows equivalent here) and save it to disk You can use the example verbatim if you're looking to setup the Duke's Bank demonstration on the same box you installed the ControlTier server on.

<?xml version="1.0"?>

<!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
<!-- Defaults data for ElementsProjectBuilder generate-objects project XML document.       -->
<!--                                                                                       -->
<!-- Defines a set of objects that implements the Duke's Bank sample application           -->
<!-- development and staging environments deployed to one or more systems.                 -->
<!--                                                                                       -->
<!-- The target platform is JBoss 4.0 and HSQLDB database.                                 -->
<!--                                                                                       -->
<!-- Defines deployment and setting type objects as they are first used, and subsequently  -->
<!-- refers to objects by type and name whenever they are needed again.                    -->
<!--                                                                                       -->
<!-- This file can be supplied to the ProjectBuilder "generate-objects" command to load    -->
<!-- sample objects into a project that contains the "content" library's type model.       -->
<!--                                                                                       -->
<!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->

<defaults>
        <default>
                <!-- The default node is the framework node of the Antdepo client invoking the -->
                <!-- generate-objects command. (For the default installation this will be      -->
                <!-- "localhost").                                                             -->
                <node>${framework.node}</node>
        </default>

        <!-- The default location of the installation directory for the Duke's bank    -->
        <!-- application source files and packages:                                    -->
        <dukesbankroot>${user.home}/dukesbank</dukesbankroot>

        <!-- The out-of-the box Duke's Bank demonstration can be configured to run on  -->
        <!-- from one to six systems. One system must be designated the ControlTier    -->
        <!-- server. This is the system where Workbench and Jobcenter run. Another     -->
        <!-- system is designated as the build box where CruiseControl and the Ant     -->
        <!-- based build are run.                                                      -->
        <!--                                                                           -->
        <!-- One or two boxes are assigned to the development environment. Builds are  -->
        <!-- run here, and the development JBoss instance and "source" HSQLDB database -->
        <!-- instance are deployed here. A fifth and possibly sixth system host the    -->
        <!-- staging environment which includes both JBoss and HSQLDB instances.       -->
        <!--                                                                           -->
        <!-- Note that all these systems can be the same box in which case separate    -->
        <!-- JBoss and HSQLDB server instances are started on separate ports.          -->

        <server>
                <node>${defaults.default.node}</node>
        </server>
        <development>
                <buildserver>
                        <!-- The default copy of the JBoss 4.0 modified Duke's Bank source code is kept under Subversion on Sourceforge: -->
                        <scmconnection>https://moduleforge.svn.sourceforge.net/svnroot/moduleforge/elements/branches/2.0/demo/DukesBank/src</scmconnection>
                        <node>${defaults.default.node}</node>
                </buildserver>
                <applicationserver>
                        <node>${defaults.default.node}</node>
                </applicationserver>
                <databaseserver>
                        <node>${defaults.default.node}</node>
                </databaseserver>
        </development>
        <staging>
                <applicationserver>
                        <node>${defaults.default.node}</node>
                </applicationserver>
                <databaseserver>
                        <node>${defaults.default.node}</node>
                </databaseserver>
        </staging>
</defaults>

You are at liberty to change any of the six box names to systems that make sense in your environment, thereby ending up with the sample Duke's Bank development and staging environments deployed to as many as six and as few as a single system.

Note
A copy of the defaults.xml template file can be found in the WebDAV under your project (substituting a host name for "localhost" if the ControlTier server is deployed remotely): "http://localhost:8080/webdav/project/modules/ElementsProjectBuilder/templates/defaults.xml" ... where project is "DukesBank" if you are following the demonstration setup.

Step #2: Configure library objects

Register and install an ElementsProjectBuilder object:

ad -p project -m Deployment -c Register -- \
      -name name -type  ElementsProjectBuilder \
      -basedir $CTIER_ROOT/src/project -installroot $CTIER_ROOT/target/project \
      -install

... or, specifically for the Duke's Bank demonstration:

$ ad -p DukesBank -m Deployment -c Register -- -name elements -type  ElementsProjectBuilder -basedir $CTIER_ROOT/src/elements -installroot $CTIER_ROOT/target/elements -install
Checking for existing object, (ElementsProjectBuilder) elements, in project, 'DukesBank'...
Registered new object.
.
.
.
[command.timer.Deployment.Register: 3.096 sec]

C:\>ad -p DukesBank -m Deployment -c Register -- -name elements -type ProjectBuilder -basedir %CTIER_ROOT%\src\elements -installroot %CTIER_ROOT%\target\elements -install
.
.
.
[command.timer.Deployment.Register: 2.359 sec]

Copy the defaults.xml you created in Step #1 to $CTIER_ROOT/src/project/defaults.xml and run the generate-objects command:

ad -p project -t ElementsProjectBuilder -o name -c generate-objects -- \
      -name aName \
      -defaults $CTIER_ROOT/src/project/defaults.xml -upload

... or, specifically for the Duke's Bank demonstration (using different defaults XML files for Linux and Windows):

$ ad -p DukesBank -t ElementsProjectBuilder -o elements -c generate-objects -- -defaults $ANTDEPO_BASE/depots/DukesBank/lib/ant/modules/ElementsProjectBuilder/templates/defaults.linux.xml -upload
.
.
.
[command.timer.generate-objects: 0.156 sec]
generate-objects completed. execution time: 0.156 sec.

After this command successfully completes, a new set of objects will be loaded into the ControlTier repository. You can view them via ElementsProjectBuilder's find-objects command: 

ad -p project -t ElementsProjectBuilder -o name -c find-objects -- \
      -name aName

Before you can run any commands, it is necessary to deploy the objects. This is done via the AntDepo command, depot-setup. Run the following installation command on all boxes in your configuration: 

depot-setup -p project -a install

... or, if you're following the Duke's Bank demonstration setup instructions, more specifically (for a single box example):

$ depot-setup -p DukesBank -a install
"Install" command running for object: (Site) developmentCI
"Install" command running for object: (HsqldbRdb) staging
"Install" command running for object: (Updater) development
"Install" command running for object: (HsqldbRdb) development
"Install" command running for object: (Site) developmentApplicationServer
"Install" command running for object: (ElementsProjectBuilder) elements
"Install" command running for object: (Site) stagingDatabaseServer
"Install" command running for object: (CruiseControl) development
"Install" command running for object: (AntBuilder) development
"Install" command running for object: (Site) stagingApplicationServer
"Install" command running for object: (Site) developmentDatabaseServer
"Install" command running for object: (HsqldbRdbExportBuilder) development
"Install" command running for object: (Updater) staging
"Install" command running for object: (JBossServer) development
"Install" command running for object: (JBossServer) staging

Step #3: Upload job definition

The generate-objects command run in Step #2 will have produced a job.xml file with a filename aName-job.xml. This file can be used to define a new job in the JobCenter application.

  1. Login to JobCenter (e.g, go to URL: http://localhost:9090/jobcenter/menu/index
  2. Press the "Create a new Job..." button
  3. Press the "Upload job.xml" button
  4. Locate and select the file, aName-job.xml, produced by generate-objects in the file chooser
  5. Press "Save" button

The new job will be listed on the home page of JobCenter.

Step #4: Upload packages to the repository

The solution library also manages all the platform (3rd party) software packages needed to establish environments. The only assumptions are that you have a compatible OS image at your disposal, a user account with the ControlTier client installed and available, and sufficient disk space to deploy the platform and application software.

Note
The Duke's Bank demonstration assumes you have a system with Java and Subversion pre-installed.

The general method of adding 3rd party packages to the ControlTier repository is by uploading them using Workbench:

If your intention is to run the Duke's Bank demonstration and you have generated and configured the example library objects mentioned above, then upload the following packages into the ControlTier repository via Workbench (i.e. select package objects and upload one at a time):

Next: Run →

Run

Run

Overview

This section describes how to prepare for and run a code module and data BuildAndDeploy workflow job or command and is pertinent to users responsible for releasing content changes from the source code repository and refernce database to targeted deployment environments.

Their are two interfaces available to execute the BuildAndDeploy commands:

  • Using Jobcenter, the web-based graphical application,
  • Using AntDepo's ad shell command

Instructions for running the BuildAndDeploy command using either interface are explained below.

Preparing to run the Duke's Bank demonstration

In general, there are some one-off package installation and builder/service configuration commands to be run before normal ("day-to-day" build and deployment) operations can commence. In particular (regarding the Duke's Bank demonstration that has been installed and configured in the previous tabs of this site) execute the following commands from the Antdepo client designated as the server system:

Note
While instructions for these one-off commands are given in terms of using the Antdepo shell command it is quite possible to configure these (and any other) commands to execute from Jobcenter too.
  • Checkout the Duke's Bank application source code to the build box:
    Note
    By default the source is checked out from Sourceforge's Subversion repository. If you would like to be able to make updates, transfer the source into your own repository and modify the development AntBuilder's BuilderScmConnection resource setting via ProjectBuilder or Workbench.
    $ ad -p DukesBank -t AntBuilder -o development -c scmCheckout
scmCheckout parameters: {basedir="/home/anthony/dukesbank/development/src", connection="https://moduleforge.svn.sourceforge.net/svnroot/moduleforge/elements/branches/2.0/demo/DukesBank/src", module="",  label="" }
.
.
.
Checked out revision 635.
[command.timer.Builder.scmCheckout: 1:22.116 sec]
  • Install and configure the Cruise Control based continuous integration environment by issuing the following command from your ControlTier server box: 
    $ ad -p DukesBank -t Site -o developmentCI -c Deploy
Start: "Run the coordinated deployment cycle across the configured Sites." commands: dispatchCmd
begin workflow command (1/1) -> "dispatchCmd -command Deploy -resourcename .* -resourcetype [^\.]*" ...
dispatching command: "Deploy " to:  (CruiseControl) development ...
dispatching to object: development [CruiseControl] -> "Deploy "
.
.
.
command completed successfully. Execution time: 1:12.122 sec
dispatched command: Deploy completed for:  (CruiseControl) development
end workflow command (1/1) -> "dispatchCmd -command Deploy -resourcename .* -resourcetype [^\.]*"
[command.timer: 1:13.990 sec]
Completed: execution time: 1:13.990 sec

    The CruiseControl instance should be available from build box at http://localhost:8081 or similar (depending on your node setup). In addition you'll find a "boot.log" and "cruisecontrol.log" in the CruiseControl installation directory.

    Note
    As soon as CruiseControl comes up for the first time it will kick-off an initial build and put the resultant package into the ControlTier server's package repository.
    Note
    Deploying CruiseControl also installs the package dependencies (JBoss and Ant) required by the AntBuilder object.

  • As a general rule, you can find an object's configuration properties (for example, in this case, the Cruisecontrol installation directory - ccDir) as follows:

    $ ad -p DukesBank -t CruiseControl -o development -c Properties 
[MULTI_LINE]
# development [CruiseControl] #

Duke's Bank continuous integration server

## Attributes ##

*  basedir: "/home/anthony/dukesbank/development/src"
*  ccDir: "/home/anthony/dukesbank/development/cruisecontrol-bin-2.7.1"
*  interval: "60"
*  javaHome: "${env.JAVA_HOME}"
*  jmxport: "8001"
*  path: "/bin:/usr/bin"
*  rmiport: "1100"
*  targetdir: "/home/anthony/dukesbank/development/src/j2eetutorial14/examples/bank"
*  webport: "8081"
.
.
.
  • Install and configure the development and staging JBoss and HSQLDB database instances executing these commands from the ControlTier server node: 
    $ ad -p DukesBank -t Updater -o development -c Deploy
Start: "Run the coordinated deployment cycle across the configured Sites." commands: dispatchCmd
begin workflow command (1/1) -> "dispatchCmd -command Deploy -resourcename .* -resourcetype [^\.]*Site" ...
dispatching command: "Deploy " to:  (Site) developmentDatabaseServer, (Site) developmentApplicationServer ...
.
.
.
Completed: execution time: 33.946 sec
dispatched command: Deploy completed for:  (Site) developmentDatabaseServer, (Site) developmentApplicationServer
end workflow command (1/1) -> "dispatchCmd -command Deploy -resourcename .* -resourcetype [^\.]*Site"
[command.timer: 33.946 sec]
Completed: execution time: 33.946 sec
				
    $ ad -p DukesBank -t Updater -o staging -c Deploy
Start: "Run the coordinated deployment cycle across the configured Sites." commands: dispatchCmd
begin workflow command (1/1) -> "dispatchCmd -command Deploy -resourcename .* -resourcetype [^\.]*Site" ...
dispatching command: "Deploy " to:  (Site) stagingDatabaseServer, (Site) stagingApplicationServer ...
.
.
.
Completed: execution time: 15.775 sec
dispatched command: Deploy completed for:  (Site) stagingDatabaseServer, (Site) stagingApplicationServer
end workflow command (1/1) -> "dispatchCmd -command Deploy -resourcename .* -resourcetype [^\.]*Site"
[command.timer: 15.775 sec]
Completed: execution time: 15.775 sec

    The initial development and staging JBoss instances are available from the application server box(es) at http://localhost:8180 and http://localhost:8280, or similar (depending on your node setup). You'll find the JBoss "boot.log" and "server.log" under the "default" server instance in the JBoss installation directory for each environment. e.g:

    $ cd ~/dukesbank/development/jboss-4.0.3SP1/server/default/log
ls *.log
boot.log  server.log
    Note
    The JBossServer type uses the "spawn" attribute of the Exec Ant task. As a result, the console output of the JBoss "run.sh" is always lost.

To recap, execution of these deployment commands completes configuration of:

  • The build box hosting both the CruiseControl continuous integration service and the ControlTier AntBuilder.
  • The development application server box hosting JBoss.
  • The development database server box hosting HSQLDB.
  • The staging application server box hosting JBoss.
  • The staging database server box hosting HSQLDB.

Along with the server box hosting the ControlTier server (Workbench, WebDAV & Jobcenter) you may have deployed software to as many as six boxes (or as few as one).

Note
By this stage you will have seen what may appear to be several redundant installations of the same software be executed (Java, ATG, JBoss). These are necessary in order to ensure each of the application "service" instance deployments can operate completely independantly of one another.

Execute the application and data build and deployment via Jobcenter

In Configure Step #3, a new job will have been defined for the BuildAndDeploy process. After logging into Jobcenter a list of the defined jobs will be displayed. Choosing and running a job will execute the underlying BuildAndDeploy command.

The general steps to using Jobcenter to operate code and data BuildAndDeploy and Update are listed:

  1. Login to Jobcenter
  2. Identify the desired job
  3. Run the job
  4. Customize a report

Considering the specific set of jobs loaded to Jobcenter as part of the Duke's Bank demonstration:

  • Locate and select the "development.BuildAndDeploy" job and use the "Choose Options and Run Job ..." button to set the "buildstamp" option and "Run Job Now" to run the application code build and deployment process against the development environment:
    Note
    The "buildstamp" is equivalent to the "release version" of the application. You can choose whatever scheme appeals to you to ensure that each build is uniquely identified.
  • You can follow the job's progress using the "Tail Output" view in Jobcenter
  • Once the job has succeeded there will be two packages (JBossEar and HsqldbRdbDmp) in the package repository, and the development environment will running this new build and database dump.
  • Similarly, using the same buildstamp, find the "developmentCatalog.BuildAndDeploy" job and follow the same process to acquire a database dump and reload it to the development environment.
  • Lastly, find the "staging.Update" job and follow the same process to update both the staging application server and database instances with the selected buildstamp.

At this stage the development environment is up and running the Duke's Bank sample application: http://localhost:8180/bank/main, and the staging version of the application is at http://localhost:8280/bank/main, or similar (depending on your node setup). You can login to either site using the username "200" and the password "j2ee".

Execute via 'ad'

An alternative to executing the these commands via Jobcenter is to execute them directly via the ad shell command. The general usage is shown below: 

$ ad -p project -t Updater -o name -c BuildAndDeploy -- \
      -buildstamp buildstamp

A typical convention is to use the date and time as the -buildstamp argument. For example: 

$ ad -p project -t Updater -o name -c BuildAndDeploy -- \
      -buildstamp 200711071500

If you are an experienced AntDepo user, you may also know how to run individual parts of the build and update workflow by running the appropriate command from one of the subordinate commands. For example, to run just the Deploy: 

$ ad -p project -t Updater -o name -c Deploy

Or to run just the build, you can execute the Build workflow separately: 

$ ad -p project -t Updater -o name -c Build -- \
      -buildstamp buildstamp

Reference

Reference

Overview

This section is useful to developers interested how the library works and users interested in knowing all the command syntax offered by the modules in this library.

This library builds on the standard ControlTier "process building blocks" - Package, Builder, Updater, Site and Deployment - each a separate workflow that compose into the single end-to-end workflow command, BuildAndDeploy.

General Command Workflow

The sequence diagram below describes the workflow structure defined by the modules in the library.

Type Model

The diagram below describes the inheritance and composition hierarchies defined by the core types in the library.

All the attributes and commands are defined in the Type Reference section of the documentation. There you will find a document page for each type.

Next: Type Reference →

Types

Type Reference

All