Saturday, March 24, 2007

Spring, JdbcTemplate and Transactions

Yet another item in my dirty-secrets category is the fact that so far, until about a week ago, I had never used database transactions with Spring and JDBC, although I have been using Spring for more than 3 years now. Before Spring, I did a lot of classic JDBC coding with the try..catch..finally pattern. My first application using Spring was to wrap another third party application which updated its MySQL databases using JDBC as well. In a few instances we would add hooks to insert/update additional data using JDBC. Subsequently I discovered JdbcTemplate, but I was now working on applications where the main focus was transaction speeds and it was acceptable to lose a few records here and there, so we used MySQL with the MyISAM storage engine (which does not support transactions) was chosen. To the readers who are cringing in horror, I am happy to report that we only had about half an hour of downtime in about 3 years of operation because of data corruption, and that half hour included figuring out why our servers were hung and switching to the replication server.

At my current job, we are using Spring with Oracle. So far I did not have to use transactions in my Spring JDBC code. There is a lot of code written in classic JDBC (using transactions) in the application, and most of the work I had done so far was reading that data, which did not need transactions. However, sometime early last week, I had to write the Data Access Object for a bean which consisted of one parent table and three child tables with foreign key dependencies on the parent. The save() and delete() methods of this DAO needed to be transactional since all the JDBC operations needed to succeed or fail as a unit.

I have used transactions with Spring and Hibernate in a small project (also on MySQL, but with the InnoDb storage engine) before. However, since Spring-Hibernate usage is so widely documented on the web, the effort did not involve much understanding. Spring-JDBC usage gets significantly less coverage, perhaps because it is not as popular. I began by applying the prescriptive examples I found on the web, but no matter what I tried, Oracle would complain about missing foreign keys and throw an exception. It ultimately turned out to be an application bug, but the problem led me to read up on transactions, and look at Spring code and find out how the TransactionTemplate is implemented, etc, so happily, I now understand a little more about Spring and J2EE transactions than I did before.

Setting up the Transaction Manager

No matter what approach (Declarative or Programmatic) you take with making your Spring-JDBC code transactional, you will need a Transaction Manager. Spring provides a DataSourceTransactionManager which you can use if your application speaks to a single database, otherwise you may need to choose the appropriate JTA Transaction Manager implementation. Regardless, switching out the Transaction Manager just involves changing the bean definition below. Here is the definition:

1
2
3
4
5
6
7
8
9
  <!-- you should already have one of these -->
  <bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    ...
  </bean>

  <!-- this should be added for transaction support -->
  <bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
     <property name="dataSource" ref="dataSource"/>
  </bean>

The Programmatic Approach

My initial implementation was using the programmatic transaction approach using a TransactionTemplate to wrap the code that needs to be in a transaction. To do this, my DAO bean declared a dependency on the TransactionManager bean defined above.

1
2
3
4
  <bean id="myDao" class="com.mycompany.daos.MyDao">
    <property name="dataSource" ref="dataSource"/>
    <property name="transactionManager" ref="transactionManager"/>
  </bean>

Our DAO code has methods to select various attributes based on various input parameters (named getXXX() and findXXX() by convention) and a save() and delete() method that inserts and deletes the bean from the database. The additional things a transactional DAO must have is the reference and setter for the Transaction Manager, code to build a Transaction Template from the Transaction Manager, and a TransactionCallback to wrap the transactional bit of code. This is shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
public class MyDao extends JdbcDaoSupport {

  private TransactionTemplate txTemplate;
  ...

  public void setTransactionManager(PlatformTransactionManager txManager) {
    this.txTemplate = new TransactionTemplate(txManager);
    this.txTemplate.setPropagationBehavior(DefaultTransactionDefinition.PROPAGATION_REQUIRED);
  }

  ...
  public int save(MyBean bean) throws Exception {
    final int savedBeanId = (beanId == 0 ? getNextBeanId() : beanId);
    txTemplate.execute(new TransactionCallbackWithoutResult() {
      public void doInTransactionWithoutResult(TransactionStatus status) {
        if (beanId == 0) {
          getJdbcTemplate().update("insert into beans(id,name) values (?,?)",
            new Object[] {savedBeanId, bean.getName()});
        } else {
          getJdbcTemplate().update("update partner set name=? where id=?",
            new Object[] {bean.getName(), savedBeanId});
        }
        for (String beanProp1 : bean.getProps1()) {
          getJdbcTemplate().update(
            "insert into bean_props1(bean_id,prop_value) values (?,?)",
             new Object[] {savedBeanId, beanProp1});
        }
        for (String beanProp2 : bean.getProps2()) {
          getJdbcTemplate().update(
            "insert into bean_props2(bean_id,prop_value) values (?,?)",
             new Object[] {savedBeanId, beanProp2});
        }
        for (String beanProp3 : bean.getProps3()) {
          getJdbcTemplate().update(
            "insert into bean_props3(bean_id,prop_value) values (?,?)",
            new Object[] {savedBeanId, beanProp3});
        }
      }
    });
    return savedBeanId;
  }
  ...
}

Note that I did not want to return anything from my TransactionCallback, so I used a TransactionCallbackWithoutResult object. If there is a need to pass a result back from the callback, you can use TransactionCallback instead.

The Declarative Approach

Spring and most authors recommend using the Declarative approach. With this approach, the DAO code has no reference to the TransactionTemplate. It is identical to a DAO which would have been written to work without transactions. The DAO itself is wrapped inside a TransactionProxyFactoryBean which provides the transactional behavior for the bean. The transactional behavior can be configured in the application context using XML. Our configuration to make the save() and delete() methods work within a transaction but leave the getXXX() methods alone would look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
  <bean id="myDaoTarget" class="com.mycompany.daos.MyDao">
    <property name="dataSource" ref="dataSource"/>
  </bean>

  <bean id="myDao" class="org.springframework.transaction.interceptor.TransactionProxyFactoryBean">
    <property name="transactionManager" ref="transactionManager"/>
    <property name="target" ref="myDaoTarget"/>
    <property name="proxyTargetClass" value="true"/>
    <property name="transactionAttributes">
      <props>
        <prop key="*">PROPAGATION_REQUIRED,-Exception</prop>
        <prop key="get*">PROPAGATION_SUPPORTS</prop>
      </props>
    </property>
  </bean>

I used CGLIB proxying because I did not want to define an interface for MyDao, but if an interface was defined, I could have used standard Java interface based proxying as well. The resulting DAO code now looks identical to one without transactions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public class MyDao extends JdbcDaoSupport {
  ...
  public int save(MyBean bean) throws Exception {
    final int savedBeanId = (beanId == 0 ? getNextBeanId() : beanId);
    if (beanId == 0) {
      getJdbcTemplate().update("insert into beans(id,name) values (?,?)",
        new Object[] {savedBeanId, bean.getName()});
    } else {
      getJdbcTemplate().update("update partner set name=? where id=?",
        new Object[] {bean.getName(), savedBeanId});
    }
    for (String beanProp1 : bean.getProps1()) {
      getJdbcTemplate().update(
        "insert into bean_props1(bean_id,prop_value) values (?,?)",
        new Object[] {savedBeanId, beanProp1});
    }
    for (String beanProp2 : bean.getProps2()) {
      getJdbcTemplate().update(
        "insert into bean_props2(bean_id,prop_value) values (?,?)",
         new Object[] {savedBeanId, beanProp2});
    }
    for (String beanProp3 : bean.getProps3()) {
      getJdbcTemplate().update(
        "insert into bean_props3(bean_id,prop_value) values (?,?)",
        new Object[] {savedBeanId, beanProp3});
    }
    return savedBeanId;
  }
  ...
}

For both the transaction and declarative approaches, note that it is not necessary to have all the JDBC code in a single method. If we wanted to refactor our save() code into multiple doSaveMyBeanMain(), doSaveMyBeanProps1(), etc, methods, using either of the above approaches without any modification will retain the transactional behavior.

Conclusion

This post described two approaches to making the DAO transactional with Spring and JDBC. The approaches described here are not new, they are based on approaches detailed in Spring documentation and other web pages, but I do it in more of a prescriptive style, and in more detail. Hopefully, it will be helpful to people using transactions with JDBC and Spring.

References

The following documents helped me to understand transactions and how they are used within Spring and EJB3.

Saturday, March 17, 2007

Scriptable Java Closures

This mini-project was inspired by another recent project where we had to deliver a fairly complex data set as flat files. The specifications were quite simple to start with, so I started writing it as a single Java class, but then began to morph very rapidly, so before you knew it, the one class had begun to look pretty hairy. Another person joined me on the project, and both of us began to work on different sections of the same file, which brought up more complications with CVS merging and refactoring. He mentioned in passing that his preferred approach was the Unix utility approach, i.e. little Java classes that did specific things rather than the monolithic approach I had set up. This set me thinking, and since I didn't particularly like what I had set up myself, once the initial dataset was created, modifications to it were done with little one-off Java classes. While this approach worked well, the disadvantage was that you are soon inundated with little classes, and you need to remember what each class does and how and when to use it, and what classes to keep and what to throw away. To manage the complexity, I thought it may be good to take the Unix utility analogy one step further, and have some way to link these little classes using some sort of scripting language.

I looked around the internet for scripting languages that could link up one or more Java classes, but found none that fit my needs, so I rolled my own. This post describes a system to parse a LISP like scripting language that I call SCLOS, that can be used to set up and execute chains of Closure objects. SCLOS also provides control-flow using standard Predicate implementations in the Jakarta commons-collections package..

What I needed was a simple way to create a structure of Closure and Predicate implementation outside of Java code, and I needed a simple way of passing in data to this structure. Both Closure and Predicate take an Object argument in their execute() and evaluate() methods, and I usually pass in a DynaBean and let these methods extract what they need from it. Since we are now working with a chain of Closures and Predicates, I needed the parser to also be able to introspect these Closures and tell me what parameters are needed. A side effect of this approach is that now we need only a single shell script to run the Java class, not a script per class.

Providing Life cycle awareness

To allow the Closures and Predicates to report on what parameters they needed to function or what they would set as a result of execution, I set up two Strategy implementations of Predicate and Closure, the LifecycleAwarePredicate and LifecycleAwareClosure, which act as Decorators to our user-defined Predicate and Closure implementations. All these do is to enforce that the actual class of Object being passed in is a DynaBean, then execute all setPropXXX() methods on the target Closure or Predicate to populate it from the DynaBean, run the underlying method, then execute all the getPropXXX() methods on the target Predicate or Closure to populate the DynaBean. The code is shown below:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
// LifecycleAwareClosure.java
public class LifecycleAwareClosure implements Closure {

  private Closure closure;

  public LifecycleAwareClosure(Closure closure) {
    this.closure = closure;
  }

  public Closure getClosure() {
    return closure;
  }

  public void execute(Object obj) {
    if (!(obj instanceof DynaBean)) {
      throw new RuntimeException("Passed in object must be a DynaBean");
    }
    try {
      DynaBean dynaBean = (DynaBean) obj;
      LifecycleUtils.invokeAllSetters(closure, dynaBean);
      closure.execute(dynaBean);
      LifecycleUtils.invokeAllGetters(closure, dynaBean);
    } catch (Exception e) {
      throw new RuntimeException(e.getMessage(), e);
    }
  }
}

// LifecycleAwarePredicate.java
public class LifecycleAwarePredicate implements Predicate {

  private Predicate predicate;

  public LifecycleAwarePredicate(Predicate predicate) {
    this.predicate = predicate;
  }

  public Predicate getPredicate() {
    return this.predicate;
  }

  public boolean evaluate(Object obj) {
    if (!(obj instanceof DynaBean)) {
      throw new RuntimeException("Passed in object must be a DynaBean");
    }
    try {
      DynaBean dynaBean = (DynaBean) obj;
      LifecycleUtils.invokeAllSetters(predicate, dynaBean);
      boolean result = predicate.evaluate(dynaBean);
      LifecycleUtils.invokeAllGetters(predicate, dynaBean);
      return result;
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }
}

// LifecycleUtils.java
public class LifecycleUtils {

  public static void invokeAllSetters(Object obj, DynaBean dynabean) throws Exception {
    List<Method> setters = LifecycleUtils.getAllSetters(obj);
    for (Method setter : setters) {
      setter.invoke(obj, new Object[] {dynabean.get(getVarName(setter))});
    }
  }

  public static void invokeAllGetters(Object obj, DynaBean dynabean) throws Exception {
    List<Method> getters = LifecycleUtils.getAllGetters(obj);
    for (Method getter : getters) {
      dynabean.set(getVarName(getter), getter.invoke(obj, new Object[0]));
    }
  }

  public static List<Method> getAllSetters(Object obj) {
    return LifecycleUtils.getAllMethodsStartingWith(obj, "setProp");
  }

  public static List<Method> getAllGetters(Object obj) {
    return LifecycleUtils.getAllMethodsStartingWith(obj, "getProp");
  }

  public static List<String> getAllSettableVars(Object obj) {
    List<Method> setters = LifecycleUtils.getAllSetters(obj);
    List<String> settableVars = new ArrayList<String>();
    for (Method setter : setters) {
      settableVars.add(LifecycleUtils.getVarName(setter));
    }
    return settableVars;
  }

  private static List<Method> getAllMethodsStartingWith(Object obj, String prefix) {
    List<Method> setters = new ArrayList<Method>();
    Class clazz = obj.getClass();
    Method[] methods = clazz.getDeclaredMethods();
    for (Method method : methods) {
      if (method.getName().startsWith(prefix)) {
        setters.add(method);
      }
    }
    return setters;
  }

  private static String getVarName(Method method) {
    return StringUtils.uncapitalize(method.getName().substring(7));
  }
}

A user-defined Predicate or Closure implementation just implements Predicate or Closure, provides a default constructor, a set of setPropXXX() methods that will set its member variables from the DynaBean, and a set of getPropXXX() methods that will populate the DynaBean from the member variables. The script parser will decorate these user-defined Predicates or Closures with their corresponding Lifecycle aware decorator.

The script parser

The script parser parses a LISP style script that defines a Closure chain (see the Usage example below) and converts it to a Java Closure chain, then runs it. It defines keywords based on a number of standard Predicate and Closure implementations available in commons-collections and builds them from the LISP forms in the script. One of the nicest things about LISP is how close its syntax is to the actual parse tree. In fact, this contributed significantly to the simplicity of the parser. Here are the keywords and their corresponding implementations, and links to their Javadocs.

Keyword Example Implementation
AND (AND Predicate1 Predicate2) AndPredicate
ANY (ANY Predicate1 ...) AnyPredicate
CHAIN (CHAIN Closure1 ...) ChainedClosure
FOR (FOR number Closure) ForClosure
IF (IF Predicate TrueClosure FalseClosure) IfClosure
NOT (NOT Predicate) NotPredicate
OR (OR Predicate1 Predicate2) OrPredicate
SWITCH (SWITCH Predicate... Closure... DefaultClosure) SwitchClosure
WHILE (WHILE Predicate Closure) WhileClosure
DO-WHILE (DO-WHILE Predicate Closure) WhileClosure
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
// ScriptParser.java
public class ScriptParser {

  private Set<String> argumentSet = new HashSet<String>();

  private class OneOrMorePredicate implements Predicate {
    public boolean evaluate(Object object) {
      Integer argc = (Integer) object;
      return (argc > 0);
    }
  };

  private class OddPredicate implements Predicate {
    public boolean evaluate(Object object) {
      Integer argc = (Integer) object;
      return (argc %2 != 0);
    }
  }

  public ScriptParser() {
    super();
  }

  public Closure parse(String scriptFileName) throws Exception {
    argumentSet.clear();
    BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(scriptFileName)));
    StringBuilder scriptBuilder = new StringBuilder();
    String line = null;
    while ((line = reader.readLine()) != null) {
      if (line.startsWith(";")) {
        continue;
      }
      scriptBuilder.append(line).append(" ");
    }
    reader.close();
    StringTokenizer tokenizer = new StringTokenizer(scriptBuilder.toString(), "() ", true);
    Closure closure = (Closure) parseRecursive(tokenizer, 0);
    return closure;
  }

  /**
   * Populates a Set of settable variables for the user-defined Closures and
   * Predicates as a side effect of the parsing. A call to parse() will reset
   * this set, so it must be used immediately after the parse() is called.
   * @return a Set of settable variables for the pipeline.
   */
  public Set<String> getAllSettableVars() {
    return argumentSet;
  }

  private Object parseRecursive(StringTokenizer tokenizer, int depth) throws Exception {
    List<Object> list = new ArrayList<Object>();
    while (tokenizer.hasMoreTokens()) {
      String token = tokenizer.nextToken();
      if ("(".equals(token)) {
        // descend
        Object obj = parseRecursive(tokenizer, depth + 1);
        if (depth == 0) {
          return obj;
        }
        list.add(obj);
      } else if (")".equals(token)) {
        // return
        return buildClosureOrPredicate(list);
      } else if (StringUtils.isBlank(token)) {
        // skip whitespace
        continue;
      } else {
        // accumulate tokens into list
        list.add(token);
      }
    }
    if (depth != 0) {
      throw new Exception("Parsing error, possibly unbalanced parenthesis");
    }
    return buildClosureOrPredicate(list);
  }

  /**
   * Builds a specific Predicate or Closure implementation based on the command
   * and the arguments. Performs some amount of validation.
   * @param list the list representing the S-expression.
   * @return a Closure or Predicate implementation instance.
   * @throws Exception if there are problems with validation.
   */
  private Object buildClosureOrPredicate(List<Object> list) throws Exception {
    try {
      String command = (String) list.get(0);
      List<Object> args = list.subList(1, list.size());
      int argc = args.size();
      if ("CHAIN".equals(command)) {
        assertArgCount(argc, new OneOrMorePredicate(), "CHAIN needs one or more arguments");
        LifecycleAwareClosure[] closures = new LifecycleAwareClosure[argc];
        for (int i = 0; i < argc; i++) {
          closures[i] = getClosure(args.get(i), command, i);
        }
        return new ChainedClosure(closures);
      } else if ("FOR".equals(command)) {
        assertArgCount(args.size(), new EqualPredicate(2), "FOR must have exactly 2 arguments");
        int count = 0;
        if (args.get(0) instanceof String) {
          try {
            count = Integer.parseInt((String) args.get(0));
          } catch (NumberFormatException e) {
            throw new Exception("FOR argument[0] must be an Integer");
          }
        } else {
          throw new Exception("FOR argument[0] must be an Integer");
        }
        LifecycleAwareClosure closure = getClosure(args.get(1), command, 1);
        return new ForClosure(count, closure);
      } else if ("IF".equals(command)) {
        assertArgCount(args.size(), new EqualPredicate(3), "IF must have exactly 3 arguments");
        LifecycleAwarePredicate predicate = getPredicate(args.get(0), command, 0);
        LifecycleAwareClosure trueClosure = getClosure(args.get(1), command, 1);        LifecycleAwareClosure falseClosure = getClosure(args.get(2), command, 2);
        return new IfClosure(predicate, trueClosure, falseClosure);
      } else if ("WHILE".equals(command) || "DO-WHILE".equals(command)) {
        assertArgCount(args.size(), new EqualPredicate(2), command + " must have exactly 2 arguments");
        LifecycleAwarePredicate predicate = getPredicate(args.get(0), command, 0);
        LifecycleAwareClosure closure = getClosure(args.get(1), command, 1);
        boolean isDoWhile = "DO-WHILE".equals(command);
        return new WhileClosure(predicate, closure, isDoWhile);
      } else if ("SWITCH".equals(command)) {
        assertArgCount(args.size(), new OddPredicate(), "An n-way SWITCH must have exactly 2n+1 arguments");
        int n = (argc - 1) / 2;
        LifecycleAwarePredicate[] predicates = new LifecycleAwarePredicate[n];
        for (int i = 0; i < n; i++) {
          predicates[i] = getPredicate(args.get(i), command, i);
        }
        LifecycleAwareClosure[] closures = new LifecycleAwareClosure[n];
        for (int i = 0; i < n; i++) {
          closures[i] = getClosure(args.get(i + n), command, (i + n));
        }
        LifecycleAwareClosure defaultClosure = getClosure(args.get(argc - 1), command, (argc - 1));
        return new SwitchClosure(predicates, closures, defaultClosure);
      } else if ("AND".equals(command)) {
        assertArgCount(args.size(), new EqualPredicate(2), "AND must have exactly 2 arguments");
        LifecycleAwarePredicate[] predicates = new LifecycleAwarePredicate[2];
        for (int i = 0; i < 2; i++) {
          predicates[i] = getPredicate(args.get(i), command, i);
        }
        return new AndPredicate(predicates[0], predicates[1]);
      } else if ("OR".equals(command)) {
        assertArgCount(args.size(), new EqualPredicate(2), "OR must have exactly two arguments");
        LifecycleAwarePredicate[] predicates = new LifecycleAwarePredicate[2];
        for (int i = 0; i < 2; i++) {
          predicates[i] = getPredicate(args.get(i), command, i);
        }
        return new OrPredicate(predicates[0], predicates[1]);
      } else if ("NOT".equals(command)) {
        assertArgCount(args.size(), new EqualPredicate(1), "NOT must have exactly one argument");
        LifecycleAwarePredicate predicate = getPredicate(args.get(0), command, 0);
        return new NotPredicate(predicate);
      } else if ("ANY".equals(command)) {
        assertArgCount(args.size(), new OneOrMorePredicate(), "ANY must have one or more arguments");
        LifecycleAwarePredicate[] predicates = new LifecycleAwarePredicate[argc];
        for (int i = 0; i < argc; i++) {
          predicates[i] = getPredicate(args.get(i), command, i);
        }
        return new AnyPredicate(predicates);
      } else if ("ALL".equals(command)) {
        assertArgCount(args.size(), new OneOrMorePredicate(), "ALL must have one or more arguments");
        LifecycleAwarePredicate[] predicates = new LifecycleAwarePredicate[args.size()];
        for (int i = 0; i < predicates.length; i++) {
          predicates[i] = getPredicate(args.get(i), command, i);
        }
        return new AnyPredicate(predicates);
      } else {
        throw new Exception("Invalid keyword: " + command);
      }
    } catch (Exception e) {
      throw new Exception("Problem building closure from (" +
        StringUtils.join(list.iterator(), ' ') + ")", e);
    }
  }

  private Object resolveBeanName(String beanName) throws Exception {
    Object bean = null;
    if (beanName.indexOf(':') > -1) {
      String[] pair = StringUtils.split(beanName, ':');
      IBeanResolver beanResolver = BeanResolverFactory.getBeanResolver(pair[0]);      if (beanResolver == null) {
        throw new Exception("No resolver available for context:" + pair[0]);
      }
      bean = beanResolver.getBean(pair[1]);
    } else {
      IBeanResolver beanResolver = new DefaultBeanResolver(null);
      bean = beanResolver.getBean(beanName);
    }
    if (bean == null) {
      throw new Exception("Bean " + beanName + " not found");
    }
    return bean;
  }

  /**
   * Evaluate the actual size against the predicate and if the evaluation
   * returns false, throw an exception.
   * @param argc the number of arguments.
   * @param predicate the Predicate object to evaluate against.
   * @param message the Exception message to throw if evaluation fails.
   * @throws Exception if the evaluation fails.
   */
  private void assertArgCount(int argc, Predicate predicate, String message)
      throws Exception {
    if (! predicate.evaluate(new Integer(argc))) {
      throw new Exception(message);
    }
  }

  private LifecycleAwareClosure getClosure(Object obj, String command, int argPosition)
      throws Exception {
    Closure containedClosure = null;
    if (obj instanceof String) {
      containedClosure = (Closure) resolveBeanName((String) obj);
    } else if (obj instanceof Closure) {
      containedClosure = (Closure) obj;
    } else {
      throw new Exception(command + " argument[" + argPosition + "] must be a Closure");
    }
    argumentSet.addAll(LifecycleUtils.getAllSettableVars(containedClosure));
    LifecycleAwareClosure closure = new LifecycleAwareClosure(containedClosure);    return closure;
  }

  private LifecycleAwarePredicate getPredicate(Object obj, String command, int argPosition)
      throws Exception {
    Predicate containedPredicate = null;
    if (obj instanceof String) {
      containedPredicate = (Predicate) resolveBeanName((String) obj);
    } else if (obj instanceof Predicate) {
      containedPredicate = new LifecycleAwarePredicate((Predicate) obj);
    } else {
      throw new Exception(command + " argument[" + argPosition + "] must be a Predicate");
    }
    argumentSet.addAll(LifecycleUtils.getAllSettableVars(containedPredicate));
    LifecycleAwarePredicate predicate = new LifecycleAwarePredicate(containedPredicate);
    return predicate;
  }
}

Pluggable bean resolvers

As you will notice in the code for the Script Parser above, the resolveBeanName(String name) is responsible for instantiating the bean instance named in the script. The first implementation uses reflection to instantiate an instance of the named class. However, I use Spring in my command line code as well, so I thought it would be a good idea to allow a client to plug in the required bean factory implementation. Spring beans can be particularly useful if you wanted one of your Closures to be pre-instantiated with a connection to the database, for example. In order to signal to the parser that an bean resolver implementation other than the default is being used, the beans need to have a prefix (such as spring:myBean) which needs to be declared in the BeanResolverFactory. Multiple implementations can be used in the same script as long as they are suitably initialized from the CLI.

Here is the code for the IBeanResolver interface, the BeanResolverFactory, and the default (reflection based) and Spring based implementations of IBeanResolver.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
// IBeanResolver.java
public interface IBeanResolver {
  public Object getBean(String name) throws Exception;
}

// BeanResolverFactory.java
public class BeanResolverFactory {

  private final static String[][] RESOLVER_CLASSES_DATA = {
    {"spring", "com.healthline.sclos.SpringBeanResolver"},
  };

  private static Map<String,IBeanResolver> RESOLVERS = new HashMap<String,IBeanResolver>();

  static {
    RESOLVERS.put("default", new DefaultBeanResolver(null));
  }

  /**
   * Returns a bean resolver for the bean type, or null if there is none.
   * @param string the string to signal which resolver to use.
   * @return an instance of the Bean resolver.
   */
  public static IBeanResolver getBeanResolver(String string) throws Exception {
    return RESOLVERS.get(string);
  }

  /**
   * Should be called from the CLI handler module to initialize the bean
   * resolver factory from command line parameters.
   * @param contextMap the context map to use.
   * @throws Exception if one is thrown.
   */
  public static void initialize(Map<String,String[]> contextMap) throws Exception {
    for (String[] resolverClassData : RESOLVER_CLASSES_DATA) {
      String[] contexts = contextMap.get(resolverClassData[0]);
      if (contexts == null || contexts.length == 0) {
        continue;
      }
      RESOLVERS.put(resolverClassData[0],
        (IBeanResolver) Class.forName(resolverClassData[1]).
        getConstructor(new Class[] {Object[].class}).
        newInstance(new Object[] {contexts}));
    }
  }
}

// DefaultBeanResolver.java
public class DefaultBeanResolver implements IBeanResolver {

  private static final Logger LOGGER = Logger.getLogger(DefaultBeanResolver.class);

  public DefaultBeanResolver(Object[] obj) {;}

  public Object getBean(String name) throws Exception {
    return Class.forName(name).newInstance();
  }
}

// SpringBeanResolver.java
public class SpringBeanResolver implements IBeanResolver {

  private ApplicationContext context;

  public SpringBeanResolver(Object[] obj) {
    String[] applicationContextFiles = (String[]) obj;
    context = new ClassPathXmlApplicationContext(applicationContextFiles);
  }

  public Object getBean(String name) throws Exception {
    return context.getBean(name);
  }
}

Putting it all together - the CLI

Finally, we have a command line interface which is going to be called from our shell script. It provides command line help, a --list option which allows us to list the parameters that are used by the named SCLOS script, and an --execute option which will execute the SCLOS script with the named context and parameters. The code is shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
// Main.java
public class Main {
  public void help() {
    System.out.println("Usage information:");
    System.out.println("Options:");
    System.out.println("--help|h    : print this message");
    System.out.println("--list|l    : list settable variables.");
    System.out.println("--context|c : specify context files. Not needed for\n" +                       "              fully qualified beans");
    System.out.println("--execute|c : Execute the script.");
    System.out.println("Examples:");
    System.out.println("java Main --help");
    System.out.println("java Main --list scriptfile");
    System.out.println("java Main [--context spring:classpath:applicationContext.xml,...]\n" +
                       "    varname1=value1 [varname2=value2 ...] scriptfile");
  }

  public void initializeContexts(String contextParam) throws Exception {
    // if multiple contexts, the variable is bounded by double quotes and
    // contexts are separated by space
    boolean hasMultipleContexts = contextParam.startsWith("\"") &&
      contextParam.endsWith("\"");
    String[] contextParamElements = null;
    if (hasMultipleContexts) {
      contextParamElements = StringUtils.split(
        contextParam.substring(1, contextParam.length() - 1), ' ');
    } else {
      contextParamElements = new String[] {contextParam};
    }
    Map<String,String[]> contextMap = new HashMap<String,String[]>();
    for (String contextParamElement : contextParamElements) {
      // each individual context looks like type:filename1,filename2,...
      int typeFileSepPos = contextParamElement.indexOf(':');
      String type = contextParamElement.substring(0, typeFileSepPos);
      String[] files = StringUtils.split(contextParamElement.substring(typeFileSepPos + 1), ',');
      contextMap.put(type, files);
    }
    BeanResolverFactory.initialize(contextMap);
  }

  public void reportVariables(String scriptFile) throws Exception {
    ScriptParser parser = new ScriptParser();
    parser.parse(scriptFile);
    Set<String> settableVars = parser.getAllSettableVars();
    System.out.println("List of settable variables");
    for (String settableVar : settableVars) {
      System.out.println("  " + settableVar);
    }
  }

  public void execute(List<String> arguments) throws Exception {
    // arguments ::= (name=value)* scriptfile
    List<String> nameValuePairs = arguments.subList(0, arguments.size() - 1);
    String scriptFile = arguments.get(arguments.size() - 1);
    ScriptParser parser = new ScriptParser();
    Closure closure = parser.parse(scriptFile);
    DynaBean input = new LazyDynaBean();
    for (String nameValuePair : nameValuePairs) {
      int eqSepPos = nameValuePair.indexOf('=');
      String key = nameValuePair.substring(0, eqSepPos);
      String value = nameValuePair.substring(eqSepPos + 1);
      input.set(key, value);
    }
    closure.execute(input);
  }

  @SuppressWarnings("unchecked")
  public static void main(String[] argv) {
    try {
      Main main = new Main();
      // parse the command line
      CommandLineParser parser = new BasicParser();
      Options options = new Options();
      options.addOption("h", "help", false, "Print usage information");
      options.addOption("l", "list", false, "List settable parameters");
      options.addOption("c", "context", false, "Specify available contexts");
      options.addOption("x", "execute", false, "Execute the script with required context and parameters");
      CommandLine commandLine = parser.parse(options, argv);
      List<String> args = commandLine.getArgList();
      if (commandLine.hasOption('h') || args.size() == 0) {
        main.help();
        System.exit(0);
      }
      if (commandLine.hasOption('l')) {
        if (args == null || args.size() == 0) {
          throw new Exception("Script file name should be set");
        }
        main.reportVariables(args.get(0));
        System.exit(0);
      }
      if (commandLine.hasOption('c')) {
        main.initializeContexts(commandLine.getOptionValue('c'));
      }
      main.execute(args);
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

The shell script is a standard shell script that pulls in the required third party JAR files in its classpath and invokes java with the Main class. All command line processing is delegated off to the Main class. The last part of the classpath points to target/classes (for the SCLOS parsing code) and target/test-classes (for the user defined classes that the SCLOS script names). Both should probably be replaced with JAR files in a production environment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
#!/bin/bash
# sclos.sh
export M2_REPO=$HOME/.m2/repository

$JAVA_HOME/bin/java -classpath \
$M2_REPO/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:\
$M2_REPO/commons-cli/commons-cli/1.0/commons-cli-1.0.jar:\
$M2_REPO/commons-collections/commons-collections/3.1/commons-collections-3.1.jar:\
$M2_REPO/commons-io/commons-io/1.2/commons-io-1.2.jar:\
$M2_REPO/commons-lang/commons-lang/2.1/commons-lang-2.1.jar:\
$M2_REPO/commons-logging/commons-logging/1.1/commons-logging-1.1.jar:\
$M2_REPO/log4j/log4j/1.2.8/log4j-1.2.8.jar:\
$M2_REPO/org/springframework/spring/2.0/spring-2.0.jar:\
$HOME/src/sclos/target/classes:\
$HOME/src/sclos/target/test-classes \
com.healthline.sclos.Main $*

Usage example

My test case was a Closure chain to read a file with words and numbers in them, discard the numbers and sort the words so they rhyme. The rhyming is done by first reversing each word, sorting them, and then reversing them back again. Here is the algorithm and the corresponding SCLOS script.

foreach line in inputfile:
  if alpha string:
    add to list1
  else:
    add to list2
reverse list 1
sort list1
reverse list 1
print list1
      
; example.sclos
(CHAIN
  com.mycompany.closures.ReadFileToListClosure
  (WHILE com.mycompany.predicates.ListNotReadPredicate
    (IF com.mycompany.predicates.AlphaPredicate
      com.mycompany.closures.AddToListClosure
      com.mycompany.closures.NOPClosure
    )
  )
  com.mycompany.closures.ReverseListContentsClosure
  com.mycompany.closures.SortListClosure
  com.mycompany.closures.ReverseListContentsClosure
  com.mycompany.closures.PrintListClosure
)
      

To see what parameters sclos.sh takes run the command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
[sujit@cyclone sclos]$ src/main/shell/sclos.sh --help 
Usage information:
Options:
--help|h    : print this message
--list|l    : list settable variables.
--context|c : specify context files. Not needed for
              fully qualified beans
--execute|c : Execute the script.
Examples:
java Main --help
java Main --list scriptfile
java Main [--context spring:classpath:applicationContext.xml,...]
    varname1=value1 [varname2=value2 ...] scriptfile

To find the parameters that the script needs, we run the following command:

1
2
3
4
5
6
[sujit@cyclone sclos]$ src/main/shell/sclos.sh --list src/test/resources/example.sclos
List of settable variables
  listIterator
  currentLine
  filename
  strList

Out of these, the only one we can set from the command is the filename parameter. This is not obvious from the command line, but I could not find a way to distinguish the ones that are passed around versus the one that comes in from the user. Perhaps some naming convention can be set up for this. So finally, we want to run the script against an input file of words input.txt.

1
2
3
4
5
6
7
8
9
[sujit@cyclone sclos]$ src/main/shell/sclos.sh \
  --execute filename=src/test/resources/input.txt src/test/resources/example.sclos
1:place
2:something
3:nothing
4:everything
5:this
6:roddy
7:nobody

The user-defined Closure and Predicate implementations are shown below, to give you an idea of what is involved in setting up the closure chain for the script.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
// AddToListClosure.java
public class AddToListClosure implements Closure {
  private String currentLine;
  public void setPropCurrentLine(String currentLine) { this.currentLine = currentLine; }

  private List<String> strList;
  public void setPropStrList(List<String> strList) { this.strList = strList; }
  public List<String> getPropStrList() { return strList; }

  public void execute(Object input) {
    if (strList == null) {
      strList = new ArrayList<String>();
    }
    strList.add(currentLine);
  }
}

// NOPClosure.java
public class NOPClosure implements Closure {
  public void execute(Object input) { // NOOP }
}

// PrintListClosure.java
public class PrintListClosure implements Closure {
  private List<String> strList;
  public void setPropStrList(List<String> strList) { this.strList = strList; }

  public void execute(Object input) {
    int i = 0;
    for (String element : strList) {
      i++;
      System.out.println(i + ":" + element);
    }
  }
}

// ReadFileToListClosure.java
public class ReadFileToListClosure implements Closure {
  private List<String> list;

  private String filename;
  public void setPropFilename(String filename) { this.filename = filename; }

  private Iterator<String> listIterator;
  public Iterator<String> getPropListIterator() { return listIterator; }

  public void execute(Object input) {
    list = new ArrayList<String>();
    try {
      BufferedReader reader = new BufferedReader(
        new InputStreamReader(new FileInputStream(filename)));
      String line = null;
      while ((line = reader.readLine()) != null) {
        list.add(line);
      }
      listIterator = list.iterator();
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

}

// ReverseListContentsClosure.java
public class ReverseListContentsClosure implements Closure {
  private List<String> strList;
  public List<String> getPropStrList() { return strList; }
  public void setPropStrList(List<String> list) { this.strList = list; }

  public void execute(Object input) {
    List<String> reversedList = new ArrayList<String>();
    for (String element : strList) {
      reversedList.add(StringUtils.reverse(element));
    }
    this.strList = reversedList;
  }
}

// SortListClosure.java
public class SortListClosure implements Closure {
  private List<String> strList;
  public void setPropStrList(List<String> strList) { this.strList = strList; }
  public List<String> getPropStrList() { return strList; }

  public void execute(Object input) {
    Collections.sort(strList);
  }
}

// AlphaPredicate.java
public class AlphaPredicate implements Predicate {
  private String currentLine;
  public String getPropCurrentLine() { return currentLine; }
  public void setPropCurrentLine(String currentLine) { this.currentLine = currentLine; }

  public boolean evaluate(Object object) {
    boolean isAlpha = true;
    char[] chars = currentLine.toCharArray();
    for (char ch : chars) {
      if (! Character.isLetter(ch)) {
        isAlpha = false;
        break;
      }
    }
    return isAlpha;
  }
}

// ListNotReadPredicate.java
public class ListNotReadPredicate implements Predicate {
  private Iterator<String> listIterator;
  public void setPropListIterator(Iterator<String> listIterator) { this.listIterator = listIterator; }

  private String currentLine;
  public String getPropCurrentLine() { return currentLine; }

  public boolean evaluate(Object object) {
    if (listIterator.hasNext()) {
      currentLine = listIterator.next();
      return true;
    } else {
      return false;
    }
  }
}

Conclusion

This has been a fairly long post, and if you had the patience to get here, I hope this post has been helpful. This style of scripting Java objects is probably not what my colleague had in mind when he mentioned Unix utilities, and this approach probably introduces more complexity on another plane. For example, now you are forced to think in terms of Predicates and Closures. Actually, while I agree that the Predicate style of coding where the evaluation happens in a separate class takes a little getting used to, implementing Closures to do some specific task is probably not that far out. Also, since this approach depends on naming conventions for data, its probably a little more inconvenient than simply throwing classes together.

However, I think there are advantages to this approach as well. It allows you to develop in small independent units, and allows you to loosely couple these units into a command line script, as long as you are prepared to deal with (fairly minor, in my opinion) name integration issues during integration.