Adept Open Source Library

User Management

2007-08-31T11:59:00.000+10:00

Adept is a browser-based system. As such it is inherently multi-user. With more than one operator, the need for user management becomes manifest. The Adept Library provides this support with a User object - a persistent DAO.

The User object contains a name and password for authentication. Authentication is not enough. Any system that needs user management needs a concept of authorisation. Who can access which form. Who can update which data. Adept provides this functionality with the concept of groups. The system holds persistent one list of group names in the object User.Groups.

The User.Groups object is quite literally and simply an array of string group names. The User object keep an array of indexes to groups the user belongs to. Groups can be added but not deleted. Actually there is an exception. If a user is added and deleted immediately, their personal group is removed.

When a new user is created, a group of the same name is also created. They are automatically joined to the groups Everybody and the one with their own name.

A newly created system will automatically create users and groups with the names Everybody and Administrator. It is a good idea to give both these users passwords as soon as practical.

How is it used? The system requires that someone be logged in at all times. It can be configured to ask or to default to Everybody an open interface. The log-in can be told to 'stick' so that the user need only log in once on a single system. The Adept user is for authorisation and convenience, not high security. Use the client operating system for that. For this reason there is no user time-out.

The User object for the current user is attached to the session. The menu system, for example, limits display of selections by group names. Only users in the group Administrator, for example, can open the page to add or administer users and groups. In a similar manner, tree nodes can be set to be hidden or unopenable for users not is a specific set of groups. Finally, edit components can be forced to hidden or read-only mode.

Lastly, all applications have a group. The list of available applications on the main desktop will only include those that the current user is in the group for. So, when creating an account for Sally, make sure that she is in the Accounts group if she is to open the accounts system.

Time Sensitive Caches

2005-09-26T13:26:00.000+10:00

Nothing is forever (ok ok - death and taxes). If you have a program that is short-lived compared to the mean time between change of the data being cached, then a LRU cache will be more than adequate. If the program is an application server and could be left running for days, weeks or even months - then there's a good chance that the data being cached will need to be changed. Enter the time sensitive cache. A time sensitive cache can be created with a special constructor or built from a LRU cache with the ageingCache method. In both cases you provide the number of minutes cached items are useful for (as either a property or a number) and a flag set to true to fix the expiry time.

// Once 50 elements are cached the least recently accessed will be discarded.
Cache cache = new Cache( 50);
// Or if the items are 6 hours old
cache.ageingCache( 6 * 60, true);

Every 30 seconds the cache daemon checks all caches for expired entries and removes them. If you attempt to retrieve a cached item after it expires it will be cleared then. It's the responsibility of the calling code to reload an expired item; as this is exactly the same as opening it the first time, no additional code should be necessary.

byte[] buffer = cache.get( fileName);
if (buffer == null)
  buffer = readFile( fileName);

If you need for an entry to be reloaded early, call expire(). If you want to clear the entire cache, just create a new instance and let the old clean itself up. Since time sensitive caches rarely hold Closeable there should be few concerns about locked entries.

Time sensitive caches are typically used to store data that's needed regularly and is more expensive to retrieve from the original source. Two common examples include files and reference data in a database.

Next week will talk about how to use Cache for time-limiting caches. This is the same technology but for a completely different application.

The LRU Cache

2005-08-31T16:06:00.000+10:00

In truth, all caches created using the Cache class are LRU caches. If you need to use the other caching properties without restricting the number of elements, give it a huge size or use the default constructor.

// Once 50 elements are cached the least recently accessed will be discarded.
Cache cache = new Cache( 50);

Just think of a linked list. When you add an item it's added to the end of the list. When you access an item it's unlinked from its current location and linked again at the end. Therfore if you add an item to a list with the maximum number of entries, then the item at the start of the list is removed, close called if closeable, and discarded.

Note that because the Cache class is used for more than just simple LRU caches, the clean operation does not happen synchronously. An item is added on request then the method returns to the caller. This makes the cache very fast. A lower priority background thread will pick up oversized caches, closing and removing least-recently-used items that aren't currently locked.

The LRU cache is commonly used to cache objects in memory that would otherwise require a more expensive retrieval procedure. The Adept browser code, for example, uses a LRU cache to hold static elements such as HTML files and images. This way they are loaded from disk once and served from memory if they are popular enough to always be at the end of the list. Items that are used infrequently are more likely to be discarded and require a disk retrieval to get back. Note that the browser caches static content in a similar way. In addition, the Adept server cache will effect files requested from multiple browsers.

Using a LRU cache, rather than storing the item in a Map, allows the developer to control memory usage.

If there's any possibility that an item being cached may change at the original source, use a trigger or time sensitive cache as well as the LRU method.

Next time we'll talk about Time Sensitive Caches.

Caching - a Commonly Used Optimisation Method

2005-08-10T17:14:00.000+10:00

Every enterprise level application implements caching somewhere. Even loading static class-level fields when the class is first loaded is a form of caching.

The Adept Library Cache class is designed to implement pools, time sensitive and least recently used caches. Since each of these similar structures are used for completely different purposes, they are the subject of an article each. In summary:

LRU (Last Recently Used) size limited cache are commonly used for files or data records. Once the cache size is exceeded the least recently used items are closed to make room for new ones. This way commonly used items are retrieved from memory.
Time sensitive caching. Files or data that change infrequently can be instructed to be reloaded if they exceed a certain cache age. This is a method often combined with LRU caching. Files or data may change overnight, so having a maximum age of 12 hours guarantees that they are re-read when necessary but provide optimal retrieval at other times. Set fixedExpiry on so that the entry expires no matter how popular it is.
Time limited caching. Sessions are a good example. Set the maximum size to large enough that it won't be reached in normal usage. It can be used to protect memory from a run-away situation. Call the ageingCache() method to tag the cache as holding items that expire. Even if the LRU limit is exceeded, non-expired items will not be deleted.
Pools are specialised caches that are not retrieved by key. Free items are retrieved from the cache and locked for use. File handle, database connections and similar shared resources can be pooled to save creation time. Pools will frequently use the time-limiting features to make sure that unused items are closed rather than hanging around forever. Size limiting can also be used to enforce resource usage limitations. Make sure fixedExpiry is set for LRU cleaning to occur.

The Adept Cache class is based on a LRU algorithm. This method allows a cache to be limited to a specific size. Once that size has been exceeded the item that has not been used for the longest period is discarded.

Cache Entries

Entries can be any form of Java POJO, or classes that have the Closeable interface. In the latter case, the classes close() method will be called when the item has been discarded from the cache for whatever reason.

Entry Lifetimes

Because cache items can hold non-memory resources (such as connections), any item with a Closeable interface retrieved must be unlocked before it can be eligible for removal from the cache. For this reason, take great care when using caches to ensure that a cache entry is unlocked in an inescapable stream of code after the retrieval. This usually means doing it immediately or wrapping a retrieval in a try/finally block with the unlock in the finally. I cannot emphasize enough that this practice should never be missed, else you will have resource leaks. For ordinary data that does not need closing, ignore the locking and unlocking features entirely and treat a cache as you would a Map or any other generic container.

MyData data = null;
try
  {
    data = cache.get( key);
    ...
  }
finally
  {
    cache.unlock( data);
  }

General Caching Operations

Once a cache is created, items can be entered with add(), retrieved with get() and removed with delete(). Adding and retrieval is by key, and said key can be any object or an int primitive. For caches containing closeable items, the delete can fail if they are currently locked. The cache is synchronised so it can be accessed by multiple threads.

There will be other articles in the specific forms of cache listed above.

Message - Server to User Communications

2005-07-29T12:02:00.000+10:00

What happens when a user fills in a form and presses the submit button? If all goes well he will move on as expected. But what if the data in the form is not adequate to the task? Perhaps a field was not filled in, or business logic dictates that the answers do not match. The server code needs to stay on the same form and give meaningful messages.

Enter the Messages class. It is designed to be used by the SOA tier, although in truth it can be used at whichever layer knows enough to be able to give the user meaningful feedback. The user does not want to be confused by being told that idxdb43a has a duplicate entry. They want to be told that they have already registered and should type in their password.

Generally messages are created by the validation code and business logic. Sometimes, as above, the business tier creates a message based on an error reported by a more technical tier.

Retrieving the Message List

Messages that relate directly to the converstation between user and machine can be retrieved with the getThreadInstance() method. A conversation interchange between browser and server runs on a single thread. The thread message instance is cleared as soon as the browser communicates to the server. Messages are collected here during server processing. Once complete the messages are sent to the browser as part of the response.

Messages can also be held in the session and retrieved with getSessionInstance(). Used to collate messages from non-interactive threads with information for the user. A classic example is an alarm or countdown timer.

The Message Structure

Because I designed the message system for thin-client browser/server applications, messages needed a text based structure. Consequently a message consists of key and display parts separated by double colons (::).

key::this is the message portion to display

In the most basic form, you can add a message without a key. It is saved under the special global key of start (*). The following three add() statements are identical:

Messages messages = Messages.getThreadInstance();
messages.add( "This is a global message");
messages.add( "*::This is a global message");
messages.add( "*", "This is a global message");

Reporting Programming Errors

In many environments I have worked in it was common practice for programming errors and system problems to throw an exception that was then passed all the way back to the client. Said client then gets an generic error page and the thrown exception is written to the log. I do not consider this acceptable. All complex systems will generate unexpected errors and attempt to pass them back to the user. If these message are processed like any other, the user has the opportunity to work around the issue. Message in these curcumstances should include a key to be used by support staff to investigate the log. Messages does this with overloaded add() methods that take a Throwable argument. Since exception messages are less likely to be understood by a user, it includes an automatically generated reference to the log that support staff can use to track down the issue.

Device /dev/hd1 out of space (see log reference 20050722-21:44:53.531)

When to Generate a Message List

The Service or business tier needs to process a user request and send back the results for the user to read. If exceptions were to be used in their basic form, the user would get a single problem reported. Once this is fixed and the request submitted again the next problem is found and reported again. We have all worked with systems such as this - when we have to. They are very frustrating. It is far better to find and report on as many problems together. The user can then fix them in one pass also.

With Messages the code continues to work until it finds a problem that stops further processings. It then passes the menu structure back to the GUI tier for display. Actually it does not need to pass it since the GUI tier can also retrieve a reference using the static method getThreadInstance().

A typical service may work as follows:

Messages problems = Messages.getThreadInstance();
validateData( data):
if (! problems.none()) return;
applyBusinessRules( data);
if (! problems.none()) return;
writeResults( data);
if (! problems.none()) return;
commitAllProcessing();

Why Do Messages Have Keys?

When the messages are retrieved, they are sorted in key order with the message for a key being stored in a multi-line string in the order they were generated. The use for multiple keys is implementation dependant. It can be used to ensure that messages are sorted by priority, associated with fields, or both. The GUI, for example can cycle through the messages. Those with a key the same as an input field can be displayed beside that field while others can be listed in a special messages region. Since the general messages are sorted alphabetically, they can be given a numeric key with the highest priority having the lowest number. It would even be possible to assign different colours for different priority messages:

0: Urgent error message (typically system failure) - displayed in bright red.
1: Validation or similar error message - displayed in dark red.
2: Warnings - displayed in orange.
3: Informational messages - displayed in blue.

Note that these are examples only. Implementation is dependant on the user of the Message class.

How Messages are Retrieved

If keys are not used at all in the application, getGeneral() can be used to retrieve all the messages. Otherwise, firstKey() and nextKey() are used to retrieve each message group, with get() for retrieving the messages for the group.

for (String key = messages.first();  key != null;  key = messages.next())
  displayMessages( key, messages.get( key);

The messages retrieved by get() are an array of the messages. The array is in the order the messages were generated by the application.

Fractions are Useful Too

2005-07-07T09:53:00.000+10:00

Fractions are Useful Too

There was a time before floating point co-processors and built-in floating point arithmetic that some systems chose to hold real numbers as fractions.

The Advantages of Fractions

Some numbers can be held more accurately as fractions. 1/3 cannot be accurately represented in floating point.
All fractions that do not cause an overflow are accurate, whereas floating point is always an approximation. It's common, for example, to see a result of 0.999999999998 when the result should have been one. This is particularly true when passing the number around. Writing a floating point to a relational database and retrieving it again for comparison to the original value can return false. This is because the floating point representation may change inside different systems.
The calculations are faster than floating point emulators. This advantage doesn't hold out against dedicated floating point hardware.
It was very cool for numbers that could be displayed as a fraction. The human mind mostly finds it easier and quicker to visualise two and an eigth, as compared to 2.125.
Some areas of use are more comfortable with fractions than decimals. Shares in a property or horse, for example, are commonly things like 1/3, 1/2 and 1/6.

The Disadvantages of Fractions

Floating point holds a more limited representation. A Java long is only 19 decimal digits. This is still a big number. If a fraction were to represent microseconds, it would hold nearly 585 thousand years. If course if it were to represent nanoseconds, it would be limited to 585 years.
All fractions that do not cause an overflow are accurate. Yes, this is a disadvantage also. Accuracy requires space. Each calculation can potentially increase the size of the denominator, thereby decreasing the largest number that can be held. 1/3 + 1/5 = 8/15. The largest possible number has now been reduced by 1/3. This is an implementation restriction that could be eradicated by having a separate whole part to fraction. A lot of the internal maths becomes more complicated and expensive, so I have chosen to keep the limitation.
Fractions are not as fast as integers at arithmetic. To minimise the issues with lost maximum whole parts, each calculation reduces the fraction to the smallest it can be. 1/3 + 1/6 is kept as 1/2, not 3/6. It might be possible to make the Fraction class smarter and only do the expensive greatest-common-denominator when converting to a viewable string or when a calculation is at risk of overflowing.

The Fraction Class

The Fraction class is an immutable class for creating and manipulating fractions. Immutable means that once a fraction has been created it cannot be changed, operations create a new instance. So,

Fraction result
  = new Fraction( 1, 1, 2)
    .times( new Fraction( 2, 0, 0));

creates 3 instances of Fraction. Creating these fractions is non-trivial since the fraction is normalised as part of the operation. Don't be scared. Normalisation isn't that intensive, but it is well to be concious that any method call can take valuable CPU time. I believe that it's always a good idea to save data in a variable with a lifetime to match it's static nature. In the previous example, given that I may often want to multiply by 2, I would create a final static called two and create it only once.

Why Immutable?

There are very few uses for immutable classes outside of the primative wrappers provided in the standard library. Fractions are one of them. Here are the benefits:

No synchronisation needed if used in threaded code. It can't change so it won't be changed accidentally.
The internal data is private, so it can be open to the world for read access safely.
They are safe from fiddly or unexpected side-effects.

Besides, an immutable class is easier to implement.

Creating a Fraction

The standard numerical constructor was shown before. It takes 3 long values for whole part, numerator and denominator. There is also a short form that just takes a whole number. Because whole numbers are used commonly and don't require normalisation, this becomes a good optimisation technique.

Fractions often come from text sources, so there's also a constructor that takes a string. It provides a more flexible way of representing a fraction. It can be a standard whole-numerator/denominator (i.e. 5-3/16). The whole part can be separated from the numerator by a space, a colon or a dash. String fractions can be decimal representations (as in 5.1875). Lastly, just for fun, it can be a percentage (10% == 1/10). With text entry comes the possibilty that the data cannot be interpreted as a fraction or decimal. In these cases, a public variable empty is set. If you want an exception, throw one. Since the fraction may be only part of the text, a public int is set to the next index in the string not used to build the fraction.

String fractionString = "1/3rd";
Fraction result = new Fraction( fractionString);
if (result.empty)
  throw new NumberConversionException(
    fractionString+" not a fraction");
String rest = fractionString.substring( result.after);
assertTrue( rest.equals( "rd");

The Greatest Common Denominator

The greatest common denominator is the largest integer that divides into both numbers without a remainder. Obviously it is useful in fractions as we have discussed above. It has other uses, too. Some PKI formulae use it. If the GCD (for short) is 1 then both numbers are relatively prime.

if (Fraction.greatestCommonDenominator( numerator, denominator) == 1)
 System.out.println(
   numerator+"/"+denominator+" are relatively prime");

Note that this is a static method so it can be used without generating a fraction.

Fractional Output

A Fraction instance has a toString() method that will display the fraction in the form W-N/D.

assertTrue( new Fraction( "1.5").toString()
  .equals( "1-1/2");
assertTrue( new Fraction( "24%").toString()
  .equals( "6/25");

It also has a toHTML() method that uses subscripts and superscripts to display a fraction. One and a half would display as 1^1/2 If you want to do anything else with the fractions, the numerator and denominator are public (if final).

// Display a fraction as a decimal value.
System.out.println(
  (Double) fraction.numerator
    / (Double) fraction.denominator);

Sorting and Maps

The Fraction class implements Comparable, equals() and hashCode(), so it can be used for sorting and as HashMap keys.

Fractional Maths

The Fraction class only implements basic math at this stage. It's all I've needed for dealing with shares.

negative()
inverse()
plus()
minus()
times()
divide()

So, if you need to play with fractions, this class is freely available in the Adept library at http://library.marringtons.com,

Data Migration Support

2005-05-31T19:20:00.000+10:00

The Adept Object Database stores Java objects directly into persistent storage. It's not just the primary way of accessing stored data, in fact it's the only way. This makes for a fast efficient system with one major drawback - data migration. If you change the contents or order of persistent fields in a Java DAO or any of it's internal objects then all databases using those objects will fail when they attempt to use the updated code.

This is a common problem with databases. The resolution is called migration. In the relational database world the DBA will write a migration script. It's then a manual process to ensure that the migration script is run on production data at the same time as the new code is promoted. In the shrink-wrap software world it's the responsibility of the install program to recognise an upgrade and migrate the client data.

The Adept Library has a Migration class that does most of the work for you once you have completed some manual steps:

Recognise that a DTO needs to change. This isn't as easy as it sounds, since a DTO can have POJO fields - and if those fields change the DTO is no longer valid.
If it doesn't yet exist, create a new class called Migrations in the same package as the DTO that is to change.
Inside the Migration class create a public static class extending DAO with the same name as the class that requires migration. If the migrating class is itself an inner class, match the class path inside Migration.
Copy the code from the class being migrated into the newly created inner class as a further depth of inner class.
Change the name of the class just copied to something unique. I use the date or release number.
Make it static.
Have it implement Migration.Interface.
Implement the migrate() method that will return the new changed DAO.
If there is a previous migration for this DAO, change it's migrate method to return this DAO rather than the external one.
Implement the code changes to the original external DAO.

This is actually easier than it sounds. The best way, as always, is to provide an example:

// File DAOTest.java
public class DAOTest extends TestCase
  {
    static class TestDAO extends DAO
      {
        /***/ public String string;
        /***/ public int integer;
        /***/ public float real;
        /***/ public static class PrimaryIndex
            extends Index
          { /***/ public String string; }
        /***/ public static class SecondaryIndex
            extends Index
          { /***/ public int integer; }
        /***/ public static class TertiaryIndex
            extends Index
          {
            /***/ public int integer;
            /***/ public String string;
          }
      }
  }

.......................................

// File Migrations.java
public class Migrations
  {
    /**
     * Migration for DAOTest DAOs - note that these
     * are inner class DAOs.
     */
    public static class DAOTest
      {
        /** Test migration path */
        public static class TestDAO extends DAO
          {
            /** First on migration path */
            public static class V050511a extends DAO
                implements Migration.Interface
              {
                /***/ public String string;
                /***/ public int integer;
                /***/ public float real;
                /***/ public static class PrimaryIndex
                    extends Index
                  { /***/ public String string; }
                /***/ public static class SecondaryIndex
                    extends Index
                  { /***/ public int integer; }
                /***/ public static class TertiaryIndex
                    extends Index
                  {
                    /***/ public int integer;
                    /***/ public String string;
                  }
                
                /**
                 * @see Migration.Interface#migrate()
                 */
                public DAO migrate()
                  {
                   /*
                    * Originally created DAOTest - 
                    * changed when new migration added
                    */
                    V050511b testDAO = new V050511b();
                    /*
                     * Use DTO deep copy since objects
                     * will be so similar.
                     */
                    if (dto == null)
                      dto = new DTO( this, testDAO);
                    dto.copy( testDAO, this);
                    /*
                     * This is where we would do the
                     * processing that is the essence
                     * of migration. If we added fields
                     * we may need to fill them here.
                     */
                    testDAO.integer++;
                    /*
                     * Lucky last - we return the newly
                     * created replacement DAO.
                     */
                    return testDAO;
                  }
                private static DTO dto;
              }
              
            /** Second on migration path */
            public static class V050511b extends DAO
                implements Migration.Interface
              {
                /***/ public String string;
                /***/ public int integer;
                /***/ public float real;
                /***/ public static class PrimaryIndex
                    extends Index
                  { /***/ public String string; }
                /***/ public static class SecondaryIndex
                    extends Index
                  { /***/ public int integer; }
                /***/ public static class TertiaryIndex
                    extends Index
                  {
                    /***/ public int integer;
                    /***/ public String string;
                  }
              
                /** @see Migration.Interface#migrate() */
                public DAO migrate()
                  {
                    com.marringtons.object.DAOTest.TestDAO
                      testDAO = new
                        com.marringtons.object.DAOTest.TestDAO();
                    if (dto == null)
                      dto = new DTO( this, testDAO);
                    dto.copy( testDAO, this);
                    testDAO.real++;
                    return testDAO;
                  }
                private static DTO dto;
              }
          }
      }
  }

The Act of Migration

The worst migration case I've personally experienced was with an accounting program. When upgrading a 5 year old system it required me to install two complete intermediate versions of the software, migrate to them in sequence then install the latest version and migrate a third time.

Adept migration is more transparent. Your setup code must call Migration.database(), giving it the name of the database to migrate. It will review every persistent class and record the operation, so only the parts of the migration path not yet performed will be done. As an example: suppose over 10 releases, a particular DAO has changed 5 times (shame on you). Any user could be upgrading from an earlier version to the latest at that time and migrate again later. The migration code will account for this and pick up the migration thread from where it left off the last time the software was upgraded.

Migration Limitations

It only works with classes that inherit from DAO. If a POJO field inside a DAO changes, you will need to create a new POJO with a new name and migrate from the old to the new manually in the DAO migrate(). This is a good reason to inherit from DAO for any persistent non-trivial data object.
It must be called by a single threaded setup routine to stop any other possible simultaneous access of the database being migrated.
The developer is responsible for providing a migration path before changing a DAO or a POJO inside a DAO. This goes against the practice of changing internals without concern for effect. If you have application specific POJOs inside DAOs, comment the source so they will not be changed by accident. Better still, have them inherit from DAO or copy them to a DAO as part of the persistence code.

String Formatting - A Step Towards Internationalisation

2005-05-12T11:38:00.000+10:00

The Java alternative to the C printf() function is java.text.MessageFormat, a rarely used class to create parameterised strings. It is, in fact, considerably more powerful than printf, but whereas a C programmer would regularly use printf as they standard string output method, few Java programmers even know they can format text easily. I can't count how often I've seen Java classes with code specific to padding numbers or adding an s for plurals in a string.

Why? The three reasons in order of excuse are:

Most developers don't know that a text format class exists. It was introduced in a later Sun release in 2003 and doesn't appear to be a common part of basic Java training.
It has a complex syntax for both code and template string that are hard to remember.
The syntax isn't very readable.
And if a developer actually looks at the implementation, he/she will realise how expensive it is. Every time you create a new MessageFormat instance it compiles the pattern. This means that the best way to use it is with static final in the class. I dislike this as it separates the message from the code, making the code less readable.

The com.marringtons.string.Format class is a wrapper that resolves all but the first of the problems listed above.

We can't help here. Since the wrapper isn't part of the base Sun Java library, it is unlikely to ever be included in general training. By meeting the next 2 challenges, at least it can be used as a project standard if the principals of said project deem it so.
MessageFormat is created with a new and a format method to produce the formatted string. This isn't very readable when used in-line. Format uses a static factory method called pattern(). MessageFormat also has a static format method, but it's very expensive to use as it recompiles the pattern every time. The patterns are hard to remember. I have created examples of each below for your reference.
MessageFormat takes an Object array for parameters, while Format has a with() method for objects and primatives.
The Format.pattern() factory method is the only way to create a format. It caches the MessageFormat instances so that they are only compiled once per program run.

Enough words - time for an example of formatted strings both ways. Both ways below will produce the same result.

private static MessageFormat myMessage
  = new MessageFormat(
    "Result is: Integer {0}, String {1}");
...
System.out.println(
  myMessage.format( new Object[] 
    { new Integer( 12), "twelve" }));

// is the same as:

System.out.println(
  Format.pattern(
    "Result is: Integer {0}, String {1}")
    .with( 12).with( "twelve"));

Format and MessageFormat Pattern Templates

The Sun documentation is correct, but not very clear. The best references are by example, so I'll provide an example from my unit tests that show each format type. The check method takes the pattern in the first parameter and the value in the second and tests then in the expected result in the 3rd.

check( "{0}", 1234, "1,234");
check( "{0,number}", 1234, "1,234");
check( "{0,number,integer}", 1234, "1,234");
check( "{0,number,percent}", 0.16, "16%");
// @see java.text.DecimalFormat
check( "{0,number,#,##0.00;(#,##0.00)}",
    -1234, "(1,234.00)");

Calendar calendar = new GregorianCalendar(
    2005, 2, 13, 20, 38, 55);
Date date = new Date( calendar.getTimeInMillis());
check( "{0,date}", date, "13/03/2005");
check( "{0,date, short}", date, "13/03/05");
check( "{0,date, medium}", date, "13/03/2005");
check( "{0,date, long}", date, "13 March 2005");
check( "{0,date, full}", date, "Sunday, 13 March 2005");
// @see java.text.SimpleDateFormat
check( "{0,date,yyyy.MM.dd G ''at'' HH:mm:ss z}",
  date, "2005.03.13 AD at 20:38:55 EST");

check( "{0,time}", date, "20:38:55");
check( "{0,time, short}", date, "20:38");
check( "{0,time, medium}", date, "20:38:55");
check( "{0,time, long}", date, "20:38:55");
check( "{0,time, full}", date, "08:38:55 PM EST");

String choice = // @see java.text.ChoiceFormat
    "{0,choice,-1#negative ({0})|0#none|1#one"
 +"|1<{0,number,integer}|101#greater than 100}";
check( choice, -23, "negative (-23)");
check( choice, 0, "none");
check( choice, 1, "one");
check( choice, 11, "11");
check( choice, 111, "greater than 100");

Why Internationalisation?

You may have been wondering why the title of this article proclaimed MessageFormat as "A Step Towards Internationalisation". There are many aspects to internationalisation with respect to software development, but translation of text would have to be one of the core ones. I'll discuss all the options as the subject of another essay, but my preferred method is to develop with in-line strings as part of the code where applicable, then use the English strings as a key into a dictionary of translated text when it is needed. This will only work if you use a Format class to separate templates from variable data. The same restriction applies to keeping messages in a database or configuration file.

In short, using

message = "I think there are "
    +aNumber+" people in the group";

cannot easily be translated, while

message = Format.pattern(
  "I think there are {0} people in the group")
  .with( aNumber);

can.

Yet Another Date Object

2005-05-03T12:32:00.000+10:00

Sun really mucked up date processing in Java. First there was java.util.Date - a great little class that did most of what anyone would want with dates. Except that - oops - it didn't account for internationalisation. Enter the Calendar object - big, clunky, hard to use - but able to do everything. Now, deprecate almost everything of use in Date, but have it still tied to the hardware while Calendar is not. Fun.

The Adept library also includes a Date class that provides easy-to-use facilities where a Calendar object is unwieldy and overkill. Like java.util.date it keeps the date and time in Epoch time. This is a fancy name for the number of milliseconds since the first of January 1970. And, no - there will be no Y2K-like issues in Java since it is a long, being 64 bits long. It was only the old C programs where a long is 32 bits that the time will roll over in about 25 years.

Like the String class, the Date class is immutable - with all the additional benefits an immutable class carries with it. If you're scratching your head here, immutable simply means that it cannot be changed. To quote Brian Goetz of IBM, "Immutable objects have a number of properties that make working with them easier, including relaxed synchronization requirements and the freedom to share and cache object references without concern for data corruption". Get the whole story here.

A Date object can be created with the current time, an Epoch time, a Calendar or an internal Date mathematics function.

Once you have a date, the class allows you to:

Change it to other forms: such as seconds() for Epoch seconds - being a date, accurate to the second, that fits into an int (32 bits); timeOfDay() which then returns a Time object for more granular work; calendar() for dealing with the rest of the Java world; or timestamp or dateStamp() for the date in a reverse order string (20050425 for April 25, 2005) that can be used for lists or filenames that need to be sorted by date.
Date/Time Maths: being add() to add a time interval (days, hour, minutes, seconds) to a date or next() to set the date for a specific time of day (as in 5pm in 3 days time).
Date Terms: being able to add or subtract a term from a date or to be able to extract a term between two dates. Very useful for loans, leases and other similar software requirements.
Days in Month: I've often found a need to work out the days in a particular month for various scheduling or display purposes. For this reason I've exported a static method that provides this information, given the year and month.

String Utilities

2005-04-20T09:29:00.000+10:00

The Strings class in Java is very special. Not only can instances be created by putting double-quotes around text, but it is allowed to use special overloaded operators (such as plus-sign) that the reset of us can't implement without reverting to C++. Hmmm, I sound bitter. Well, only slightly - after all we sometimes have to live with the tools we are given.

The String class is also final - which means you cannot add functionality. This is a shame because it doesn't do everything it could to make life easy. The Adept library contains a com.marringtons.string.Strings class of static methods for providing tools to make strings a little easier to work with.

There are methods to make string tests easier - even if the reference is null. There are methods to split strings and others to join them again. There's a method to create an array of strings from a collection of objects and another to create the array from a regular expression matcher. Lastly, there is a method to order strings by length (Strings.sortDescendingByLength).

Null Strings

It's not uncommon to have a empty string as either a null reference or a reference to a string with no content. This is not a problem with Java 1.5 and above, but in earlier code the tests all looked unwieldy and there was a risk of NullPointerException occurring - unchecked to the interface.

if (name != null  &&  name.length() > 0)
  name = askForName();

Strings provides a more readable alternative:

if (Strings.isEmpty( name))
  name = askForName();

if (Strings.isNotEmpty( name))
  sayHello( name);

The other related problem is the good old toString() object. When it is called explicitly it gets very upset if the object is null. Strings to the rescue:

Object response = getResponse();

if (response != null)
  name = response.toString();
else
  name = "";

// ... is the same as ...

name = Strings.toString( response);

Splitting Strings

Java.util.regex.Pattern has a split method that separates a string into an array of strings based of a separator pattern. Strings provides some convenience methods for comma separated and OO named lists. The adept library uses comma separated variables (csv) from the properties file and from HTTP forms. The OO version is often used to map form data to classes as part of form processing.

String[] parts = Strings.split( "ab, cd, ef");
parts = Strings.splitJava( "Java.util.regex.Pattern");

Joining Strings

Another common function is to create a string from component parts - joining them.

String[] parts = { "a", "b", "c", "d" }
string = Strings.join( parts); // a, b, c, d
string = Strings.join( parts, "."); // a.b.c.d
string = Strings.join( parts, ".", 1); // b.c.d
string = Strings.join( parts, ".", 1, 3); // b.c

There is also a version of the second method that takes a list for when the string array is not yet an array.

String Arrays

It's not uncommon to collect a list of strings - without knowing how many there will be - and then return an array of them.

public String[] listArchive() throws IOException
  {
    ArrayList files = new ArrayList();
    Enumeration enumeration = zipFile.entries();
    while (enumeration.hasMoreElements())
      files.add( ((ZipEntry) enumeration.nextElement()).getName());
    return Strings.toArray( files);
  }

There is also a method for generating a string array given a string and regular expression.

Pattern pattern = Pattern.compile( "(\\d+)\\D+(\\d+)");
Matcher matcher = pattern.matcher( "123<==>456");
check( matcher.matches());
String[] strings = Strings.toArray( matcher);
check( strings.length == 2);
check( strings[0].equals( "123"));
check( strings[1].equals( "456"));

Map Initialisation

2005-04-13T08:31:00.000+10:00

Java provides a dictionary-like facility called a Map of which the most popular version is HashMap. They are great for keeping name/value pairs and are used heavily in almost all programs where type-safe naming is too restrictive. Most interpretive languages such as Perl or Python provide methods to initialise a dictionary in code, but native Java has no such built-in tool.

No fear, it's easy to build - and the Adept library has a Map helper class in the util package with two static load() methods. One to create a HashMap and another where the map is provided so that you can use another variant such as a LinkedHashMap. Both methods use arrays to provide the information to load the map.

Examples

Map fieldTranslation = Maps.load( new Object[]
  {
    "integer", "integer",
    "aLong", "anotherLong",
  });

Map pronounce = Maps.load( new LinkedHashMap(),
  new Object[]
    {
      "a", "aye",
      "b", "bee",
      "c", "see",
      "d", "dee",
      ...
    });

Sets

Sometimes maps are used as sets - being a collection of keys without values. The library class com.marringtons.util.Maps has two methods for working with map sets.

FileReader fileReader = new FileReader();
fileReader.setArchive(
    "com/marringtons/util/system.resourceTest.zip");
String[] files = fileReader.listArchive();

Map map = Maps.loadSet( files);
assertTrue( map.containsKey(
    "level1/level2/test in zip.txt"));
assertTrue( map.containsKey(
    "META-INF/PATH-BASE.TXT"));
assertTrue( map.containsKey(
    "META-INF/PROPERTIES.TXT"));
assertTrue( map.containsKey(
    "level1/system.properties.txt"));

String[] files = fileReader.listArchive();
Arrays.sort( files);
Map map = Maps.loadSet( new LinkedHashMap(), files);

Transactions and the Adept Object Database

2005-04-07T07:13:00.000+10:00

In the normal course of events you may read in some data, mark it for update, change it then commit it. The Adept Object Database system will queue it for writing to the database as soon as possible - typically in the next tenth of a second or so.

Now, it's far easier to design software by compartmentalising functionality. This not only means keeping disparate functions in different places, but also limiting how much they need know about each other.

Let's take an example form that the user enters for an application. It could include address, contact and application specific details. The processing code would look something like:

user = createUser( form.userName);
addAddress( user, form.addressFields);
addContact( user, form.email);
addApplication( user, form.applicationDetails);

Now, what happens if the application includes a date of birth and that shows that the applicant is under-age. Sure, we'll output and tell the applicant, but what to do with the data? We now have user, address and contact details that are of no use without an application. We could write a delete routine to remove the information already added. This wouldn't be a bad solution - particularly if you needed a delete routine anyway for maintenance. It would be better, however, if the inconsistent or unnecessary data never saw the database. This is what transactions are for.

try
  {
    Transaction.start();
    user = createUser( form.userName);
    addAddress( user, form.addressFields);
    addContact( user, form.email);
    addApplication( user, form.applicationDetails);
 Transaction.end();
  }
catch (Throwable throwable)
  { Transaction.abort(); }

Note that once a transaction starts, it must be completed or aborted - otherwise the persistence is never achieved.

Nesting Transactions

The Adept Transaction system supports full transaction nesting. On every start() a new nesting level is created. An abort() will drop all items updated since the last start() while an end() will add them to the next higher transaction group. An end() for the highest transaction group will have all items within the transaction queued for a commit. A global endAll() or abortAll() will process all nesting levels at once.

The Adept Development System

If you are working inside of the Adept Development System the connection manager does this all for you. It maintains a transaction across each conversation. You only need to call transaction management explicitly if you have a group of transactions that need to be controlled - but not effect other outer transactions if they need to be aborted. The end of the conversation can use endAll() or abortAll() to clear the nested transaction stack if needed.

In short, ignore transaction management as it's all done for you behind the scenes. Throw something if you want all database writes to be rolled back.

The Adept Object Database

2005-03-29T07:21:00.000+10:00

The Adept Object Database - What is it?

The Adept Object Database is a persistent storage and retrieval system for Java Objects. The design goal was simplicity over completeness, and for this reason it's not JDO complient. It doesn't try and compete against SQL by providing a query language. Data from Java objects are persisted directly to the database as binary - and indexed from information in the same class. It is, however, relational with DAO object within DAO objects referred to by reference.

The Adept Object Database - Why is it?

Adept is designed to create shrink-wrapped software packages to be installed by the end-user. One of the primary differences between this environment and application servers with myriads of support staff is the lack of a standard database. Windows boxes will probably have Jet installed, while Unix derivatives will have one of the hashing databases such as berkeley-db. I could not find a reasonable simple stand-alone database library for Java. Now there are good SQL ones, such as HSQLDB, that would have fitted within the Adept concept of leveraging corporate environment skills. But I personally had a hankering to create an object database. Among other things it's a lot faster than a relational SQL database.

The Adept Object Database - When is it?

The Adept Object Database is available now as open source software at SourceForge under the name adze-db. Adze was the earlier name for Adept before I realised that it sounded like spamming software. A new version will be release soon (May 2005) with XML I/O, backups, screen scraping, lazy loading, data migration and database transactions. Oh, you want to know when to use it? Picky, picky...

For shrink-wrapped desktop applications.
Where data retrieval speed is a primary goal.
When you need to persist Java objects (as opposed to matching pre-designed database schema).

It may be more valid to list when not to use the Adept Object Database.

When you have in-depth SQL corporate experience you want to use. If you enjoy SQL then why not use it? I recommend HSQLDB for stand-alone Java applications.
When the data is to be accessed by multiple sources. Watch this one: it's often better to provide services than allow access to the raw data.
When the on-the-fly reporting and investigation tools of relational databases are of use.
When your customers have a DBA to maintain the data.

The Adept Object Database - How is it used?

Extend com.marringtons.object.DAO and any public, non-static, non-volatile data is persisted. Indexes are specified as inner classes that inherit from com.marringtons.object.Index. If you want to pass POJOs (plain on Java objects) around there is a package, com.marringtons.object.DTO, for converting data access objects to/from POJOs. It allows a many-to-many relationship between objects. Either can be converted to/from XML with classes from the com.marringtons.xml package. There is a com.marringtons.object.ObjectScraper class that converts objects to/from dictionaries of text. While primarily designed to filling and retrieving from screen forms, it can be used wherever dictionaries or maps are more useful than strongly typed objects.

Limitations of the Adept Object Database

Permanence: Once an object is recorded in a database, bad things will happen if the fields change order, are removed or are added to. Adept does include data migrations tools that can make most changes transparent to the end-user, but the risk exists because developers are used to having free reign with the internal fields of objects.
Maintenance: A modern SQL database has utilities and an interpreted SQL command language for a DBA to be able to review and investigate the database internals.
Reporting: The application must provide reporting tools since the database itself does not have the transparency SQL provides.

The Competition: Relational Databases

Corporate databases are now almost exclusively table based, sophisticated, optimised, relational databases with an SQL query/update interface. There are also a number of open source equivalents, with MySQL and Postgress providing engine support and HSQLDB as a Java library. Most, including MySQL, require licensing for commercial use. Most, excluding HSQLDB require complicated installation and connection shenanigans.

The Competition: Object/Relational Mapping

Hibernate and EJB 3 are the 2 most promising O/R mappers. They allow a developer to define an object that maps more-or-less directly to a database table. They provide all the facilities of a relational database while maintaining Java objects directly. Because of the complexity of the underlying database, the mapping layers are large, complex and non-trivial. People 'in the know' rave over them.

Personally I am less sure of the benefits. I have used direct JDBC, Hibernate and EJB 2 Entities in similar environments. Because it is best to keep the tiers independent, you end up copying the data into a DTO that more closely matches the service layer's needs than a single table would provide. It is often easiest, fastest and most obvious to do this with SQL and JDBC directly.

Of course the Adept Object Database drops data directly into the object - so the rules change for it.

Adept Requirements

2005-03-04T11:33:00.000+10:00

I developed the Adept library for my own learning and for my own uses. So I set the following goals for the project:

No compromises. I develop the software to my best ability without compromising quality. I've had my fill of working on projects where time and money overrule the principle need for good design and code.
Fully Documented. I provide ongoing full documentation as part of the code - line by line - and keep it up during updates and changes.
Full Unit Testing at all levels. Unit testing is a concept I've embraced, and I want to see how it fairs when fully implemented. See ;Unit Testing - Good For All in the Software Development Weblog for more details.
Generalise only when necessary. I believe that unless there's an immediate need for a generalised class, then it's better to make one specific to the problem. We should only consider generalisation when two or more classes need to share the same logic. See Code Generalisation - Do's and Don'ts in the Software Development Weblog for more details.

This means that when I see a weakness in my design, or a class or package can be extended to provide other needed functionality then I take the time to:

consider refactoring to generalize with superclasses or helpers;
design the change;
make the changes;
update calling code if needed;
add or update unit tests and;
update the documentation with examples.

Introduction

2005-02-28T11:09:00.000+10:00

My name is Paul Marrington, software developer. Well, just Paul Marrington, actually. The software development part, as you might have imagined, is what I do. Before you join the other 99% of visitors to any given web-page, why not take a moment to hear a story that has a good chance of hitting close to home.

I joined the software industry in its infancy, and by the early 80's I was spending my time exclusively developing software for myself and my customers. I was among those who saw the merits of micro-computers before they became mainstream, and so the PC was my bread and butter for many years. Shareware was the word of the decade, and each person owned themselves.

But - and this is the part that might sound familiar - a family and the necessities of life forced me to realise that the only customers who could afford to consistently support developers in their trade were the corporations, and so I joined the queue and was contracted out. For the last six or seven years my primary source of income has been working as an architect and development manager for corporate client/server systems - the mainframe'd masses - and it scares me how easily I joined their ranks, and how quickly my skillset began to be molded exclusively fit the Information Systems model.

However, working as a corporate development manager I came in contact with many developers who were highly interested in application development, and I came to see that a great deal of client/server architects would be developing shareware and freeware for themselves, if they could use the same techniques and tools that they were used to.

I've also watched designers develop HTML pages for the business to sign off and give to developers to be converted into xSP (JSP, ASP, PHP, etc) for implementation. Not only does that mean developers are wasting time in the conversion but more importantly, the designer cannot easily change the interface as the customer's needs change. I like the idea of the web page signed off by the client going into production with little or no change. That way the GUI designer can make visual changes without interference from development.

So, wanting to try my hand at application development again, and hoping to eventually provide the same opportunity to my colleagues, I decided to create a system for enterprise developers to be able to create shrink-wrap systems. Adept (Application Development - Enterprise to Personal Transformation) was born. I have divided my development efforts into two source trees. There is a support library in com.marringtons.library, while the Adept specific code is in com.marringtons.adept.

As yet only the Adept library has been released to open source on Source Forge under the name adze-db. Why the seemingly unrelated title? Because I'd originally decided on naming the system as Adze for Application Development with ZEn. I enjoyed the acronym, and liked the image of the carpenters tool relating to application development. Unfortunately, my son pointed out that the name Adze could easily be misinterpreted as one of the many spam-related publicity/advertising tools available, and I didn't want to place that stigma onto the project. So I changed the name to Adept, but the adze-db project had already been opened on Source Forge, so I decided to keep the old name for the library.

So, why adze-db? The world does not need a new generic library, but a simple effective shrink-wrap database was not available when I started this project. I decided on creating an object database. Like all my development efforts it is based in the KISS principle - so it is not nor will it ever be DAO complient. I think there should be 3 S's in 'simple' - one for simple to use, one for simple to maintain and one for simple to develop. The database ended up good for the first, OK for the second, but the third did not get a look-in. At best I built it up from well separated classes, most of which are coded for simplicity, but I was optimistic to believe that a database could ever be simple.

Lastly, why this blog? I have a separate blog for software development issues at http://marringtons .com/Adept/blog/Software.Development/. It's primarily there to discuss technical industry issues. This blog, however, is specific to the Adept open-source library. I will discuss various aspects of the system and how they will help you solve your problems, why I have developed them and the decision I made that may be of interest.