Data Migration Support

The Adept Object Database stores Java objects directly into persistent storage. It's not just the primary way of accessing stored data, in fact it's the only way. This makes for a fast efficient system with one major drawback - data migration. If you change the contents or order of persistent fields in a Java DAO or any of it's internal objects then all databases using those objects will fail when they attempt to use the updated code.

This is a common problem with databases. The resolution is called migration. In the relational database world the DBA will write a migration script. It's then a manual process to ensure that the migration script is run on production data at the same time as the new code is promoted. In the shrink-wrap software world it's the responsibility of the install program to recognise an upgrade and migrate the client data.

The Adept Library has a Migration class that does most of the work for you once you have completed some manual steps:

1. Recognise that a DTO needs to change. This isn't as easy as it sounds, since a DTO can have POJO fields - and if those fields change the DTO is no longer valid.
2. If it doesn't yet exist, create a new class called Migrations in the same package as the DTO that is to change.
3. Inside the Migration class create a public static class extending DAO with the same name as the class that requires migration. If the migrating class is itself an inner class, match the class path inside Migration.
4. Copy the code from the class being migrated into the newly created inner class as a further depth of inner class.
5. Change the name of the class just copied to something unique. I use the date or release number.
6. Make it static.
7. Have it implement Migration.Interface.
8. Implement the migrate() method that will return the new changed DAO.
9. If there is a previous migration for this DAO, change it's migrate method to return this DAO rather than the external one.
10. Implement the code changes to the original external DAO.

This is actually easier than it sounds. The best way, as always, is to provide an example:

// File DAOTest.java
public class DAOTest extends TestCase
  {
    static class TestDAO extends DAO
      {
        /***/ public String string;
        /***/ public int integer;
        /***/ public float real;
        /***/ public static class PrimaryIndex
            extends Index
          { /***/ public String string; }
        /***/ public static class SecondaryIndex
            extends Index
          { /***/ public int integer; }
        /***/ public static class TertiaryIndex
            extends Index
          {
            /***/ public int integer;
            /***/ public String string;
          }
      }
  }

.......................................

// File Migrations.java
public class Migrations
  {
    /**
     * Migration for DAOTest DAOs - note that these
     * are inner class DAOs.
     */
    public static class DAOTest
      {
        /** Test migration path */
        public static class TestDAO extends DAO
          {
            /** First on migration path */
            public static class V050511a extends DAO
                implements Migration.Interface
              {
                /***/ public String string;
                /***/ public int integer;
                /***/ public float real;
                /***/ public static class PrimaryIndex
                    extends Index
                  { /***/ public String string; }
                /***/ public static class SecondaryIndex
                    extends Index
                  { /***/ public int integer; }
                /***/ public static class TertiaryIndex
                    extends Index
                  {
                    /***/ public int integer;
                    /***/ public String string;
                  }
                
                /**
                 * @see Migration.Interface#migrate()
                 */
                public DAO migrate()
                  {
                   /*
                    * Originally created DAOTest - 
                    * changed when new migration added
                    */
                    V050511b testDAO = new V050511b();
                    /*
                     * Use DTO deep copy since objects
                     * will be so similar.
                     */
                    if (dto == null)
                      dto = new DTO( this, testDAO);
                    dto.copy( testDAO, this);
                    /*
                     * This is where we would do the
                     * processing that is the essence
                     * of migration. If we added fields
                     * we may need to fill them here.
                     */
                    testDAO.integer++;
                    /*
                     * Lucky last - we return the newly
                     * created replacement DAO.
                     */
                    return testDAO;
                  }
                private static DTO dto;
              }
              
            /** Second on migration path */
            public static class V050511b extends DAO
                implements Migration.Interface
              {
                /***/ public String string;
                /***/ public int integer;
                /***/ public float real;
                /***/ public static class PrimaryIndex
                    extends Index
                  { /***/ public String string; }
                /***/ public static class SecondaryIndex
                    extends Index
                  { /***/ public int integer; }
                /***/ public static class TertiaryIndex
                    extends Index
                  {
                    /***/ public int integer;
                    /***/ public String string;
                  }
              
                /** @see Migration.Interface#migrate() */
                public DAO migrate()
                  {
                    com.marringtons.object.DAOTest.TestDAO
                      testDAO = new
                        com.marringtons.object.DAOTest.TestDAO();
                    if (dto == null)
                      dto = new DTO( this, testDAO);
                    dto.copy( testDAO, this);
                    testDAO.real++;
                    return testDAO;
                  }
                private static DTO dto;
              }
          }
      }
  }

 

The Act of Migration

The worst migration case I've personally experienced was with an accounting program. When upgrading a 5 year old system it required me to install two complete intermediate versions of the software, migrate to them in sequence then install the latest version and migrate a third time.

Adept migration is more transparent. Your setup code must call Migration.database(), giving it the name of the database to migrate. It will review every persistent class and record the operation, so only the parts of the migration path not yet performed will be done. As an example: suppose over 10 releases, a particular DAO has changed 5 times (shame on you). Any user could be upgrading from an earlier version to the latest at that time and migrate again later. The migration code will account for this and pick up the migration thread from where it left off the last time the software was upgraded.

Migration Limitations

1. It only works with classes that inherit from DAO. If a POJO field inside a DAO changes, you will need to create a new POJO with a new name and migrate from the old to the new manually in the DAO migrate(). This is a good reason to inherit from DAO for any persistent non-trivial data object.
2. It must be called by a single threaded setup routine to stop any other possible simultaneous access of the database being migrated.
3. The developer is responsible for providing a migration path before changing a DAO or a POJO inside a DAO. This goes against the practice of changing internals without concern for effect. If you have application specific POJOs inside DAOs, comment the source so they will not be changed by accident. Better still, have them inherit from DAO or copy them to a DAO as part of the persistence code.

String Formatting - A Step Towards Internationalisation

The Java alternative to the C printf() function is java.text.MessageFormat, a rarely used class to create parameterised strings. It is, in fact, considerably more powerful than printf, but whereas a C programmer would regularly use printf as they standard string output method, few Java programmers even know they can format text easily. I can't count how often I've seen Java classes with code specific to padding numbers or adding an s for plurals in a string.

Why? The three reasons in order of excuse are:

1. Most developers don't know that a text format class exists. It was introduced in a later Sun release in 2003 and doesn't appear to be a common part of basic Java training.
2. It has a complex syntax for both code and template string that are hard to remember.
3. The syntax isn't very readable.
4. And if a developer actually looks at the implementation, he/she will realise how expensive it is. Every time you create a new MessageFormat instance it compiles the pattern. This means that the best way to use it is with static final in the class. I dislike this as it separates the message from the code, making the code less readable.

The com.marringtons.string.Format class is a wrapper that resolves all but the first of the problems listed above.

1. We can't help here. Since the wrapper isn't part of the base Sun Java library, it is unlikely to ever be included in general training. By meeting the next 2 challenges, at least it can be used as a project standard if the principals of said project deem it so.
2. MessageFormat is created with a new and a format method to produce the formatted string. This isn't very readable when used in-line. Format uses a static factory method called pattern(). MessageFormat also has a static format method, but it's very expensive to use as it recompiles the pattern every time. The patterns are hard to remember. I have created examples of each below for your reference.
3. MessageFormat takes an Object array for parameters, while Format has a with() method for objects and primatives.
4. The Format.pattern() factory method is the only way to create a format. It caches the MessageFormat instances so that they are only compiled once per program run.

Enough words - time for an example of formatted strings both ways. Both ways below will produce the same result.

private static MessageFormat myMessage
  = new MessageFormat(
    "Result is: Integer {0}, String {1}");
...
System.out.println(
  myMessage.format( new Object[] 
    { new Integer( 12), "twelve" }));

// is the same as:

System.out.println(
  Format.pattern(
    "Result is: Integer {0}, String {1}")
    .with( 12).with( "twelve"));

 

Format and MessageFormat Pattern Templates

The Sun documentation is correct, but not very clear. The best references are by example, so I'll provide an example from my unit tests that show each format type. The check method takes the pattern in the first parameter and the value in the second and tests then in the expected result in the 3rd.

check( "{0}", 1234, "1,234");
check( "{0,number}", 1234, "1,234");
check( "{0,number,integer}", 1234, "1,234");
check( "{0,number,percent}", 0.16, "16%");
// @see java.text.DecimalFormat
check( "{0,number,#,##0.00;(#,##0.00)}",
    -1234, "(1,234.00)");

Calendar calendar = new GregorianCalendar(
    2005, 2, 13, 20, 38, 55);
Date date = new Date( calendar.getTimeInMillis());
check( "{0,date}", date, "13/03/2005");
check( "{0,date, short}", date, "13/03/05");
check( "{0,date, medium}", date, "13/03/2005");
check( "{0,date, long}", date, "13 March 2005");
check( "{0,date, full}", date, "Sunday, 13 March 2005");
// @see java.text.SimpleDateFormat
check( "{0,date,yyyy.MM.dd G ''at'' HH:mm:ss z}",
  date, "2005.03.13 AD at 20:38:55 EST");

check( "{0,time}", date, "20:38:55");
check( "{0,time, short}", date, "20:38");
check( "{0,time, medium}", date, "20:38:55");
check( "{0,time, long}", date, "20:38:55");
check( "{0,time, full}", date, "08:38:55 PM EST");

String choice = // @see java.text.ChoiceFormat
    "{0,choice,-1#negative ({0})|0#none|1#one"
 +"|1<{0,number,integer}|101#greater than 100}";
check( choice, -23, "negative (-23)");
check( choice, 0, "none");
check( choice, 1, "one");
check( choice, 11, "11");
check( choice, 111, "greater than 100");

 

Why Internationalisation?

You may have been wondering why the title of this article proclaimed MessageFormat as "A Step Towards Internationalisation". There are many aspects to internationalisation with respect to software development, but translation of text would have to be one of the core ones. I'll discuss all the options as the subject of another essay, but my preferred method is to develop with in-line strings as part of the code where applicable, then use the English strings as a key into a dictionary of translated text when it is needed. This will only work if you use a Format class to separate templates from variable data. The same restriction applies to keeping messages in a database or configuration file.

In short, using

message = "I think there are "
    +aNumber+" people in the group";

 

cannot easily be translated, while

message = Format.pattern(
  "I think there are {0} people in the group")
  .with( aNumber);

can.

Yet Another Date Object

Sun really mucked up date processing in Java. First there was java.util.Date - a great little class that did most of what anyone would want with dates. Except that - oops - it didn't account for internationalisation. Enter the Calendar object - big, clunky, hard to use - but able to do everything. Now, deprecate almost everything of use in Date, but have it still tied to the hardware while Calendar is not. Fun.

The Adept library also includes a Date class that provides easy-to-use facilities where a Calendar object is unwieldy and overkill. Like java.util.date it keeps the date and time in Epoch time. This is a fancy name for the number of milliseconds since the first of January 1970. And, no - there will be no Y2K-like issues in Java since it is a long, being 64 bits long. It was only the old C programs where a long is 32 bits that the time will roll over in about 25 years.

Like the String class, the Date class is immutable - with all the additional benefits an immutable class carries with it. If you're scratching your head here, immutable simply means that it cannot be changed. To quote Brian Goetz of IBM, "Immutable objects have a number of properties that make working with them easier, including relaxed synchronization requirements and the freedom to share and cache object references without concern for data corruption". Get the whole story here.

A Date object can be created with the current time, an Epoch time, a Calendar or an internal Date mathematics function.

Once you have a date, the class allows you to:

1. Change it to other forms: such as seconds() for Epoch seconds - being a date, accurate to the second, that fits into an int (32 bits); timeOfDay() which then returns a Time object for more granular work; calendar() for dealing with the rest of the Java world; or timestamp or dateStamp() for the date in a reverse order string (20050425 for April 25, 2005) that can be used for lists or filenames that need to be sorted by date.
2. Date/Time Maths: being add() to add a time interval (days, hour, minutes, seconds) to a date or next() to set the date for a specific time of day (as in 5pm in 3 days time).
3. Date Terms: being able to add or subtract a term from a date or to be able to extract a term between two dates. Very useful for loans, leases and other similar software requirements.
4. Days in Month: I've often found a need to work out the days in a particular month for various scheduling or display purposes. For this reason I've exported a static method that provides this information, given the year and month.