Gemini Issue Tracking – Mailbox Processing

One of the reasons I have been very quiet on the blog is I have been extremely busy launching the Gemini Mail Box Processor tool. It is an add-on to Countersoft’s Gemini Issue Tracking / Project Management tool whereby you can submit issues and comments via your email account.

They have a basic version which will allow you to create only new issues from a pop3 box and even then you have to pre-specify the issue type, component etc. So if you think this is restrictive, GMBP should be a breath of fresh air for you.

GMBP will create new issues and also comments from the inbox; let’s take both in turn.

New issues

When a new email arrives, it is imported, along with attachments, into the GMBP database. From here, it is processed into Gemini. This process is complex and will try a variety of comparisons to ascertain the correct Project, Component, Issue Type, Priority, Reporter etc. The first thing it will try is to Smart Match the email against the Gemini data. Comparing text in the subject and body it will predict the most suitable values based on the content of the email. If it fails to find a [unique] match, then it will resort to the defaults set per project, and/or pick the first item (eg component) from the list. These are all based on the GMBP settings.

A failed match will be put into Pending state for review. Adding text to the subject (such as a component name) will ensure pending emails can be processed accurately.

New Comments

Unlike the built in Gemini processor, GMBP will allow comments to be created from the email. If it contains a gemini project code (eg [ACME-23]) in the subject line, then it will be treated as a comment and attached as a comment to Issue 23. This is actually a simple case, as no matching is required, it is merely appended as a comment to the issue Id specified.

Other Features

Truncation

Truncation is another cool feature of GMBP. Email threads can become very long and cumbersome and the last thing you would want is pages of Gemini issue with entire email threads. Truncation will truncate the email at various points with the option of including the full contents as an attachment – so nothing is lost!

GMBP will automatically truncate text above it’s ----Please reply above this line----- marker, but you can include others. Regular expressions and fixed values can be searched and truncated at that point. For example, you could add an expression for ‘From : support@domain.com’ which could be the top line that appears when people reply to the Gemini notifications. Then, only the actual content of the email will appear in Gemini.

Security

If this was not exciting enough, there is more. GMBP comes with an administration website, and only those people in Gemini Administrators security group (or another group you specify) can access the website. Furthermore, there are options of Blacklisting certain email addresses (again specific, or regular expression) to prevent matches being able to post into Gemini.

Convenience

Since it is possible that people have many email addresses, such as a work and home email, they might want to make requests to Gemini from any email. It is possible to set up aliases to link numerous email accounts to their Gemini account. Irrespective of which email address the issue is received from, it will still end up being reported from the aliased Gemini account.

Also, Out of Office reply notifications can be configured to be ignored; preventing pointless issues being created.

Summary

If you need to get an Issue Tracker / Project Management application, then I would fully recommend Gemini from Countersoft as a serious contender. Combined with the GMBP I don’t think there is much extra you would want!

NHibernate Querying across Composite Keys

Error Message: ‘The multi-part identifier “manager1_.MAN_ID” could not be bound’

Firstly, let me say I do not like composite keys. Unique constraints work much better. That said, NHibernate does have support for Composite Keys, for these legacy databases, and it is fairly intuitive with the intellisense. However, I have come across a bizarre issue when querying across these relationships.

Database Diagram

This is the database diagram. Nothing too special about it. Projects have Managers and managers can manager numerous projects.

It is worth saying that the same issue occurs with a two way composite key as well as this three way version. The project roles could probably be ignored since the ‘role’ is stored in the Managers table.

Classes / Mappings

One thing to note, that you have to override the Equals and GetHashCode methods to use with NHibernate.

Project

again some lines have been removed for brevity.

ProjectMember

This is quite interesting example of the composite-id mapping. There are three that make up the key and all are many-to-one elements that map to classes in their own right.  

Manager

Some of the above namespaces have been blurred for client confidentiality.

The Problem

I am currently working on a large search mechanism across various fields of the application. I am basing the search on the blog from Ayende Rahien so I create a DetachedCriteria object for the Project and then everything is based from that.

The method in question is as follows:

A little explanation. criteria (IAdvancedSearchCriteria) is my class which holds the search data. In this case, it is the LineManagerIds pulled from a multiple select box.

So this code runs and is then executed against the current NHSession and using NHProfiler I can review the SQL being generated:

If you remember, this should link projects –> Project Members –> Managers in a Many to Many style situation (although it is mapped as Many to one for the audit information)

Look at the inner SQL and you can see there is no mention of the Managers table. Why? I created an Association to the Manager object. It merely results in a list of errors like ‘The multi-part identifier “manager1_.MAN_ID” could not be bound’.

I assume it is something to do with composite-ids (Did I mention I don’t really like these…)

the association to the Manager is mapped up in the <composite-id> section of the mapping file. Since I have made many searches before, I deduce there is a high probability that this may have something to do with it.

So I assume that NHibernate is not using the Class association in the composite-id section. I could either re-design the database to remove the Composite Id (Tempting… did i mention i hate them?) but my remit is to leave the database alone…

So, to fix the problem, I help NHibernate out. If I add another <property> to the ProjectMembers mapping file for the Manager, I can make the query work.

Now, an important thing to note. I assume, NHiberate will do the inserting and updating based on the <composite-id> mapping, so I have set insert and update = “false” otherwise I think we’d have some confusing errors / SQL statements. I will update this post if this proves incorrect! but it has solved the searching mechanism as shown:

As you can see, we are now joining in on the managers table. There was no change to the code, just the mapping file.

NHibernate – When ‘Load’ does ‘Load and Update’

I came across a truly bizarre update occurring within my Gemini Mailbox Processer Application. It was reported to me by a friend who did some monitoring on the application. We were both completely baffled why NHibernate would be doing any form of update on a simple List<IEmail> call. Let’s look at the details.

Firstly, let’s see the interface which defines the class. I have removed some fields for brevity.

 

Then here is our code which will load the Emails. Here we create our criteria object, and add some filtering, some ordering, and then retrieve a list of IEmail objects. Nothing seems strange here.

This is the mapping (abbreviated version) to the email class. You might spot the issue in here (but I’ll spell it out later)

Ok, so running this, let’s use NHibernate Profiler to see what is going on.

As you can see, there is a bizarre update statement. At least NHibernate was batching it up into one statement, but WHY! Let’s look at the specific details for this query. Again some items have been removed to keep the snippet small.

I decided to comment out lines in the mapping and run a test app to see what results were returned. Each time I ran it until I saw this update appear. Trial an error narrowed it down to the Priority column.

We can see the priority is looking a little strange. In the interface it is an enum, and in the mapping it is specified as an Int32. So it seems strange that NHibernate it sending it as Text? Let’s remove the Type=”Int32” from the mapping and see what happens.

Rerunning it shows it is all ok now. So while I do not know why, I do now know how to prevent it. I assume it is to do with the type conflict between the enum and specifying it as an Int32.

Windows Live?

Apparently you can make posts with Windows Live. It seems fairly straightforward to do.

Just enter your URL, your user name and password and it logs in and downloads stuff. We will see if this post makes it to the blog, or I’ve just compromised the security of the blog :-)

Could not load file or Assembly CustomMarshalers.dll

I had an interesting error after installing VS2008 and trying to create ANY new project. I tried, web, console, services, and none would be created. Each time I had the following error:

Could not load file or assembly CustomMarshalers, Version=2.0.0.0 etc

For some reason, this file was not installed into the GAC

c:\>gacutil -i C:\Windows\Microsoft.NET\Framework\v2.0.50727\CustomMarshalers.dll

command into the VS2008 command prompt solved the issue.

Useful Links for New Projects

NHibernate Download
NHibernate Homepage

NHibernate Profiler

Resharper

NVelocity and Engine Wrapper


Template Web Project Structure using NHibernate and NVelocity

Validation and Testing

We all know that validation of user input is always a good thing. After all, even the most astute of user, is still at the end of the day, a user and that means if there is a way to cock something up, they normally find it... So therefore we validate what a user inputs to make sure it is sensible.

So when I read this article on the BBC news site http://news.bbc.co.uk/1/hi/world/americas/8152278.stm I was left pondering quite what had gone wrong. For those that don't want to read the article, it is basically about a man being charged $23 quadrillion for a packet of cigs! Maybe America are trying a new tactic of health awareness, preventing people from smoking? We'll probably not, since that amount is several times more than the entire national debt of the USA!

Just how could someone be charged so much, and just how big was his credit limit? I want one of those cards! I did find his response of "Something I could never, ever, afford to pay back. " was very astute. If he worked 8 hours a day, for 364 days of the year (well let's have Christmas off!) and for 20 years, he would need to earn $397,473,477,132 per hour. that's $400 Billion an hour.

I want that job :)

Thankfully, the bank has rectified the "blip" and even refunded the $15 overdraft fee. How kind of them :-) perhaps some more unit testing would be beneficial here, it would be most unfortunate if the transaction credited him that much!

Microsoft's Web Standards

Just a little light humour from the microsoft website...

Oops!

NHibernate Parameter Error

We were getting an IndexOutOfRange Exception when trying to save a Period object in our application. Invalid Index 17 for this SqlParameterCollection with Count=17

It was a little confusing as we'd not seen any error like this before:




After a bit of head scratching, we interrogated the mapping file, see if you can spot the error:



Did you spot it? Well, there are two columns called "MORId", one in the main class, and one in the component AgreedWorkLevel. The important thing to remember is that components are stored in the same table and although they are in different classes, they need to be uniquely named.

I would imagine if you wanted the component property to be the same as the class' property, you could link this in code and leave NHibernate pointing to the single reference point.

NHibernate Querying - Getting list of null objects

This weekend we had a very bizzare error on NHibernate. We did a query, nothing clever in it, but we got a list of null's back. The length of the list was the expected value but each item was null.

IList result = PeriodFactory.GetAllPeriods(person);
if (result[0] == null)
Throw New Exception("Item Was Null!");

It was throwing the null exception. So what was NHibernate doing?

Well, let's look at the mapping file:

It all looks OK. It certainly compiled and ran OK. So maybe it was something to do with the code querying the Periods:

p>Well, doesn't seem to be anything wrong here. Maybe NHibernate is not generating the SQL correctly. So using the NHibernate Profiler I checked out the SQL being generated:

Well, even this looks fine. Things are getting very strange! What could be going wrong. NHibernate has worked fine for the last 30 odd classes we'd persisted. What was so strange now? Well, another feature of the NHProfiler is to view the results that are returned. So that was the next stage:

This was the first clue we had. Notice the first column, it is blank, yet it is suppose to be the primary key. So what happens if we try loading a record based on the primary key. This is the most basic of operation, therefore surely this will work. A little hacking of the code for testing:

Three guesses for what result was? you guessed it, sweet Nothing, value 3 did exist in the database, so this should have worked. NHibernate was clearly not being able to create the object. So the query brought back the right number of results. The Get did not. Let's take a closer look at the table structure and compare the primary keys:

Wait! what is this? we have a column called PeriodId (highlighted) that is not the primary key. What is going on here, where did that column come from?

I rechecked the mapping file to located any stray PeriodId's but the only ones were the Foreign Key values which correctly are in the Referenced Table. Then I remembered SchemaUpdate. It was updating the table based on the schema's. My naming convention was class has PK of "Id" and the database was "{tablename}Id" I'd added the column attribute at a later stage and the Update had obviously updated the table to include the new column.

This must be considered a bug in the schema update as it should have updated the primary key and not created a random new column.

With thanks to my trusted memory (eventually) and the great NHProfiler we've managed to get to the bottom of this frustrating issue. Moral of the story, be careful with what the schema update is actually doing...

Virtual PC Installation woes

Virtualisation is a great technology. Not only can we test installs of applications against pretty much any operating system, we can also test websites against different browsers (such as IE 6, 7, 8) which do not normally run side by side.

However, I came across a rather annoying, undocumented feature of Microsoft Virtual PC 2007 this week. Having created a VM on my laptop, I wanted to move it to my main PC where it's final home will be. I am running Vista Business (no comments pls) and when I tried to install it I received an error product.cab is corrupt. I first thought this was a bad download, so I redownloaded it - same result.

Bizzare. I then tried the Service pack 1 version, after all this should fix lots of issues, and maybe this would be one of them. Sadly, no. I got the same error.

I then tried to download the version via my MSDN subscription, rather than the public download. Sadly, same result.

I then decided, I would just install VMWare's player and convert my MS VPC to a VMWare version. Sadly, I had a similar error when trying to install VMWare Player.

I'd not had this issue before, so was now quite stumped. I had been trying to install this over a remote deskop connection, so when I returned home, I tried installing it locally. I got the same issue.

Just before giving up, I remembered that old, rarely known trick... I restarted the computer. Low and behold, the next time I tried to install it, everything was fine!

So if you are getting this error, don't be afraid to try the good old restart!

Code Monkey on Speed

Recently, I was on an NHibernate course, with Ayende Rahien, a prominent figure in the NHibernate world. He demonstrated all the various mappings and code needed to use NHibernate. I was seriously impressed with is ability to write code and more importantly, the speed at which he did.

Usually, I am very good at following what people are doing and learning that way. Ayende raised the bar a level with his speed of typing and of code refactoring. Super human powers to refactor code, way beyond your rename and copy/paste tools available. Then I remembered the good old C Resharper addon that you can get from http://www.jetbrains.com/resharper/index.html

I have been using it this past week (you may have noticed the slight lack of posting! I've been busy playing!) and am fully amazed with the flexibility and speed increase in development I have seen.

From my previous post about Oracle and Specifications, this week has been very much about creating a whole new business layer for our application, on that will use NHibernate to connect to both SQL server and Oracle. All my business objects will implement and interface (as all good business layers should!) and Resharper has allowed me, with a few key strokes to:

• Create the concrete class
• Make it public
• Implement the interface (with public virtual auto properties) - NHibernate requires virtual properties
• Move the concrete class to a file of it's own name
• Manually dragging the class into it's own folder structure
• Resharper then offers to update the namespace to match folder structure and adds an using statement where needed to refer to the old namespace

With the addition of automated code snippets (i made an interface property snippet to save even more time) I have created half the application in 3 days!

If I didn't have to buy a license for Resharper, I'd definitely buy those guys a pint! Maybe they deserve the pint anyways, it certainly has saved *lots* of time. Try it out and see what you think.

It's all in the specifications

Some programmers are happy to dive into the code and get programming straight away. After all, prototypes are useful things, even if they do become production code without much further attention. Are there any risks of this approach, and should developers dive right in?

At the company I work for now, they have developed a fairly complex web application in a short time. I am actually quite impressed with how much has been achieved in about 2 months. I would have estimated it was at least a 4 month development time scale. Has the jump in approach worked in this case?

As a small company, the pressure was to impress the client who set impossible deadlines. The team worked tirelessly and amazingly quickly and produced the application within the deadline and the customer is happy. So this is all good right? Well, let's take a step back and look at what the requirements for this application were.

Right from the outset, the platform that was being developed; to be used by this first client on Oracle. The client uses Oracle, and wants to host the application on Oracle. However, the application is developed using MS SQL Server. Surely, whenever you are starting a new project, you must look at the requirements and develop accordingly. Since this platform is to be reused, clearly, as a company, we want to use MS SQL Server as it is where our strengths lie. So, therefore, the requirement MUST be that the application must be able to communicate with MS SQL and Oracle databases.

Sadly, this basic observation has been missed. Due to the speedy coding of this application, it seems little care was taken in thinking how it was to be installed, on Oracle, at the client's offices. In fact, the technology used was LINQ to SQL. As you may be aware, it is really "LINQ to MS SQL Server and nothing else". So my task for the next 2 weeks is to find a solution to the problem that was known at the outset. We've gone down one path, and now I need to find a solution that will help us use both SQL and Oracle servers.

Options Available:
- NHibernate is an ORM mapper, like LINQ to SQL (but better, richer and DB Server Neutral)
- LINQ to Oracle Component.

NHibernate ORM
This is a powerful option, but the specifications state that they are using Oracle 11g (helpful to read these!) and NHibernate currently seems to have Dialects up to 10g. While it may be possible to work around this (there can't be THAT much difference between the two versions!) we must prove this concept and not just assume it will work. This is an important lesson to learn before picking any technology. It must solve the problem/meet the requirements.

I have created a test application which works with SQL 2005 and now I am having issues connecting to the Oracle. Working out the configuration is the first hurdle, (I think this has been solved, i will post how it all works when complete) now it cannot find the tables. This is probably my lack of Oracle knowledge, than an NHibernate issue.

Should I get this working, it will mean a complete re-write of the data access code. Oh Linq to SQL uses it's own generated classes which I might be able to steal and remap, else it might be worth re-writing. Since each developer has their own dbml in their projects, and code sharing is not a technique they feel is worth doing. I will have to rewrite 4 Data Layers (admittedly it will be with ONE! single shared DAL dll.

LINQ to Oracle
So if NHibernate is so much work, surely this option sounds the best. It might be, and it is an option I will consider carefully. Link to Oracle is a 3d party component which we'd need to buy (circa £1k). We would also need to rework some of the classes / namespaces and would want to create a single Data Layer dll. I will also need to remember the requirement to connect to BOTH SQL Server and Oracle, else we'll be stuck with an application only able to talk to Oracle.

One option could be to abstract the methods to an interface, and have an Oracle implementation and SQL implementation, and load the correct one up depending on a config setting. NHibernate would handle this automatically.


Conclusion
Nothing has actually been concluded yet. Both options involve lots of (avoidable) work and again with time constraints, the most elegant solution might not be the best. This must be complete in two weeks and could have been avoided if the requirements had been thought about.

So I asked the question, whether developers can jump into code. I think the answer is No. There must be some time spent considering what is required. Even if it is a short time, some time is required. This lack of thought/assumptions will result in longer development time and headaches further down the line. Agile development will only work if you keep your eye on where you need to be otherwise, you can be going down the wrong path from the very beginning, with little or no way of getting back on track easily.

There is an old saying:

If you fail to plan, then you plan to fail

Memory Issues!

I mentioned in my first post that I work with a small business server for a church. This was purchased around 4 years ago and is a Xeon processor with 512MB of RAM. As you have probably thought from the post title, 512MB, are you mad? Presumably it seemed adequate at the time, but currently it is running slower than a snail in ball and chains.

So clearly, the option is to upgrade the memory, but this is actually proving to be more of a challenge than you would think. Even purchasing an identical spec chip copied from the label of the current memory chip, it was not compatible. Maybe this is so you HAVE to go back to Dell to get the upgrades, certainly if this is the case it would make cunning strategy for future business!

I have purchased as returned three sets of memory chips, and saverstore has been great in accepting these returns back. So what is the next?

Next is time to upgrade the machine. Currently, am looking at a dual processor, quad core, 8GB memory (don't want to get caught short again!) If you have heard of VMware ESXi my current plan is to create a new Virtual server on this platform. Virtualisation has many benefits, ability to create snapshots, host multiple virtual machines on the same physical hardware and adjust the hardware resource between virtual machines.

It looks a good option for pro's and also self-taught IT Monkeys like myself. Without any formal training I can be assured that before making any critical changes, I can take a backup (snapshot) and if things go wrong, I can revert. Simple.

First Entry!

For the first entry, I thought it would be worth explaining why I feel the need to post more information into the world wide web. Well as an active IT user and software developer aka IT Monkey all the issues I have run into I'm sure someone else might also run into at some point.

I have been involved in many areas of IT and found the internet a great source of information and helpful people willing to spend time to try and fix issues and problems. It is great to think that someone across the world who you have never met is willing to help.

I have been involved in the following areas:

• Server Administration - I have been running a 2003 small business server for the past 5 years for a Church. This has included, exchange, ADUC, File server, IP Routing. I have also recently looked at Virualisation and am currently running a 2 quad core, 8gb ram, server running VMwares ESXi Hypervisor. (Very Cool!)
• Programming - I have qualified as a Microsoft Certified Technical Specialist in web development and have experience in using VB, C#, Ajax Tool Kit, NHibernate, NVelocity, and many other areas.

I hope that I can publish answers and solutions that people would find both useful and also intelectually provoking. We'll see where we can take this!