Building Custom Solutions vs. Buy-and-Build Software

The Challenges of Building a FIX Protocol

The first day I was introduced to FIX was when I worked at an investment bank in London as a developer, I was told to write a feed handler to retrieve market data. Bear in mind that at this time, I knew nothing about FIX, apart from googling it for about 10 mins on the internet. With a touch of overconfidence and slight arrogance, I set to work coding a direct socket connection to the remote FIX endpoint, thinking, “How hard could it be?”

So, What Is FIX? 

Financial Information eXchange (FIX) is both a market data format and a protocol: it is used by investment banks to place orders and receive market data and has become a global language in financial trading. The format of a FIX message controls how it is encoded. All FIX messages start with 8=FIX, which denotes the start of a FIX message. They then go on to list key and value pairs. The keys are represented as numbers (known as TagNumbers) followed by a = delimiter to delimit the values. Each key=value combination is then delimited by the \u0001 character, which is sometimes visually represented as either ^ or |. The value is often written in a semi-human-readable format. I say semi-human readable because most of the time it is human-readable, but all too often, FIX will use a single character to denote a state or type of message. These characters are not always that obvious. I agree that the character B for "Buy" and S for "Sell" makes sense, but other characters are used that make no sense. For example, D denotes a "New Order Single" message, which is a message that is often used when you wish to place an order with your counterparty. 

Image 1: An example FIX message in FIX format

When it comes to the FIX protocol, you have to be careful, as it turns out to be much more involved than just having the correct FIX message format and being able to decode the key=value pairs. The protocol also includes ensuring the counterparty has successfully logged in and that any missed messages are retransmitted. It also requires support for gap fills and sequence resets. 

Back again to the coding: The first challenge was to get past the login phase. It looked like this login message was mandatory! Over time, I managed to put together enough of a login message that the other end came back with a login reply, but it was not long before my colleagues started taunting me, saying, “Why don’t you just use Quick/FIX4J?”

Saving face and trying not to show that I knew nothing about QuickFIX, I quietly down my IDE and jumped back onto the web browser. While doing a bit of research about QuickFIX, I was thinking, “Perhaps using this open-source library could be simpler than implementing it all myself," and, “Maybe it’s going to take me longer to understand how to use this library.”

The library was extensive, and they clearly put a lot of work into it, but I was overwhelmed by how complicated it all looked. I thought, "Does it really need to be like this?"

At the time, I was working alongside a team of developers that I looked up to. These guys stood by the “no code is the fastest code” mantra. Everything they wrote was super optimal: they would work with raw byte buffers rather than create unwanted objects, and their Java code looked more like "C" code than any Java I had ever seen before.

So, I got back to my PC and proceeded to implement the lightweight raw byte buffer parser route. Over the next few weeks, I was starting to get end-to-end data going through the FIX adapter, and when I was finished, it screened.

Now looking back on those early days, I ponder,  "Did I make the right decision? Should I have implemented it myself?" The answer is, I think, yes. It was best to implement the custom simple, lightweight solution that I ended up with, but I only got away with it because I had an extremely simple FIX use case. I was only connecting a single session to a single counterparty with a single version of FIX and, apart from the login and heartbeat messages, just a single message.

My problem would have been if later this was to be upgraded to another FIX version, we had to support resend-requests, sequence resets, and gap fills, or offer production support and a dashboard where administering existing running sessions on the live system.

Image 2: Example dashboard for FIX sessions

At this point, it becomes a serious undertaking, and it starts to make more sense to turn to a tried and tested solution, already with all these bells and whistles, to use products with the ability to search historic FIX messages: 

Image 3: Searching for all FIX messages with a given client order id

Or, to raise alerts when sessions disconnect:

Image 4: Alerts being raised in FIX UI

Not long after, I joined a company founded by Peter Lawrey. Peter is one of those super-optimal developers I mentioned earlier. He developed the widely used open-source software Chronicle Queue and Chronicle Map. Together, we set to work and built such a FIX engine that has all these bells and whistles, and in doing so, we also built a company that has grown to a team of over 20 people.

Looking back, developers can save time and effort using a buy-and-build software solution. As mentioned, I would have been stuck if we had wanted to provide production support with my simple solution. Moreover, I certainly would have needed more time to build a UI. The lesson learned was: “Sometimes time spent reinventing the wheel results in a revolutionary new rolling device. But sometimes it just amounts to time spent reinventing the wheel.”