API Specifications FinSpec
There is little over a month to go now before the annual London FIX Conference. Skimming through the agenda, one session caught my eye:
"Operational Efficiency and FIX - a candid discussion. Can FIX Orchestra be part of the solution? If so, how?"
This is a topic close to our hearts at FixSpec, as we spend most of our days trying to understand the root cause of inefficiency in FIX, and how to improve it.
If you are new to FIX messaging in general, I should start by clarifying that the question isn't about whether FIX - the messaging protocol - is inefficient, but instead the processes around it. These processes typically involve large (often out-of-date) FIX API documents sent as PDFs, trial-and-error development and bug fixing, brittle "workaround" scripts and manual certification.
FIX Orchestra is a project within the FIX Trading Community which is aiming to define a schema that can form the basis of participants creating and exchanging machine-readable versions of their specifications instead of the large PDFs. FIX Orchestra is not an entirely new schema, but instead an extension of prior schemas from the organisation, with some new functions which I'll cover later. FIX Orchestra is not the only machine-readable schema available to the industry either; QuickFIX (XML) is the most widely used today, FIXatdl (XML) is common for broker algorithms, and FixSpec offers an open-source, JSON-based schema called FinSpec.
So what's the connection between machine-readable schemas and efficiency?
Well, the thinking goes that the typical FIX integration project involves a lot of either manual work today (searching log files etc), or re-scripting effort to adjust automated tools to understand a new specification or version of it. The problem is that since specifications are typically sent as PDF files, a human still needs to do that work - it can't be automatic. So by defining a schema to describe specifications in a way that machines can understand directly, you can remove the error-prone human from the chain and - hopefully - automate a variety of inefficient manual tasks.
So the question to be posed in the "candid discussion" at the conference is whether FIX Orchestra could be a key component in achieving that aim; i.e. is it suitable to be used as an input into newly-automated processes? Let's look at some of the key challenges to its adoption.
As I wrote back in the summer of 2015, there is a huge chicken-and-egg problem with all of these formats, in that they need to be broadly adopted by both the "sender" and "receiver". Brokers and venues will wait for significant demand before producing them, but the recipients won't demand them until there are enough available to warrant changing their systems to ingest them.
This explains why - in a world where multiple schemas already exist - there is no sign today of any one format achieving critical mass. QuickFIX is common (but far from universal), mostly as a by-product of firms using the open-source QuickFIX engine. FIXatdl has some adoption thanks to its unique ability to describe the preferred order-entry layout of the broker, which saves a manual design step.
It will be tough for ANY schema to achieve tipping-point critical mass without some heavy investment in tooling to create, distribute and consume these formats. This is something that the vendors I've talked to appear to have precious little appetite for today.
We've been talking to firms over the last year about SpecServer - our spec editing toolkit that allows them to maintain specifications in one place and then export to a variety of machine-readable formats (including FIX Orchestra) as well as traditional Word / PDF.
What has continually astonished me throughout these conversations is how FIX professionals think about specifications (both their own documents and those received from other firms). They are:
... an after-thought, chore or hassle. ... to be taken with a pinch of salt. ... something akin to marketing.
While firms continue to produce Word documents that are completely disconnected from the technology that they look after on a day-to-day basis, it is easy to understand why. The Word documentation is for reference only, and can't be used for anything "meaningful" to improve their jobs, and is de-prioritised accordingly. Mention machine-readable specifications and the conversation moves towards hand-coding XML, which is even more painful.
This is the complete opposite to the vision of machine-readable specifications, where the specification is the key that unlocks efficiency. It is the FIRST thing that should be written, and not the LAST. Errors in the spec should be treated as requiring an urgent fix, and not low priority to be fixed whenever.
What is more peculiar, is that within these same organisations you often find internal equivalents to machine-readable specs; development or QA staff are manually encoding specifications in scripts, code or configurations that could (read SHOULD) be replaced by machine-readable formats, or better still pulled in automatically from a web API to ensure they are using the latest image.
Changing perception is an extremely hard thing to do, but while specifications are considered a "dead-end" (as the current Word documents are), progress will be slow and painful.
What would make FIX Orchestra uniquely suitable as an input compared to the range of existing alternatives?
Well, the key difference between the next-generation schemas such as FIX Orchestra or FinSpec and the legacy schemas such as QuickFIX is the modelling of workflow - descriptions of how FIX messages are linked to each other, and the dependencies between them. Today, these workflows are typically captured in the text descriptions of a FIX specification, which are written by a human (who may not be formally trained as a technical writer) and read by another human (who may not even speak the same language). You can see why this is prone to error and misunderstanding!
By capturing and formalising workflows precisely, the hope is that greater levels of automation can be unlocked, both for the sending application to understand what it needs to send and the expected response, and for the API owners themselves through areas such as automating QA-testing to ensure APIs behave as expected. If done correctly, they could even be used to generate full-blown simulators of matching engines such as those delivered by Aesthetic Integration.
The problem is that while this is an exciting innovation, turning it into actionable code is much harder. Firstly, you are asking the same people who de-prioritise maintaining a Word document under (2) above today to document even more and to be even more explicit and precise in what they document. (I'm yet to meet a single person who is willing to take this task on, and the technology to reliably derive it from logs isn't there yet due to the need to correctly identify object states). Secondly, because such a formalised workflow schema has never existed before, we run headlong into the same chicken-and-egg problem raised in (1).
So where does this leave machine-readable specifications in helping to improve the efficiency of FIX? The brutal truth is that they are great ideas in theory, but are likely to struggle in reality. This is not a comment about the relative strengths or weaknesses of one schema or another -- it is a statement which I believe to be true of ALL schemas, absent some other radical shake-up of the way in which FIX connectivity takes place in the future.
Explicitly, machine-readable schemas are NOT the "answer" to FIX efficiency. They are just one small piece in a much larger jigsaw; we can't solve the problem without them, but at the same time they don't solve it on their own.
So what might some of the other jigsaw pieces be? Tooling to automate manual processes around QA and certification are undoubtedly part of the picture, but to the extent that there have been commercial solutions to address these for decades, that can't be the full answer. There must be at least one other piece of the puzzle that is missing that is not offered by any vendor today.
We think we have one idea (for another post), but please feel free to leave a comment if you have suggestions!