Financial markets have been using distributed systems for a number of years to satisfy the demand for processing and storage. In fact, the size of the computing grids currently used in finance would make many research laboratories green with envy. However, in a distributed environment, the building of applications is totally different. Programming needs to be adapted for distribution.
Whether for processing or storage, using distributed systems is not simply a technical task. It very often requires a deep understanding of the nature of the calculations or the structure of the stored data.
Experience has shown that, despite a very high nominal power, a distributed system can become less efficient than a single machine if unsuitable applications are used.
These issues therefore lie at the heart of the engineer's role. It is essential to understand the purpose of the application, its functional aspects and its architecture, its technical aspects and their impact on the code to optimise the whole.
Typical projects involving a distributed environment are:
Using a computing grid for Monte Carlo Simulations
Using a grid for other calculations where parallelism is possible
Optimising the use of the grid to reduce processing time
Optimising the data transiting between the servers to reduce the required bandwidth
Implementing a distributed cache
For processing, the systems no longer need more machines but more cores. The short term future will involve using these cores. When calculating risk, for example, the processing capacity of the grids will make it possible to refine the indicators and more easily control market and counterparty risk. This is clearly an issue that will count for banks over the coming years.
As for storage, the volumes will definitely continue to grow. The financial markets are still hungry for data and every day they generate thousands of teraoctets that are added to an already considerable past history. Using this volume of data has already been clearly identified by players in the financial markets as a major challenge if they wish to remain competitive.
Real time data management
Real time systems are programmes that must guarantee very short response times in the order of a millisecond. In financial markets, activities such as market making or arbitrage require real time processing of the data received from the markets. The slightest time-lag in relation to the market price leads to losses and the best players in these activities are often the quickest.
“Real time” is a programming issue in its own right. It requires special techniques that are different from traditional programming. The whole data processing chain must be designed for real time and simply adapting a part of the code is not enough.
With real time, there are also infrastructure and network issues. These issues have to be recognised and understood in order to design effective real time systems.
Financial markets often need high speed applications. Our consultants are very often required to work on high performance applications that need to be optimized on a continuous basis.
True real time systems take things a step further. The longer the processing time, the less useful the data generally becomes. The mission of our consultants is therefore to guarantee short processing times whatever the circumstances.
Finance is becoming increasingly automated, a transition which goes hand in hand with the quest for efficiency. The mission to design and optimise real time systems will therefore continue.
The next step could be to integrate increasingly complex processing in the real time chain, such as real time risk assessment or increasingly sophisticated arbitrage algorithms.
Reduced technical debt
Technical debt is an economic concept applied to IT. It is a bad debt as it produces no benefits. It occurs due to the lack of time devoted to design and non-compliance with coding rules etc.
In an IT project, quality conflicts with the need to meet deadlines and control production costs. When developing an application and adding new functionalities, choices need to be made: quality, cost or deadline? The deadline often wins! Quality is sacrificed in the long term for the sake of reaching a short term objective: the rapid launch of a new version. We take on debt for immediate gain. Neglecting design is like borrowing money and paying it back over the long term.
Economic and technical challenges
The rising debt eats away at project resources and demotivates teams. Applications built without a plan become expensive to maintain and difficult to upgrade. The IT teams almost unknowingly reimburse the debt throughout the life time of the application.
Architects have become expert in ‘unplanned developments’, causing many hidden costs; slowness of new developments and loss of ability to adapt to change, increased support and maintenance costs, lower user satisfaction, projects that get delayed and missed launch dates…
There are solutions to avoid such debt:
Building a plan for the application that assesses the current situation and helps to control future developments
Refactoring the application
We believe that the assessment of the current situation and design phase is an essential precondition for any strategic decision and forms the foundation of any development project. What’s more, we know that automatic migration and refactoring are effective means of reducing technical debt.
To reduce the technical debt of IT systems, we:
To transform IT systems, we have created CodeCase
Audit systems with high technical debt: our tool analyses complex and voluminous IT systems
Transform existing IT systems into low technical debt systems: the technology, methodology, costs and lead times are all controlled
Maintain low technical debt for IT systems when in production: greater ability to adapt to change and low maintenance costs
Develop low technical debt IT systems
In financial markets IT, the current trend is to decommission and rebuild applications from scratch…And yet these projects are expensive!
At a time when IT systems are getting old and budgets are being reduced, how is it possible to combine lower costs and optimal efficiency for ageing IT systems?
Technical debt is therefore an ever more pressing issue that will continue to figure in the strategic thinking of IT decision makers. How to fight it when it is already present in a project? How to avoid it when developing a project?
Today, optimising the effectiveness of development methods and capitalising on existing systems fits perfectly with the prevailing need to reduce budgets.
The creators of CodeCase believe that the future of IT lies in the industrialisation and automation of developments.
We are betting on automation being the key to reducing technical debt!