DevOps Ideas for Markit EDM

What is DevOps?

DevOps is not a technology or a frame work. It is an organisation culture. Even though Dev Ops is became popular in recent years, Dev Ops is a not a new practice in IT development cycle. It exist from long time ago but with different names. Most times we called them as ‘Automation’. DevOps is nothing but ‘DEVlopment for OPerationS’. That is

• Automation of the Development
• Automation of the testing
• Automation of Code Packaging
• Automation of Releases
• Automation of BAU support

In short, automation of each and every step along the project life cycle to optimize the project delivery and operations is called as DevOps.

Challenges of bringing DevOps in Data or Markit EDM Projects

I look at the DevOps challenges in three different perspectives

1. Technology
2. Project Management
3. Developers

Technology

Most of the existing DevOps frame works and technologies are developed for programming technologies like Java and .Net. But the development of jobs in Markit EDM is completely different from Java or .Net development. Developer cannot control how the underlying Markit component XML was generated and stored. Developer cannot control how the code was released. Markit EDM uses its own method to deploy the code. So when we try to use DevOps frame work developed for Java or .Net in Markit EDM, it will not meet our objective.

We end up in developing work around for each step and the effort will outweigh the benefits. When effort outweigh benefits, shareholders will not appreciate and eventually we will go back to manual tasks. It is better to develop a DevOps frame work that suits your Markit EDM needs rather than simply following the practice in application projects.

Project Management

DevOps culture is originated from application projects. So managers of these projects would have seen the benefits of the DevOps practice and encourage to implement the same process in Markit EDM projects as well. It is very important for the managers to know the basic difference between an application development using programming languages and Markit EDM development. But in reality very few managers know the difference and push for the same process which usually ends in a failure.

On the other hand, if the manager has good experience on Markit EDM projects but do not know about DevOps practice, we will never start DevOps culture.

Developers

Most of the DevOps or automation tasks are implemented through scripting languages. But most of the Markit EDM developers are used to work in EDM and will not have any experience in scripting languages. Markit EDM Developers need to come out of Comfort zone and learn new scripting languages required for automation

Agile Vs Water fall Methodology

Since the topic was discussed lot in Markit EDM conferences, I assume Agile Methodology is used for Markit EDM development. However some of the DevOps tasks like automated testing can be used in any project management methods.

DevOps Ideas

DevOps can be introduced in each and every phase of Markit EDM Development. We will touch upon what is available in Markit and what can be developed. Please note some of the ideas may not be approved or agreed by Markit.

DevOps in Development

Markit EDM development usually have two phases

Markit EDM Development

Companies are looking to automate as much as possible to not only reduce the time required for development but also reduce the manual errors and maintain consistency. Most of the Markit jobs follows the same template of Data Porter or Data Flow to populate the ‘In’ table, Inspectors for Inspection, Constructors for Mastering.

Instead of creating similar data Flow again and again with different column names and table names manually, we create a job or a program (using Markit or any other language) to create the underlying XML from the mapping Spec and then import the XML to Markit. This may look touch too far but the method is already used in popular ETL tools. There is no reason why it cannot be done for Markit.

Automate Markit EDM code Review

Code review is a complicated step especially in the Markit EDM environment. Definition of code review is different from person to person, environment to environment and organisation to organisation. One way to standardises the code review is to automate the code review process. Markit came with few out-of-box reviews like number of inputs, etc. during code development but they can be simply discarded by the developers and are never reported.

Proposed Steps to automate the code review

1. Define the standards in metadata table
2. Taking component XML as an input, write the inspection against the ‘Standard’ metadata table (Metadata table used to define standards)
3. Report the inspection failures into a table
4. Write an UI report to show how many reviews are done and how many passed
5. If there are any critical reviews, those can be marked in a different colour

DevOps in Testing

Testing is another area that might make or break any development project. There are many frame works available in the market for data testing. You can either use out-of-box Markit Testing Automation tool or any existing framework that is available in the market e.g.: Fitnesse. Most of the framework simply follows “expected result verses actual result” rule. So you can create your own framework if required.

Testing automation can be done in three phases of the project

1. Unit testing
2. System Testing or Integration testing
3. Regression Testing

DevOps in Packaging and Deployment

Automating the Markit EDM code packaging and deployment is complicated and may depend on version control used for development. Latest version of Markit helps the developers to package directly from version control using labelling. When the ‘database’ version control is used, SQL can be used to find the component that need to be released and PowerShell script to call DOS commands to do the actual Packaging and code release. When automatic packaging is implemented using Labels, convention in labelling become important. In some places, packaging is done manually in the specified format but the deployment is done automatically. Also you have an option of packaging and deploying SQL tables and Markit components separately.

There are various tools available in market for building (e.g.: VSTS) and releasing the package (e.g.: Octopus)

Continuous Integration

Due to nature of Markit development, more than one development can update the same component at the same time. When multiple developments going on in parallel, there are high chances of deploying the untested code in Production in addition to regression issues.

Continuous integration helps to identify the regression issues early in the development life cycle. Continuous integration plays an important role in smooth transition of code from Development to Test to Production especially in BAU scenarios. Automating testing, packaging and deployment are pre-requisite tasks for implementing Continuous Integration.

Points to remember before implementing continuous integration

• Due to performance of package deployment, it is not realistic deploy full set of code every time
• Since both Business Data and Markit component data are stored in the same database, some reference data may be lost while restoring the database from the last backup
• Always set up the data required for testing at the test or suite level
• If multiple packages are submitted for continuous integration testing, what will happen to other packages when one of the package is failed in testing

Figure 5 shows how Continuous Integration can be implemented in Markit EDM projects for one development cycle. In a real world it can be complicated than this because there can be more than one release for the sprint and many development can impact the same component. However the principle is same.

Build can be two ways.

1. Release the Full set every time (for new projects)
   1. Each build will have code for the entire system
   2. ‘Build n+1’ will have code from both ‘Build n’ and ‘Build n+1’. Hence only ‘Build n+1’ will be released. New developments can follow the method. If it fails, either code can be reverted back to ‘Build n’ and test or fix the code and build n+2 version
2. Release incremental code (for BAU and Maintenance)
   1. Each build will have only incremental code
   2. ‘Build n’ and ‘Build n+1’ are two different builds and independent of each other. Both builds will be released to subsequent environments. If one of them failed, other will be still released. This is usually followed in BAU scenarios

In Short

DevOps may not solve any business problems. However it will expedite the speed and quality of the Markit EDM project delivery by

•  Finding the issues up front
•  Reduce the regression issues
•  Minimize the rework
•  Minimize the errors by minimize the manual tasks
•  Time and effort for the actual development against time spending in process like code reviews and deployment
•  Avoid boredom of doing same tasks again and again