Category: Agile

Agile and Markit EDM

In Brief

Many Organisations want to adopt Agile Methodology to implement IT Projects including Markit EDM Projects.  Since Agile Methodology is defined for the application projects, implementing the same practice in data projects like Markit EDM will have their own practical challenges especially in BAU environment.  In this blog, I will discuss about various challenges of implementing EDM Projects in agile methodology.

Before discussing the challenges in Agile methodology, I will touch upon two cultural challenges that impacts the use of Agile in EDM Projects.

  1. Developers
  2. Project Management and Agile Definition

Developers

All these years, Markit EDM developers considered data load or export as one single project. Naturally they will have an assumption that Markit EDM projects cannot be divided into minor modules to complete the implementation in one sprint. So they are always reluctant to adopt Agile methodology. Even when they adopt, they tend to develop waterfall style which is explained in next section.

Project Management and Agile Definition

DevOps culture is originated from application projects. So managers and scrum masters of these projects would have seen the benefits of the DevOps practice and encourage implementing the same process in Markit EDM projects as well. It is very important for the managers to know the basic difference between an application development using programming languages and Markit EDM development. But in reality very few managers know the difference and push for the same process which usually ends in a failure. On the other hand, if the manager has good experience on Markit EDM projects but do not know about DevOps practice, we will never start DevOps culture.

Challenges of using Agile in Data Projects

  1. Breaking down the work
  2. Definition of Done
  3. Sprint Length
  4. Team Structure
  5. Documentation and User Sign off
  6. DevOps

Breaking Down the work

One of the main agile principles is to complete the user story by end of every sprint. To complete the user story in a small sprint, user story needs to be very small but should add value to the users. Big question is how to break the Markit EDM Requirement to smaller pieces to complete develop, test and user sign off in the same sprint.

EDM Work can be mainly divided into two work streams

  1. Data Import, Matching and Mastering/ Data Export
  2. UI

In many cases, UI can be developed independently from core components provided data structure is in place. Challenge is breaking down the work in core development.

In Waterfall method, developer would have developed Import, Matching, and Mastering including Exceptions first before completing the Unit testing and hand over to testing team for Integration.

In Agile, we can have couple of approaches to break down the work which will have its own Pros and Cons.

When companies move to agile methodology, they tend to start Agile version of water fall. However in time, they will appreciate the benefits of developing in real agile way and move to 80-20 development.

Agile Version of Waterfall

  1. Sometimes, projects are developed in waterfall method but each step in the project is encapsulated in one sprint
  2. Let us assume Solution consists of 1000 Load, 2000 matching, 2500 Enrichment, 3000 Validation and 4000 Solution Mastering and each will take one week to complete in two weeks sprint
  3. Complete 1000 in sprint 1 and hand over to testing team, and then continue 2000 in next sprint
  4. As soon as testing is sign off, move the code to Prod without enabling in Prod.
  5. Even though we are following Agile, actually we are doing waterfall development without adding business value each sprint.
  6. 1000 and 2000 populates only staging tables, testers need not test at the end of 1000 and 2000 development which is like wasting their time.
  7. Since we are developing step by step and will not change the components developed in previous sprint, regression testing is not required
  8. Users are not involved will not see any benefit until sprint 4 which will not encourage project stake holders.

80 – 20 Development

  1. 80 Percent of the work in a project is usually completed in 20 percent of the time and remaining 20 percent takes 80 percent of time.
  2. Discuss with users and prioritise the important aspects of work
  3. For example, sometimes you can have 100 fields from Source files, and users require 20 field files immediately, other fields are required only for the future use. In this case, complete end to end for the 20 fields in sprint one and master other fields in future sprints.
  4. In some cases, users know system is very clean and Validations are required only for auditing purpose. In this case, loading and mastering can be developed in sprint one and Exceptions can be developed in future sprint
  5. In both cases, at the end of sprint one, users will have prioritised work in production
  6. Since same components may be touched, Challenge is to make sure the code is not regressed
  7. Also involvement of users will be very crucial which the basic aspect of Agile is.
  8. Mapping specs should be versioned correctly to reflect the agile development.

Definition of Done

One of the main agile principles is to complete the user story by end of the sprint. But How to define the ‘completion’ or in Agile terms what is the Definition of ‘Done’?

In Ideal World, Definition of ‘Done’ is to deploy the code in production and make it ready for the users to use the data. But in Markit EDM World, there can be any many outside factors which will stop the team from deploying in to Production.

Agile Teams are successful who are self-sufficient from Start of the work to until the work is ‘Done’.  It is better to define ‘done’ as a point where team cannot control the proceedings. For example, Organisation may have release cycle which will not fit into the sprint. In this case, Definition of done can be when UAT is signed off and ready for Live.

Sprint Length

Length of the sprint is very important for successful EDM implementation using Agile methodology. Traditionally it will be two weeks sprint for application projects which will be forced upon EDM projects.  But it is very important to understand the bottlenecks of EDM development before defining the sprint length. If the team thinks, increasing or reducing the sprint length will help the team increase productivity, there should not be any reason from stopping the team. It is very important to start the development only if developers are clear about the requirements.

For example, Let us assume we will have two week sprint. In traditional EDM development, teams plan as below.

Sprint Length – 10 days

Day Dev team Testing Team
1 -3 Development Testing Preparation
4-5 Testing Execution
6-7 Bug Development
8 Bug Testing
8 Release Preparation
9 Demo/ UAT Sign off Demo/UAT sign off
10 Next sprint estimation Next sprint estimation

Since testing team will not have much work in first week while development team will not have much in second week, this will not work.  Challenge is to divide the User story into multiple user stores for which development and testing team can do in parallel.

In Agile world, it will look like

Sprint Length – 10 days

Day Dev team Testing Team
1 -2 User Story (US)  1 US1 Testing Preparation
3-4 US 2 US 1 Execution
5-6 US 1 Bug US2 Testing Preparation
7 US2 Bug US1 Bug and US1 signs off
8 Release Preparation US2 Bug Sign off
9 Demo/ UAT Sign off Demo/UAT sign off
10 Next sprint estimation Next sprint estimation

Team Structure

Traditionally Markit EDM teams are managed horizontally. EDM Developers are grouped into one team who will provide the shared service to various projects. But the success of Agile implementation depends on the self-sufficiency of the team from Start to Live.

For example, when a new Fund is introduced, Agile team will have

  1. Product owner
  2. BA
  3. OMS Developers
  4. EDM
  5. Data Management team
  6. Testing Team
  7. Releasing Team (Or team should have control over the release)

Documentation and User Sign off

Agile methodology suggests less documentation and mostly defines the requirement in User stories. But realistically, this may not be possible for EDM Projects as they have multiple transformations for many fields. So unlike any other projects, irrespective of the Agile or Waterfall, EDM Projects will have mapping specs attached to the user story.

It is better to have users who need to sign off the story as part of the team. Since users are supposed to do their BAU, it may not be possible. It is very important to define the process to get the UAT sign off smoothly without impacting the team’s productivity.

DevOps

Most Organisations have well defined existing Release process which would have defined for Waterfall method followed in earlier days. Those releases may not be frequent and involve manual steps. In some places, Releases are managed and controlled by outside teams and Agile team (including Product owners) will have no say in release times.

It is very important to implement DevOps concepts like Test Automation, Continuous Integration and Release Automation to speed up the EDM implementation. It is suggested to release as soon as UAT is signed off. But if you have constraints in the release schedule, you will need to release once in a sprint for successful EDM Agile implementation.

For Markit EDM DevOps,

https://etlops.home.blog/category/markit-edm/

In Short

Success of Agile implementation of EDM projects depends on

  • Change in the development culture to see the requirement as collection of requirements rather than each requirement as a project
  • Self-sufficient teams
  • How the work is break down into small chunks which can provide value to business users in short time
  • Provide DevOps tools around the development, testing and release process to remove any bottlenecks