Configuration management

Configuration Management : Configuration Management www.scmGalaxy.com scmGalaxy Author: Rajesh Kumar info@scmGalaxy.com

Agenda : www.scmGalaxy.com scmGalaxy Agenda Introduction to software configuration management (SCM), build and release (BnR) Initiating CM activities Case study - 1.5 hours case study to help participants get a complete view of a CM process in a project. SCM tools comparison CVS – deep dive ANT – basics and advanced tasks Automated builds using cruise control

Agenda for session 1 : www.scmGalaxy.com scmGalaxy Agenda for session 1 Introduction to software configuration management (SCM), build and release (BnR) The need for SCM, BnR in a project lifecycle Objectives of a SCM BnR system Role of a configuration manager Typical activities of a configuration manager Initiating CM activities Identifying the inventory of configurable items Defining a CM plan Defining a release traceability with tagging and CR numbers Build and build automation Introduction to environments, Tools used in CM

Objectives : www.scmGalaxy.com scmGalaxy Objectives Understand the role of a CM Understand the type of activities a CM is involved in How to initiate and administer CM activities in a project Strategies for build, environments and automated builds Appreciate what is the list of tasks of a CM and be able to begin working as a CM

Need for Configuration Management : Need for Configuration Management

The Rambo developer : www.scmGalaxy.com scmGalaxy The Rambo developer John Rambo is a single developer of a mid sized project A one man team taking care of requirements, design , development, testing and delivery He has the whole source code on his machine and he overwrites his code each time a bug is fixed or a new change is brought in Application is simple easily testable. Overwriting the same piece of code is a rare occurrence Only his hard disk is backed up as a disaster recovery plan

Rambo developer and his woes : www.scmGalaxy.com scmGalaxy Rambo developer and his woes With increasing complexity the same parts of the code were getting overwritten More often than not, business users would request a change in a feature and soon expect to go back to the old feature or expect a mix of old and new features John soon finds himself with many requests to revert back to the old code Often john finds himself re-writing parts of the same code because of his overwrites

A disorganized team : www.scmGalaxy.com scmGalaxy 1 Guys, I heard you have finally cracked the cluster problem!! Congratulations !! What was it? 2 It’s a silly mistake boss!! An old version of the code was over-written which did not serialize the property files. 3 How did the old version get in there? 4 Prakash was trying to debug some problem last week from on-site and he fixed the problem in the old source and pushed them!! A disorganized team

The problems : www.scmGalaxy.com scmGalaxy The problems What is missing in these projects? How would you solve it?

The problems and solutions : www.scmGalaxy.com scmGalaxy The problems and solutions Currently there is no code changes being tracked. Once a change is made the old code is over written, there is no way to get back to the old code There is no release policy that defines what features need to be released A code release is unable to show the user the specific set of features that have been released A way to track changes and maintain versions of his code. A Version management tool is necessary A release policy should be defined A release should be done as a unit and not as a set of individual units A release document that explains the changes released in each release should be generated There should be a way to reproduce all parts of a release when ever required

Role of a Configuration manager : Role of a Configuration manager

Definitions - wikipedia : www.scmGalaxy.com scmGalaxy Definitions - wikipedia Configuration Management - The control of changes, including the recording thereof, that are made to the hardware, software, firmware, and documentation throughout the system lifecycle Software Configuration Management - set of activities designed to control change by identifying the work products that are likely to change, establishing relationships among them, defining mechanisms for managing different versions of these work products, controlling the changes imposed, and auditing and reporting on the changes made - pressman SCM concerns itself with answering the question: somebody did something, how can one reproduce it?

Software Configuration Management : www.scmGalaxy.com scmGalaxy Software Configuration Management “A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, and control changes to those characteristics, report change processing and implementation status, and verify compliance with specified requirements” Source: IEEE

Activities involved : www.scmGalaxy.com scmGalaxy Activities involved Management & Planning (SCMP) Software Configuration Management Software Configuration Identification Software Configuration Control Software Configuration Status Accounting Software Configuration Auditing Software Release Mgmt & Delivery

Need for a Configuration Manager : www.scmGalaxy.com scmGalaxy Need for a Configuration Manager With increasing complexity of a system one needs a dedicated person who can manage the changes to a large software unit What is complexity of a software system Long running projects Large teams Distributed teams (distributed development) Multiple software units that form one large chunk Assets related to the code – scripts, test cases Assets not related to the code – documents, release notes …

What is a CM’s role : www.scmGalaxy.com scmGalaxy What is a CM’s role A configuration manager works with the project leadership to provide an environment that controls, tracks and reports changes made to any unit of information Broadly speaking the various buckets of activities that a CM needs to own are Identification of configurable items Control the changes of these items Provide status and reports related to the changes to an asset Manage the build, packaging and release of the software units Provide a mechanism of traceability from the software units to the root cause of a change or the primary requirements Audit the project and ensure that the CM process is being followed

Role of CM at various stages : www.scmGalaxy.com scmGalaxy Role of CM at various stages

CM – stage 1 - Identify and establish : www.scmGalaxy.com scmGalaxy CM – stage 1 - Identify and establish The first phase of the project Along with the project initiation phase, the CM plan needs to be defined Most of the tasks in this phase is with respect to definition of a CM plan and establishing the infrastructure for the project Processes, templates, naming conventions etc. will need to be defined for the whole team to adopt in the time to come

CM – stage 1 - Identify and establish : www.scmGalaxy.com scmGalaxy CM – stage 1 - Identify and establish Objective List of CI identified Directory structure defined for all Tool for version management Change control process defined CMP defined, reviewed and accepted

CM – stage 2 - Build and release definition : www.scmGalaxy.com scmGalaxy CM – stage 2 - Build and release definition Design and development phases defines the code structure, the tools, the frameworks to be used in the project CM defines a build strategy for the project First cut build should ensure that it integrates well with the version management tool Provide automation to obtain a single integrated unit of software that can be used for development and testing Iterative process and prioritization is the key

CM – stage 2 - Build and release definition : www.scmGalaxy.com scmGalaxy CM – stage 2 - Build and release definition Objectives Base line Build script Strategy to enhance build script Integration of build with version management Strategy to release code Ability to support various environments

CM – stage 3 - enhance and strategize : www.scmGalaxy.com scmGalaxy CM – stage 3 - enhance and strategize The development and testing phases bring with it new tools and changes to source code structure The build script needs to evolve and automate as many tasks as possible Towards the end of the development phase the need to begin a release process would have arrived

CM – stage 3 - enhance and strategize : www.scmGalaxy.com scmGalaxy CM – stage 3 - enhance and strategize The ability to tag a release and the ability to roll back a release should be available In the near future code will move into a maintenance mode. The strategy to support multiple branches of code and simultaneous support and enhancements should be thought through

CM – stage 3 - enhance and strategize : www.scmGalaxy.com scmGalaxy CM – stage 3 - enhance and strategize Objectives Enhanced build script Integrated build and release process Tagging of specific releases One shot deployment and roll-back capability Reports and data analysis Branching strategy

CM – stage 4 - Continuous Improvement and auto-pilot : www.scmGalaxy.com scmGalaxy CM – stage 4 - Continuous Improvement and auto-pilot Mostly in administration mode The build script and all process have lived their time and proven their capabilities Often a full time CM may not be required Often this phase is the point to let go of a full time CM As with every software asset, continuous improvement is a must

CM – stage 4 - Continuous Improvement and auto-pilot : www.scmGalaxy.com scmGalaxy CM – stage 4 - Continuous Improvement and auto-pilot Objectives Reports of releases and release notes with each release Minor changes to build script Ability to regression test and manage releases Manage multiple branches and multiple releases Focus on traceability from released features to bugs. Control changes to the repository

CM – important tasks : CM – important tasks

Important tasks : www.scmGalaxy.com scmGalaxy Important tasks Among all the phases a few tasks will be the most important part of a CM’s job The next few slides will delve deep into some of these tasks The aim would be to understand each of these tasks in detail The intent is to provide you with a view of the over all objective of the task and there on to arrive at a few strategies for each of them

Important tasks - list : www.scmGalaxy.com scmGalaxy Important tasks - list Identify CI Version management and tool selection Prepare CMP Release and Branching strategy (Multiple) Environment management Build script Integrating tools with build Audit of the CM process Automating builds

Configuration items : Configuration items

Identify Configurable Items : www.scmGalaxy.com scmGalaxy Identify Configurable Items What is a Configurable Item (CI) A CI, is a unit of configuration that can be individually managed and versioned. Term used to describe any entity / asset (code and non-code) that may change across the project phases The type of asset and its users will help in defining a versioning strategy and its availability A CI list should be a comprehensive list of assets (or asset types) and should feed into the tool identification step

CI list : www.scmGalaxy.com scmGalaxy CI list

CI list : www.scmGalaxy.com scmGalaxy CI list

Where to version the CI : www.scmGalaxy.com scmGalaxy Where to version the CI Choice of network share, internal CVS and an external facing CVS Classified reports and sensitive documents may not be shared with the whole team or even the client. Access control and level of access should be considered while choosing the location of store Use a network share or a private CVS repository Different repository for client and internal teams with scheduled synch-up between the two A network share is not version sensitive but can be used for static items like 3rd party software downloads, documentation etc.

Directory structure : www.scmGalaxy.com scmGalaxy Directory structure Over all a directory structure is mostly common sense Ability to logically separate out asset types and put them into separate folders Based on the version management tool, one can setup directory level permissions or permissions at a repository level It may also be prudent to understand the need for private work areas and private folders for each individual in the project Leverage the existing structures at an organization / group / account level for a start point

Directory - first level : www.scmGalaxy.com scmGalaxy Directory - first level Requirements Design App Templates QA PM UI reports

Naming conventions : www.scmGalaxy.com scmGalaxy Naming conventions Naming conventions (even if a file is organized into directories), should describe the asset type to the fullest using some meta data of the asset. intent of the document (or asset) – GlobalOne_reference_exisitng_report_template.doc date (specially with MOM) – MOM_admin_role_clarififcaton_010207.doc module it is relevant to – design_admin_logging.doc All asset types identified in the CI list should be given a naming convention Source code will follow the naming convention given by the coding standards document

Version management tools : Version management tools

Version management tools : www.scmGalaxy.com scmGalaxy Version management tools There are various tools that allow version management The primary features of these tools are Version management Concurrent development (lock or merge) Client server model Branching Tagging Multiplatform support Integration with IDE Multiple authentication modes

Version management tools : www.scmGalaxy.com scmGalaxy Version management tools Some tools provide the following features Web interface (integrated with the product) Distributed development using multiple repositories Auto-replication

Selection criteria : www.scmGalaxy.com scmGalaxy Selection criteria Criteria Locking and merging Authentication and authorization mechanism Need for distributed repositories / distributed clients Web interface Licenses (cost and ease of setup) Mostly the client / organization standard will be used as the default for a project However, in case we are presented with a ground up scenario, then the criteria listed above can be used for selection of the tool

Various types of repository access : www.scmGalaxy.com scmGalaxy Various types of repository access Understand the impact of Network speed Stale data Cost of setup Security Authentication

Version management – version number : www.scmGalaxy.com scmGalaxy Version management – version number The basic versioning of an asset will be 1.0 onwards in most version management tools. Incremental changes are recorded as 1.1, 1.2 and so on From an asset perspective, a version history will be maintained in the asset itself to define the changes made across each released version The version numbering at a tool level can be used as is Thought should be applied more to tagging of assets and base lining assets

Version management - tagging : www.scmGalaxy.com scmGalaxy Version management - tagging A tag helps in creating a logical name for all assets within a repository at a particular instance A tag can be given at a branch level and based on the tool even at a directory level Tagging is the most important feature of a version management tool that helps in recreating a release The name of a tag should be intuitive enough to show the user information about the date, version number, major and minor release etc

Files in the repository : www.scmGalaxy.com scmGalaxy Files in the repository

Tagging the repository : www.scmGalaxy.com scmGalaxy Tagging the repository The number of versions that each file take is dependent on each owner However, when a delivery needs to be done, all files across multiple versions need to be taken. One needs to take these files and have the ability to recreate the version to solve a bug if need be Tagging allows us to set a string as the “tag” to various files each with different versions A tag is like a snapshot that can be recreated

CMP – configuration management plan : CMP – configuration management plan

Intent of a CMP : www.scmGalaxy.com scmGalaxy Intent of a CMP A CMP describes the configuration management plan for a project Describes the repositories, naming strategy, versioning strategy and more Reference point for the team to begin working in a project Helps in documenting all the CM activities and delegating responsibilities of a CM activity to team members A CM initiates the CM processes and rolls out a process It is for the project there on to adhere to it there on. The CMP defines the various roles and their responsibilities

Intent of a CMP : www.scmGalaxy.com scmGalaxy Intent of a CMP To be used by the whole team, the auditors of the project and the new entrants into the team to understand get hold of various assets. The tools used and their setup, the process of getting access can be a separate handbook The CMP sets the direction and helps in future audits of the project from a CM perspective

Intent of a CMP : www.scmGalaxy.com scmGalaxy Intent of a CMP A CMP should document enough data to help answer the following questions What kind of items are CI? Where should I put what CI? What are the naming conventions for various items? What kind of documents should I maintain and what is my responsibility from a CM perspective? What is the versioning strategy of any artifact? How will project level changes be tracked? What is base lining and how will releases be managed

Change management : www.scmGalaxy.com scmGalaxy Change management A key task of the CM is to control the changes to an asset A change control process defines the monitoring / approval process that allows incorporating changes into a CI Requirements, tools, criteria of acceptance etc. are CI that are stored in the repository A change in one of them could affect many other assets and the project schedule The intent here is not to resist changes, but to allow a controlled way of managing the changes

Change management : www.scmGalaxy.com scmGalaxy Change management The CMP should record the ways to track / control the changes to each CI if not all at least the critical assets such as requirements, design, acceptance criteria, plan, strategies specific to the project and more should be considered Document the ways to accept changes to a particular asset bug tracker / email can be a source of change requests. The CMP should also call out the person responsible to manage these changes This level of clarity in change management ensures that changes to appropriate documents are controlled

Change management : www.scmGalaxy.com scmGalaxy Change management Another aspect of change management is the process that one should follow to get a change incorporated More relevant to a PM However the CM needs to ensure that a CCB has been instituted for the project The change management for appropriate assets would need a CCB approval (schedule change / feature list increase)

Read a sample CMP : www.scmGalaxy.com scmGalaxy Read a sample CMP Demonstration of a CMP

Key points in a CMP : www.scmGalaxy.com scmGalaxy Key points in a CMP Ensure that all CI have a repository to go to Ensure that all CI have an appropriate location in the repository (directory structure) Ensure that each repository identified is backed up and an appropriate person / group is given the responsibility Ensure that all CI have an owner (person, group, roles). These owners will maintain the versions of these items Most importantly, ensure that the CMP is well circulated and call out the responsibilities in a team meeting or in a short training program Do not assume that people will know about their responsibilities.

Rolling out a CM process : www.scmGalaxy.com scmGalaxy Rolling out a CM process Rolling out the process, often is taken too lightly and the team is not always in synch with the process Identify who needs to be told what and tell them the process using training programs / meetings Do not shy away from setting up a meeting each time the team size swells up. For large teams, work with each module rather than a large team

Access to the repository : www.scmGalaxy.com scmGalaxy Access to the repository A well defined way to setup the client software and integrating it with the IDE should be documented Re-use the standard within the organization for a particular type of repository The steps could be simple and easy, but do not assume that a team of 10-50 people will know it Ensure that a setup document, is available within the repository / network share The setup document should speak about Where to get the installable (ensure that nobody uses their own versions) How to install it (ensure that the installation document has screenshots) How to integrate my IDE with the repository (ensure that people to not have copies of the code in their own machines)

Check point : www.scmGalaxy.com scmGalaxy Check point At this point, the CM process has been initiated and established The team can setup the clients and begin versioning their assets The next set of steps is related to building the code and creating a single composite unit Keeping the build process aside it is important to understand the concept of release, branching and traceability A build process could be a script or a manual one. The intent of the script is to create a comprehensive unit of code that can be released to an environment Lets work with the assumption that a build script exists and understand the concept of release

Release Management : Release Management

Need for a release : www.scmGalaxy.com scmGalaxy Need for a release A single unit of software has many dependencies on other entities Database Environment parameters Application parameters Parameters could differ based on the environment the application is being deployed into The intent of a release is to build that comprehensive unit and deploy to an environment A release could be a manual one, automated one or a blend the key point to note is that it should modify all related entities with a particular version of code and allow users to view new features

Notes about a release : www.scmGalaxy.com scmGalaxy Notes about a release Important considerations for a release The whole software unit should be one single unit (e.g. ear file) Ensure that all external dependencies are modified along with a release Database Properties file These modifications can be manual (not preferred), automated script (better), integrated within the application deployment (like an installation exe - ideal) Ensure that a set of happy flows are tested before the release is used by everybody Release notes should be used to document the “new features” and the “known issues” in a release

Traceability : www.scmGalaxy.com scmGalaxy Traceability Traceability helps to relate a component to its requirement Once the software is released to an environment there is the additional complexity of bugs Each change to the software to resolve a bug, should also be traceable to the bug and the requirements / updated requirement (in the form of a bug) A change could even be initiated by a review comment. All changes to the unit should be traceable to its original cause for change

Traceability : www.scmGalaxy.com scmGalaxy Traceability This concept of being able to trace back a need or a cause (requirement) from the effect (code base) is traceability A cause of change could be a requirement, a review or even a change to design A configuration manager needs to enable traceability with the CM processes It is up to the team to make adequate comments, put into version history etc to manage the traceability but the infrastructure to bring in traceability is the task of a CM

Traceability in Documents : www.scmGalaxy.com scmGalaxy Traceability in Documents Documents such as requirements or design are generally described in a word document These assets are also CI and therefore will be versioned The comments while checking them into the versioning tool is important to bring about some level of traceability against time. The comments will be seen in the future to understand why a check in happened The version history table in each of these documents provide a better point for tracking the change Descriptive version history and more importantly relating these changes to other artifacts in the repository (MOM, status reports) is most important

Traceability with code assets : www.scmGalaxy.com scmGalaxy Traceability with code assets Traceability is most relevant to code assets The traceability of a code asset to a requirement or a design is often done using external matrices and code structure Package structure Documentation of a method can trace back to a design asset Traceability to a change / bug resolution is also more important This level of traceability is also relevant in a continuous maintenance mode

Traceability and tagging : www.scmGalaxy.com scmGalaxy Traceability and tagging The ideal way to work with release is to first tag the code and then retrieve the code based on that tag Between 2 tags (or releases) one can determine the number of changes done to a repository and track the comments against each of these assets Ensuring that a comment exists and the comment follows a particular structure is one way of brining in traceability E.g. – comment should begin with BUG #NNNNN - ensures that the user writes the bug number before he checks in the code These comments can be used as a way to define the release notes

Strategy for release numbers : www.scmGalaxy.com scmGalaxy Strategy for release numbers A tag that is applied to the repository should be self explanatory of the release and can be used as the release number too If a different release number and tag number is being used, then a separate document should track the relation between the tag and the release A release number should have the following parts to it Code base name / app name Major release number (sequence) Minor / patch release number (sequence) Time (day, month, year) Type of release (qa, prod, dev, RC) Branch name

Demo of release and tagging : www.scmGalaxy.com scmGalaxy Demo of release and tagging Demonstrate tagging and obtaining a release Demonstrate the ability to trace file changes between 2 releases Show the release readiness template

How to support multiple projects / features : www.scmGalaxy.com scmGalaxy How to support multiple projects / features Once a project is delivered, and tagged, often the next release is decided at a pre-defined interval In the interim the developers continue to edit the files to prepare for the next release In the interim, if there are bugs on the production release, the files latest files cannot be used to fix them At each point where a release is made, once needs to create a branch A branch is like a copy of the whole code base except that it starts of from a particular tag / version The main branch is called the ‘HEAD’ branch. All other branches can be named as required Branches can be merged once it is no longer relevant View CodeBranch_Merge.vsd

Note on branching : www.scmGalaxy.com scmGalaxy Note on branching A new branch is more responsibility on the CM and the dev team Each bug and change there on should be done on the right piece of code Update the setup documents to reflect the steps needed to retrieve sources from a particular branch and not just the main trunk Branching as a concept should be taught to novice developers Builds and environments to deploy the build is also impacted with each new branch Often the number of CM in a project may need to be increased

Branching in CVS : www.scmGalaxy.com scmGalaxy Branching in CVS

Impact of branching : www.scmGalaxy.com scmGalaxy Impact of branching branching of a repository impacts Tags release process frequency of builds developer environments build scripts Setup documents Authentication permissions Build machine hardware requirements Understand the impact of branching for your project A new branch is a simultaneous project on the same code base but on different threads. It is simple only if pain is taken early on to strategize

Demo of branching : www.scmGalaxy.com scmGalaxy Demo of branching

Supporting multiple environments : www.scmGalaxy.com scmGalaxy Supporting multiple environments

Common issues with CM and a team : www.scmGalaxy.com scmGalaxy Common issues with CM and a team Issues a team often faces with a bad CM I do not know how to configure my client environment? What is the process to follow when I check in my code? What is traceability? What is my role in it? The DB does not reflect the latest status, what should I do to get the DB changes into the code base? Where is the documentation of the 3rd party library? May I should download and use the latest version? The build takes too long, all I wanted to do was to deploy a JSP page? Which environment is the tester getting these bugs? How many environments are there?

Audit of a process : www.scmGalaxy.com scmGalaxy Audit of a process Need for audit Understand whether the course being taken is as per the original plan Deviations are not unnatural, but the impact of deviations should be well understood De-risk the process / project to ensure that there are no serious lapses against the plan

SCM audit : www.scmGalaxy.com scmGalaxy SCM audit The purpose of the Configuration audits is to: Assure that correct versions of Configuration items are placed under the appropriate directory / folders in the central repository. Changes to be included in the version of the product to be delivered are included. All required items of hardware, software, system, data, procedures, and documentation are available with correct identification and labeling and storage. Configuration Status is tracked Show a sample audit template

Frequency of audit : www.scmGalaxy.com scmGalaxy Frequency of audit Quarterly audits or auditing between phases will keep the process compliance high Audit will usually done by PM / Leads from other projects Audits within the project is also a good practice The CM is the owner of the CMP and has to ensure its adherence Regular checks by the CM will help in the long run comments when doing a check-in maintaining the version history in documents adherence to the naming conventions

CM best practices : www.scmGalaxy.com scmGalaxy CM best practices Integrate all users with the repository. Build, development IDE and other touch points should use the repository as the source Dissuade users from making copies of the repository items Focus on automating as many tasks in the build script Automate the build kick off using tools like cruise control Manage only the source and not the derived binaries Arrive at an appropriate branching strategy for the project Roll-out the CM process and educate the team

Mindtree CM’s role : Mindtree CM’s role

SCMP Process at MindTree : www.scmGalaxy.com scmGalaxy SCMP Process at MindTree

What are PM’s Responsibilities? : www.scmGalaxy.com scmGalaxy What are PM’s Responsibilities?

What are PM’s Responsibilities? : www.scmGalaxy.com scmGalaxy What are PM’s Responsibilities?

Case study : Case study

Slide 85 : www.scmGalaxy.com Thank You ! Author: Rajesh Kumar rajeshkumar.raj06@gmail.com