Continuous integration / continuous delivery (CI/CD) is commonly seen as a prerequisite for faster delivery in higher quality, which is especially important for cloud-based products. In this blog post I will look at what this means for ABAP development, taking an outside-in view based on my experiences in several years of non-ABAP development.
In a recent blog post on CI in ABAP AS, SAP Mentor lars.hvam also shared his views on CI for ABAP and presented a list of options you already have today, a bit of a complementing inside-out view, if you like.
I'd be curious to learn about different viewpoints. Please leave your comments below!
So, what does CI actually mean in general? Let's start with Martin Fowler's definition dating back to the year 2000: "Continuous Integration is a software development practice where (…) each person integrates at least daily." - Don't we automatically integrate in ABAP every time we activate an object? Well, yes, sort of. But there is more to it, actually a whole list of best practices:
A lot of things have happened in the field of CI/CD since the 20y old list above was first created. One of the most notable ones is the concept of "proposed commits" or "pending head", i.e., the idea to review changes before integration via GitHub Pull Requests, Gerrit Changes or similar means. These change-based peer code reviews do not only support a kind of "asynchronous pair-programming" but also allow the consultation of arbitrary technical, automated voters like static code checks, unit test, etc. that check the change in isolation, thus detect errors before integration into the mainline and prevent them from affecting all other developers. Such pre-commit CI checks are considered a major game changer for both Open Source and commercial software development.
You can achieve something comparable in native ABAP via test transports (transport of copies to test systems) before releasing a transport or corrective measure (and trigger them via BAdi APIs of the Transport Organizer). Thus, you can find defects related to unobvious transport dependencies before they propagate to downstream systems. However, this will still not provide isolation during development, i.e., if you screw up some important piece of code in a central dev system, you can still become infamous very quickly.
The more independent working mode known from local non-ABAP development may be achieved by using abapGit as a complement to CTS (SAP Change and Transport System) or gCTS (git-based CTS) instead of TMS (SAP Transport Management System) to externally version ABAP development objects and configurations in git repositories. On top you can use containerization to provide dev systems per team or even per developer plus clean on-demand CI systems. Combine that with standard CI pipelines to verify changes in isolation before integrating them back and you will end up with a target picture not much different from other non-ABAP dev setups. This has been hinted at a couple of times, e.g., in the Lars’ blog post above, gCTS’ statement of direction, murbani’s recent blog post on git-based ABAP development or jamesroberts’ blog post on using abapGit and containers. One of the biggest obstacles for this novel development paradigm is, again, the lack of componentization that would force too much code into few very large Git repositories, which would quickly become unmanageable.
So, when we take away the inherent "continuous" integration given by current in-system ABAP development, how can we prevent detrimental effects of decentralized development like late integration? As often seen in the non-ABAP world, if nothing forces developers to integrate regularly, important changes are often kept local for too long so that delivery is at risk. This can be mitigated on design/development level by practices like consumer-driven contract testing, but also logistically by the way we use branches. Trunk-based development has evolved as a branching model where only one mainline/master/trunk is kept active for development and short-lived feature branches have to be integrated daily. Main consequence is that you need to program as if DEV was PROD, i.e., use feature toggles or branch by abstraction and other concepts to hide unfinished increments from others by default without risking delivery of the whole product.
This should in fact feel not unfamiliar to ABAP developers that have used profile parameters to control the activation of new features in their regular development systems.
CI/CD is often used in combination, but what does Continuous Delivery mean specifically? Well, CI was about automating your development pipeline until the test stage. CD not only goes one step beyond by including the release/deployment stage, it is also about developing in such a way so that software can be released to production at any time. "The key test is (...) that the current development version of the software can be deployed into production at a moment's notice." (Fowler)
Starting to feel queasy? ? Me too, when I consider traditional ABAP delivery processes, tailored to few on-premise shipments per year. Certification and other legal requirements force us to build up elaborate system landscapes with well-defined transport paths and quality gates that make it hard to deliver code from dev to prod in weeks, let alone in days.
Fortunately, the git-based approaches mentioned above make it possible use standard CI/CD pipelines also for ABAP development, test, and deployment to target systems. Modeled in the open-source Jenkins library of project Piper, it becomes possible to integrate these steps also with non-ABAP development parts like SAPUI5. SAP Cloud Platform ABAP Environment (Steampunk) also goes in this direction as thomas.schneider recently described in his blog post on software lifecycle management on Steampunk. Stay tuned, there is more to come.
Let me quickly summarize a few key findings from my review of best practices above:
Do you have different experiences, aspects that I forgot, or do you just want to leave some comments? I am curious to hear about them!
Photos by JJ Ying, Oscar Ivan Esquivel Arteaga, Robert Murray and Jesse Ramirez on Unsplash
In a recent blog post on CI in ABAP AS, SAP Mentor lars.hvam also shared his views on CI for ABAP and presented a list of options you already have today, a bit of a complementing inside-out view, if you like.
I'd be curious to learn about different viewpoints. Please leave your comments below!
CI - What?
So, what does CI actually mean in general? Let's start with Martin Fowler's definition dating back to the year 2000: "Continuous Integration is a software development practice where (…) each person integrates at least daily." - Don't we automatically integrate in ABAP every time we activate an object? Well, yes, sort of. But there is more to it, actually a whole list of best practices:
- Maintain a single source repository ?. Not all ABAP development objects are versioned and transportable. The outcome of tests often depends upon non-versioned, non-transportable configuration data. Because it is so hard to reproduce the setup from sources alone, ABAP systems are often treated like materialized branches -- quite an expensive way of version management, because you need to keep them alive as long as a codeline is in maintenance.
This also leads to a tendency to treat ABAP systems like pets instead of cattle, i.e., you cannot manage your infrastructure as code, a best practice for operating cloud systems. - Automate the build ?. ABAP development objects are automatically compiled when you activate them. If they don't compile, you normally can't activate them. Other build aspects like static checks and tests either have to be triggered manually or can be integrated in the release of transports. Assembly happens behind the scenes and usually remains a mystery for normal developers.
- Make your build self-testing ?. The means are there with ABAP Unit & Co. It remains a responsibility of developers to actually practice TDD, increase code coverage, etc.
- Everyone commits to the mainline every day ?. If you consider DEV as mainline, everything is fine. If development of your component is spread across multiple systems, TEST/CONS would be your mainline, i.e., you would have to release your transports daily to "commit" to that mainline. While some projects already work this way, mainstream is still to keep large collective transports open for a week or longer.
- Every commit should build the mainline on an integration machine ?. Same as above, no problem here if you consider activation without syntax errors as a build of your mainline. But if TEST/CONS is your mainline and "build" should comprise static code checks and all relevant tests of your component, situation looks different.
- Fix broken builds immediately ?. Syntax errors in dev systems are usually no problem. But if you also consider package errors, ATC violations, and unit test errors as broken builds, you will see a huge "broken window effect" in some projects, i.e., a tendency to ignore these errors until project deadlines approach or even beyond. This not only builds up a tremendous technical debt, which puts delivery at risk, but also makes it hard for well-behaved developers to verify if their development increment did not inadvertently increase the pile of errors.
- Keep the build fast ?. One of the Extreme Programming (XP) practices requires builds (incl. tests) to be finished in less the 10 minutes. The lack of componentization with clean, layered package interfaces in many ABAP projects often makes it impossible to test the impact of a change within this time frame, because it is hard to determine the right set of affected tests. It would be cool if we had something like Test Impact Analysis in ABAP to execute only those tests that a change of productive code could potentially have an impact on.
Another issue is the activation of central CDS views or DDic structures, which can sometimes take ages in large components because the transitive closure of all affected objects has to be re-activated as well. - Test in a clone of the production environment ?. Due to the lack of a single source repository (see first item above) test systems tend to have a life of their own, which sometimes leads to false positives/negatives that never/do occur in production systems.
- Make it easy for anyone to get the latest executable ?. No problem here if you consider DEV/TEST/CONS as latest executable.
- Everyone can see what's happening ?. All the test reports are usually available but the sense of urgency to fix broken builds (6. above) is sometimes missing.
- Automate Deployment ?. No problem here if you consider CTS as automatic deployment.
Proposed Commits
A lot of things have happened in the field of CI/CD since the 20y old list above was first created. One of the most notable ones is the concept of "proposed commits" or "pending head", i.e., the idea to review changes before integration via GitHub Pull Requests, Gerrit Changes or similar means. These change-based peer code reviews do not only support a kind of "asynchronous pair-programming" but also allow the consultation of arbitrary technical, automated voters like static code checks, unit test, etc. that check the change in isolation, thus detect errors before integration into the mainline and prevent them from affecting all other developers. Such pre-commit CI checks are considered a major game changer for both Open Source and commercial software development.
You can achieve something comparable in native ABAP via test transports (transport of copies to test systems) before releasing a transport or corrective measure (and trigger them via BAdi APIs of the Transport Organizer). Thus, you can find defects related to unobvious transport dependencies before they propagate to downstream systems. However, this will still not provide isolation during development, i.e., if you screw up some important piece of code in a central dev system, you can still become infamous very quickly.
The more independent working mode known from local non-ABAP development may be achieved by using abapGit as a complement to CTS (SAP Change and Transport System) or gCTS (git-based CTS) instead of TMS (SAP Transport Management System) to externally version ABAP development objects and configurations in git repositories. On top you can use containerization to provide dev systems per team or even per developer plus clean on-demand CI systems. Combine that with standard CI pipelines to verify changes in isolation before integrating them back and you will end up with a target picture not much different from other non-ABAP dev setups. This has been hinted at a couple of times, e.g., in the Lars’ blog post above, gCTS’ statement of direction, murbani’s recent blog post on git-based ABAP development or jamesroberts’ blog post on using abapGit and containers. One of the biggest obstacles for this novel development paradigm is, again, the lack of componentization that would force too much code into few very large Git repositories, which would quickly become unmanageable.
Trunk-based Development
So, when we take away the inherent "continuous" integration given by current in-system ABAP development, how can we prevent detrimental effects of decentralized development like late integration? As often seen in the non-ABAP world, if nothing forces developers to integrate regularly, important changes are often kept local for too long so that delivery is at risk. This can be mitigated on design/development level by practices like consumer-driven contract testing, but also logistically by the way we use branches. Trunk-based development has evolved as a branching model where only one mainline/master/trunk is kept active for development and short-lived feature branches have to be integrated daily. Main consequence is that you need to program as if DEV was PROD, i.e., use feature toggles or branch by abstraction and other concepts to hide unfinished increments from others by default without risking delivery of the whole product.
This should in fact feel not unfamiliar to ABAP developers that have used profile parameters to control the activation of new features in their regular development systems.
Continuous Delivery
CI/CD is often used in combination, but what does Continuous Delivery mean specifically? Well, CI was about automating your development pipeline until the test stage. CD not only goes one step beyond by including the release/deployment stage, it is also about developing in such a way so that software can be released to production at any time. "The key test is (...) that the current development version of the software can be deployed into production at a moment's notice." (Fowler)
Starting to feel queasy? ? Me too, when I consider traditional ABAP delivery processes, tailored to few on-premise shipments per year. Certification and other legal requirements force us to build up elaborate system landscapes with well-defined transport paths and quality gates that make it hard to deliver code from dev to prod in weeks, let alone in days.
Fortunately, the git-based approaches mentioned above make it possible use standard CI/CD pipelines also for ABAP development, test, and deployment to target systems. Modeled in the open-source Jenkins library of project Piper, it becomes possible to integrate these steps also with non-ABAP development parts like SAPUI5. SAP Cloud Platform ABAP Environment (Steampunk) also goes in this direction as thomas.schneider recently described in his blog post on software lifecycle management on Steampunk. Stay tuned, there is more to come.
Summary of Observations
Let me quickly summarize a few key findings from my review of best practices above:
- Maintenance codelines are materialized in systems instead of version control. Git comes to the rescue (via gCTS or abapGit).
- Lack of componentization makes it hard to determine impact of changes and limit number of relevant tests. Splitting monoliths is inherently hard. How do you refactor a system of 1M LoC? - The same way it was written, one line at a time!
- Proposed commits become possible by integrating standard CI pipelines via git.
- Test coverage and more automation are key. If you want to deliver more frequently, you cannot afford any manual steps or consolidation / cool-down phases anymore.
- Containerization can help to test changes in isolation and treat systems more like cattle instead of pets.
Do you have different experiences, aspects that I forgot, or do you just want to leave some comments? I am curious to hear about them!
References
- gCTS Statement of Direction
- Software Lifecycle Management for SAP Cloud Platform ABAP Environment(a.k.a. Steampunk)
Photos by JJ Ying, Oscar Ivan Esquivel Arteaga, Robert Murray and Jesse Ramirez on Unsplash
- SAP Managed Tags:
7 Comments
| User | Count |
|---|---|
| 6 | |
| 5 | |
| 3 | |
| 3 | |
| 2 | |
| 2 | |
| 2 | |
| 1 | |
| 1 | |
| 1 |
