DevOps — Change Velocity & CI/CD Orchestration (Zero → Pro)

اقرأ أولاً: ../MASTER_ALL_IN_ONE_SUMMARY.md · ../html/MASTER_ALL_IN_ONE_SUMMARY.html
الدليل الشامل: ../MASTER_ALL_IN_ONE.md · ../html/MASTER_ALL_IN_ONE.html
ممارسة الشاشات: ../html/MASTER_PRACTICE_GUIDE_ALL_TOPICS.html
Australia · Scope x_786686_hr_leave

Australia Instance — نسخة أستراليا

Direct Navigation URLs — روابط التنقل المباشرة

Platform target: ServiceNow Australia (Sep 2026) · next EA: Brazil
URLs: ../../SERVICENOW_DIRECT_URLS_CATALOG.md · 1-week: ../../SERVICENOW_ONE_WEEK_STUDY_DEPLOY_AUSTRALIA.md
HR Leave: Scope x_786686_hr_leave

CIS-DevOps Implementer Guide — ServiceNow Australia (post-Zurich)
الأسلوب: عربية + إنجليزية تكمّلان بعضهما — بدون ترجمة حرفية لأسماء الشاشات.

ملخص ووصف بالعربية (Arabic Summary)

ملخص في سطر: من Git commit حتى نشر إنتاج وقياس DORA — التغيير صار آليًا، آمنًا، ومُدقَّقًا.

وصف مختصر: يشرح كيف يربط ServiceNow أدوات السلسلة (Jenkins/GitHub Actions/GitLab CI/Azure DevOps)، يفتح Change Request تلقائيًا من البايبلاين، يطبّق سياسات الموافقة، يجمع نتائج الاختبارات وفحوصات الجودة والأمن، يحدّث CMDB من Artifact، ويولّد DORA metrics. مناسب CIS-DevOps والتنفيذ من الصفر.

1) Overview & Business Value

DevOps Change Velocity connects pipelines to ITSM so every build, test, and deploy is recorded as governed change. — هي الطبقة التي تجعل البايبلاين يفتح ويغلق Change Request تلقائيًا بدلاً من النموذج اليدوي.

Why customers buy it:

• Reduce mean change lead time from days to minutes by automating Standard Changes.
• اختصار زمن التغيير وتحويل الطلبات المتكرّرة إلى موافقات تلقائية.
• Provide auditable trail (SOX, ISO 27001, ITIL) — every artifact tied to a CR, every CR tied to a CI.
• إثبات للمراجع: كل نشر معروف من أين جاء، من وافق عليه، وماذا غيّر.
• Surface DORA metrics (Deployment frequency, Lead time, Change failure rate, MTTR) at executive level.
• مؤشرات DORA الأربعة جاهزة من Performance Analytics دون استخراج يدوي.
• Reduce change failure rate via gates: code quality (SonarQube), security (Snyk/Veracode), test coverage.
• بوابات قبل النشر تمنع التغييرات الخطرة من الوصول للإنتاج.
• Connect deployments to incidents and CIs through Service Graph for blast-radius awareness.
• ربط كل نشر بالخدمة المتأثرة لمعرفة من سيتألم لو فشل.

Business outcomes (typical 12-month):

• 60–80% of CRs become Standard auto-approved (was 0–15%).
• 25–40% reduction in change failure rate.
• 30–50% faster MTTR (because deploy → incident link is instant).
• Audit prep drops from weeks to hours.

⚠️ DevOps is not a replacement for Change Management — it accelerates it. Governance still applies.

2) Architecture & Data Model

Core tables (prefix sn_devops_):

sn_devops_tool — registered tool (Jenkins, GitHub, Jira ...)
sn_devops_pipeline — pipeline definition imported from tool
sn_devops_pipeline_execution — single run of a pipeline
sn_devops_step — definition of one step in a pipeline
sn_devops_step_execution — execution record per step
sn_devops_artifact — built artifact (jar, image, package)
sn_devops_artifact_version — versioned artifact (sha, tag)
sn_devops_package — group of artifacts deployed together
sn_devops_change_request — bridge table linking pipeline to change_request
sn_devops_change_ctrl — change control rules per step
sn_devops_orchestration_task — pending work for the pipeline
sn_devops_config — configuration items (feature flags, configs)
sn_devops_test_result — JUnit/TestNG ingest target
sn_devops_test_summary — per-pipeline-execution aggregated test
sn_devops_security_result — SonarQube/Snyk/Veracode ingest
sn_devops_insight_metric — pre-computed DORA metric rows
sn_devops_repo — SCM repository (GitHub, GitLab repo)
sn_devops_commit — commit metadata
sn_devops_pull_request — PR metadata
sn_devops_branch — branch metadata

Relationship to Change Management:

sn_devops_pipeline_execution
 └─ sn_devops_change_request (bridge)
 └─ change_request (ITSM)
 ├─ change_task (deployment steps)
 ├─ task_ci (impacted CIs from Service Graph)
 └─ sysapproval_approver

كل Pipeline Execution ينتج Change Request في change_request عبر جسر sn_devops_change_request.
ربط CI يتم تلقائيًا من Artifact → Service Graph → cmdb_ci.

Key foreign-keys & joins (CIS-DevOps exam favourite):

3) Plugins & Activation — كيف تفتح وتفعّل

How to activate:

Navigate: All > System Definition > Plugins
Search: com.snc.devops
Right-click → Activate/Repair
Then:
Navigate: All > System Applications > All Available Applications > All
Install: DevOps Change Velocity, DevOps Insights, DevOps Config
For spokes:
Navigate: All > IntegrationHub > Spokes Installer
Install: Jenkins, GitHub, GitLab spokes as needed

بعد تفعيل القاعدة (com.snc.devops) ركّب الـ spokes لكل أداة CI/CD تستخدمها.
لا تركّب spoke لا تحتاجه — وفّر طبقات وموارد.

⚠️ com.snc.devops يتطلّب اشتراك DevOps Change Velocity (مدفوع). جرّبه على PDI أولاً.

4) Guided Setup & All Options

Navigate: All > DevOps > Administration > Guided Setup — Open direct

Important configuration options:

ابدأ من Guided Setup ولا تتخطّاه — كل خطوة فيها فحوصات صحة.

5) End-to-End Process Cycle

The full pipeline → ITSM cycle:

Commit (developer)
 → SCM webhook to DevOps (sn_devops_commit row)
 → Build (Jenkins/GitHub Actions step → sn_devops_step_execution)
 → Test (JUnit/Cypress → sn_devops_test_result)
 → Code quality (SonarQube → sn_devops_security_result)
 → Security scan (Snyk/Veracode → sn_devops_security_result)
 → Artifact published (sn_devops_artifact_version)
 → Change Request (sn_devops_change_request → change_request)
 ├─ Standard? auto-approve via policy
 └─ Normal? CAB workflow
 → Deploy step blocked until CR approved
 → Deploy (artifact → cmdb_ci)
 → Verify (smoke tests, health checks)
 → Close CR (auto from pipeline)
 → Monitor (incidents linked back)
 → Feedback (Insights: DORA + change failure rate)

دورة كاملة من commit حتى مراقبة الإنتاج — كل خطوة لها سجلّ.

Stage breakdown:

⚠️ The pipeline is the source of truth for state, not the CR. ServiceNow mirrors pipeline state.

6) Development — Custom Step Definitions & Tools

Custom step types (for tools without out-of-box spoke):

// Script Include: CustomDevOpsStepHandler
var CustomDevOpsStepHandler = Class.create();
CustomDevOpsStepHandler.prototype = {
 initialize: function() {},

handleStepStart: function(payload) {
var step = new GlideRecord(‘sn_devops_step_execution’);
step.initialize();
step.pipeline_execution = payload.pipelineExecutionId;
step.tool = payload.toolId;
step.name = payload.stepName;
step.state = ‘in_progress’;
step.start_time = new GlideDateTime();
step.external_id = payload.externalId;
step.external_url = payload.url;
return step.insert();
},

handleStepEnd: function(payload) {
var step = new GlideRecord(‘sn_devops_step_execution’);
if (step.get(‘external_id’, payload.externalId)) {
step.state = payload.success ? ‘success’ : ‘failed’;
step.end_time = new GlideDateTime();
step.output = payload.output;
step.update();
}
},

type: ‘CustomDevOpsStepHandler’
};

Register a custom tool (Argo CD example):

var tool = new GlideRecord('sn_devops_tool');
tool.initialize();
tool.name = 'Argo CD — prod cluster';
tool.tool_type = 'orchestration';
tool.url = 'https://argo.example.internal';
tool.active = true;
tool.connection_status = 'connected';
var sysId = tool.insert();
gs.info('Argo tool sys_id: ' + sysId);

Inbound REST handler (custom webhook):

// Scripted REST resource: /api/x_acme/devops/argo
(function process(request, response) {
 var body = request.body.data;
 var handler = new CustomDevOpsStepHandler();
 if (body.event === 'sync_started') {
 handler.handleStepStart({
 pipelineExecutionId: body.pipelineExecutionId,
 toolId: body.toolId,
 stepName: 'Argo Sync ' + body.app,
 externalId: body.syncId,
 url: body.argoUrl
 });
 } else if (body.event === 'sync_completed') {
 handler.handleStepEnd({
 externalId: body.syncId,
 success: body.status === 'Synced',
 output: body.message
 });
 }
 response.setStatus(200);
})(request, response);

كل أداة لا تملك Spoke جاهز يمكن دمجها يدويًا عبر REST webhook + Script Include.

7) Administration — Roles, Tool Config

Core roles:

Tool registration checklist (per tool):

1) Generate ServiceNow integration user + role sn_devops.integration_user
2) Create OAuth credential record (System OAuth > Application Registry)
3) Register tool: All > DevOps > Administration > Tools > New
4) Set Tool Type (orchestration / scm / artifact / test / quality / security)
5) Test connection — should show "connected"
6) Map projects/repos to Application Service in CMDB
7) Confirm first webhook event arrives in Webhook Inspector

Common admin tasks:

• Rotate webhook secret quarterly (security baseline).
• Monitor sn_devops_step_execution for stuck state (“in_progress” > 24h) — see §17 script 16.
• Archive pipeline executions older than 365 days.
• Review policy approvals weekly — auto-approval should stay > 60%.
• Patch spokes monthly when Store updates publish.

⚠️ Do NOT use admin for integrations. Always use sn_devops.integration_user so audit shows source.

8) Hands-on Labs

Lab 1 — Connect Jenkins and import a pipeline

1) On Jenkins controller, install plugin "ServiceNow DevOps"
2) In Jenkins > Manage Jenkins > System, add ServiceNow:
 - Instance URL: https://<pdi>.service-now.com
 - Credentials: Username/Password (sn_devops.integration_user)
 - Tool ID: (created in next step)
3) In ServiceNow: All > DevOps > Administration > Tools > New
 - Name: Jenkins-PDI
 - Tool Type: orchestration
 - Active: true
4) Add a Pipeline job in Jenkins; in Jenkinsfile add snDevOpsStep blocks (see §17 script 3)
5) Run the job
6) In ServiceNow: All > DevOps > Pipelines — the job appears

Lab 2 — Auto-create Change Request from build

1) In Jenkinsfile add the snDevOpsChange step:
2) Trigger the build
3) In ServiceNow: change_request list — a new CR appears
 - Type: Normal (default until policy applies)
 - Description: auto-populated from Jenkins
4) Approve the CR
5) Jenkins pipeline resumes
6) Inspect bridge record: sn_devops_change_request

Lab 3 — Standard CR auto-approval policy

1) Navigate: All > DevOps > Change > Change Policies > New
2) Name: "Auto-approve docs changes"
3) Conditions:
 - Repo: docs-site
 - Branch: main
 - Tests passed: true
 - Code coverage >= 80
4) Action: Set CR type = Standard, approval = Approved
5) Trigger a build on docs-site/main with passing tests
6) Observe CR auto-approved within seconds; deploy step continues

Lab 4 — DORA metrics dashboard

1) Navigate: All > DevOps > Insights > Dashboards > DORA Metrics
2) Select Application Service: "HR Leave App"
3) Verify 4 widgets render:
 - Deployment Frequency
 - Lead Time for Changes
 - Change Failure Rate
 - MTTR
4) Trigger 3 builds in lab to populate data
5) Export dashboard as PDF for sponsor demo

Lab 5 — Block deploy on Sonar quality gate

1) Connect SonarQube as a quality tool
2) Add a quality gate to your pipeline policy:
 - Conditions: Sonar Quality Gate Status = "PASSED"
3) Push code with a critical bug; Sonar reports FAILED
4) Pipeline deploy step blocked; CR stays Awaiting approval
5) Fix bug; rerun; CR approves; deploy proceeds

9) Best Practices & Common Mistakes

Best practices:

• Start with one team and one pipeline; expand by team, not by feature.
• ابدأ بفريق واحد، لا تحاول تغطية الجميع من اليوم الأول.
• Map every pipeline to an Application Service in CMDB before turning on auto-CR.
• اربط كل بايبلاين بخدمة CMDB قبل تفعيل الـ CR التلقائي.
• Define Standard Change templates with explicit guardrails (tests, security, coverage).
• اشترط بوّابات اختبار + كود كوفر + فحص أمان قبل اعتبار التغيير Standard.
• Use Service Graph to compute blast radius — surface in CR risk score.
• استخدم Service Graph ليعرف Risk Score الفئة المتأثرة.
• Run DORA quarterly review with engineering leadership.
• اجتماع ربعي يعرض DORA على القيادة الهندسية.

Common mistakes:

• Treating every pipeline as Normal CR — kills velocity gain.
• اعتبار كل تغيير Normal CR — يلغي الفائدة من الأتمتة.
• Forgetting to rotate webhook secrets — security finding waiting to happen.
• نسيان تدوير أسرار الويب هوك — ثغرة أمنية مؤكدة في المراجعة.
• Pushing pipeline data without CMDB mapping — CRs orphaned from CIs.
• إدخال البايبلاين دون ربط CMDB — CR بدون CI = لا تتبّع تأثير.
• Letting sn_devops.integration_user get admin — over-privileged service account.
• منح المستخدم التكاملي صلاحيات أكثر من اللازم — أكبر خطأ أمني شائع.
• Allowing both ServiceNow and Jira to be source of truth — pick one master.
• ازدواج مصدر الحقيقة بين Jira و ServiceNow — اختر واحدًا.

⚠️ A CR without a CI link is a CR you cannot audit. Block at policy level.

10) CIS-DevOps Exam Focus

High-yield topics:

Likely exam scenarios:

• “A pipeline raises CR but it sits in New” — answer: missing policy or service account lacks role.
• “Auto-approval not firing” — answer: policy conditions, order, or Active = false.
• “CR closed but pipeline shows running” — answer: pipeline did not send close callback.
• “DORA shows 0 deploys” — answer: deployment step type not flagged as deploy on the step. — احفظ أسماء الجداول والأدوار بالضبط — يسأل عنها بالاسم الكامل.

11) Portals, Workspaces, Roles & Feature Catalog

Role hierarchy quick reference:

sn_devops.admin
 ├─ sn_devops.pipeline_owner
 ├─ sn_devops.user
 └─ sn_devops.read

change_manager
change_coordinator
change_user
itil

Feature catalog:

12) End-to-End Implementation (Post-Activation)

Phase 1 — Foundation (week 1):

1) Activate com.snc.devops on a clone
2) Install spokes for tools you use (Jenkins, GitHub, GitLab, Azure DevOps)
3) Create service accounts; assign sn_devops.integration_user
4) Register Tools (one for each Jenkins controller, GitHub org, etc.)
5) Validate webhook inspector receives first event

Phase 2 — First pipeline (week 2):

1) Pick a low-risk service (docs site or internal tool)
2) Add snDevOpsStep / snDevOpsChange to Jenkinsfile (or GH Actions yml)
3) Run the pipeline; confirm CR appears
4) Approve manually; deploy proceeds
5) Confirm close callback closes CR

Phase 3 — Policy automation (week 3):

1) Build first Change Policy: Standard auto-approve for docs site
2) Test 5 builds; verify auto-approval rate 100%
3) Expand policy to next service with stricter conditions
4) Document policy in KB for engineering org

Phase 4 — Quality & security gates (week 4):

1) Connect SonarQube and Snyk
2) Build quality + security policy:
 - Sonar Quality Gate = PASSED
 - Snyk Critical = 0
3) Test by intentionally failing each gate
4) Confirm CR blocked; pipeline halted

Phase 5 — Insights & DORA (week 5):

1) Tag pipeline steps with their type (build, test, deploy, verify)
2) Enable DORA collectors
3) Populate 4 weeks of data
4) Demo dashboard to engineering leadership

Phase 6 — Scale (week 6+):

1) Rollout to next team (use Phase 2 checklist)
2) Quarterly DORA review
3) Monthly policy review (drop policies that never fire)
4) Onboard new tools as needed

13) Options & Instance Notes — بدائل وميزات قد لا تظهر

• DevOps Change Velocity vs Change Acceleration: Velocity = base; Acceleration = policy engine. Some bundles require both.
• DevOps Insights vs Performance Analytics: Insights is preconfigured DORA, PA can be extended with custom indicators.
• Now Assist for DevOps: optional subscription; deployment risk score and CR summary.
• DevOps Config: separate license; feature flags, JSON config-as-code, LaunchDarkly integration.
• ServiceNow PDI does not include DevOps subscription — use trial or partner instance.
• Spokes from Store update independently — pin versions in change-controlled instances.

⚠️ Always check entitlement in your contract before promising features to stakeholders.

14) Implementation Scenarios

Scenario A — Single Jenkins → ServiceNow:

- Tool: Jenkins (1)
- Pipelines: 5–20
- Outcome: All builds raise CR; docs/configs auto-approve
- Time to value: 2 weeks

Scenario B — Multi-tool aggregation:

- Tools: GitHub (SCM), Jenkins (CI), Argo (CD), Jira (planning), Sonar, Snyk
- Pipelines: one logical pipeline graph from PR to prod
- Outcome: One CR per release containing all tools' evidence
- Time to value: 6–8 weeks

Scenario C — Federated enterprise:

- Tools: 10+ Jenkins controllers across regions
- Pipelines: 200+
- Outcome: Centralized policy, region-aware change windows
- Time to value: 6 months phased

Scenario D — Greenfield cloud-native (GitHub Actions only):

- Tools: GitHub Actions, GitHub Packages, Snyk, Datadog
- Pipelines: per-microservice, ~50
- Outcome: Standard CR is default; deploy hundreds of times per day
- Time to value: 4 weeks

Scenario E — Regulated (banking, ISO 27001, SOX):

- Tools: Azure DevOps, Veracode, Splunk
- Pipelines: per-application
- Outcome: Every artifact tied to CR, every CR tied to ticket, full audit pack export
- Time to value: 8–10 weeks

15) Zero-to-Hero End-to-End Implementation (Scratch → Go-Live)

End-to-end target: from blank instance to first auto-approved deploy in production within 30 days.

Milestones:

Exit criteria for go-live:

• All P1 services have an Application Service in CMDB.
• At least one Standard Change policy per service.
• Audit pack exportable from Insights.
• Runbook for rollback documented and tested twice.
• On-call DevOps SRE rotation set up. — لا تذهب للإنتاج قبل اختبار rollback حقيقي مرتين على الأقل في بيئة staging.

16) Full E2E Practice · Pages · Requirements · Options (AR + EN)

Pages and forms a CIS-DevOps candidate must navigate fluently:

Practice exercise (full E2E):

1) Stand up PDI, activate com.snc.devops + spokes
2) Connect GitHub (org level) and Jenkins (one controller)
3) Pick a sample repo "leave-app-docs" and a Jenkinsfile that builds + deploys to S3
4) Wire snDevOpsStep + snDevOpsChange in Jenkinsfile
5) Push a commit; observe CR raised in ServiceNow
6) Approve CR manually first time; confirm deploy proceeds
7) Build Standard policy on repo + branch + tests-passed; observe auto-approve next run
8) Connect SonarQube; add quality gate; deliberately fail Sonar; observe deploy blocked
9) Connect Datadog; correlate post-deploy incidents to the CR
10) Open DORA dashboard; capture screenshots for sponsor demo
11) Export Update Set "DevOps_Baseline_v1"

نفّذ التمرين على PDI كاملاً قبل دخول الامتحان — الأسئلة عملية وليست نظرية فقط.

§17 — Power User Playbook (DevOps)

Day-in-the-life recipes, role workflows, and 20 production-grade scripts.
ملف عملي مكثّف لكل دور — مطور، DevOps Engineer، SRE، Release Manager، Change Manager، Auditor.

17.1 Personas — what each role does in a day

كل دور له شاشة أساسية يفتحها كل صباح — لا تخلط بينها.

⚠️ Do not give Developers sn_devops.admin — sn_devops.pipeline_owner is enough for self-service.

17.2 Developer — day in the life

Step-by-step (commit to deploy):

Quick commands for developer:

git checkout -b feature/leave-balance-fix
git commit -am "fix: leave balance off by one"
git push origin feature/leave-balance-fix
gh pr create --fill --base main
gh pr merge --auto --squash

المطور لا يفتح ServiceNow أبدًا في الحالة العادية — كل شيء يبدأ من Git.

17.3 DevOps Engineer recipes — first hour with a new pipeline

Recipe A — Onboard a new Jenkins controller:

1) All > DevOps > Administration > Tools > New
2) Name: jenkins-eu-prod · Tool Type: orchestration · URL: https://jenkins-eu.example
3) Authentication: OAuth (recommended) or Basic with service account
4) Test connection — must return HTTP 200 from /api/json
5) On Jenkins controller, install "ServiceNow DevOps" plugin
6) Configure plugin with Tool ID from step 2 + service account creds
7) Pin plugin version in change control

Recipe B — Wire Jenkinsfile to ServiceNow:

1) Wrap each stage in snDevOpsStep
2) Add snDevOpsChange before deploy stage
3) Set snDevOpsArtifact when publishing build output
4) Set snDevOpsPackage before deploy to group artifacts
5) Commit Jenkinsfile, trigger first run, watch Webhook Inspector

Recipe C — Define your first Change Policy:

1) All > DevOps > Change > Change Policies > New
2) Name: "Docs site standard auto-approve"
3) Conditions:
 - Repository = docs-site
 - Branch = main
 - Test Pass Rate >= 95
4) Action: CR Type = Standard · State = Approved
5) Save, set Order = 100, Active = true
6) Run 3 builds; verify auto-approval ratio

ابدأ بسياسة واحدة بسيطة، ثم اضف الشروط تدريجيًا.

17.4 SRE recipes — deploy monitoring & rollback

Recipe A — Monitor active deploys:

1) /now/sow/home → Deployments tab (custom workspace)
2) Filter: state = In Progress AND environment = prod
3) Side panel: linked CIs, recent incidents, Datadog widget
4) Click pipeline link to view live step log

Recipe B — Trigger rollback (one-click):

// UI Action on change_request — "Rollback this deployment"
(function rollback(current) {
 if (!current.u_pipeline_execution) {
 gs.addErrorMessage('No pipeline link found.');
 return;
 }
 var roll = new GlideRecord('change_request');
 roll.initialize();
 roll.short_description = 'ROLLBACK: ' + current.short_description;
 roll.type = 'emergency';
 roll.priority = 1;
 roll.description = 'Auto-rollback of CR ' + current.number;
 roll.cmdb_ci = current.cmdb_ci;
 roll.parent = current.sys_id;
 var rollId = roll.insert();

var ihEvent = new sn_ih.IntegrationHubEvent(‘devops.rollback.trigger’);
ihEvent.setPayload({
originalExecutionId: current.u_pipeline_execution.toString(),
rollbackCr: rollId,
target: current.cmdb_ci.getDisplayValue()
});
ihEvent.send();

gs.addInfoMessage(‘Rollback CR ’ + roll.number + ’ created and pipeline triggered.’);
})(current); — السكربت يفتح CR جديد من نوع Emergency ويرسل حدث IH للبايبلاين الذي يعرف كيف يرجع للنسخة السابقة.

17.5 Release Manager recipes — release train

Recipe A — Build a release record from many CRs:

1) All > Change > Releases > New
2) Name: "2026.06 monthly platform release"
3) Window: 2026-06-25 22:00 → 02:00 AEST
4) Add CRs by filter: state = Approved AND target_release IS EMPTY AND planned_start_date BETWEEN window
5) Run conflict check (calendar overlap, freeze windows)
6) Notify owners via release notification template

Recipe B — Use Now Assist to summarize release risk:

// Background script — summarize release risk
var releaseSysId = '<release sys_id>';
var crs = new GlideRecord('change_request');
crs.addQuery('target_release', releaseSysId);
crs.query();
var notes = [];
while (crs.next()) {
 notes.push(crs.number + ' / ' + crs.cmdb_ci.getDisplayValue() + ' / risk=' + crs.risk);
}
var na = new sn_nowassist.NowAssistInline();
var summary = na.summarizeText('Release ' + releaseSysId + ' contains: ' + notes.join('; '));
gs.info('Risk summary: ' + summary);

17.6 Change Manager — governance day

- Open Change Workbench
- Tab: Awaiting CAB → 12 CRs
- For each: click Risk panel → see Now Assist risk score + CI blast radius
- Approve / Reject / Defer
- Tab: Auto-approval health → ratio per policy → flag policies < 80% success
- Schedule next CAB: pull Standard CRs from previous week into "FYI" log

مدير التغيير لا يوقف العمل — هو يحوكم السياسات.

17.7 Auditor — exporting audit pack

1) All > DevOps > Insights > Reports > Change Velocity
2) Filter: Last 90 days · Production environments only
3) Export Excel — columns: CR number, pipeline, commit hash, approver, deploy time, close time
4) Cross-check with GRC > Audit Engagement evidence requests
5) Pull Update Set logs for prod scope (sys_update_set)
6) Attach to GRC engagement

17.8 Top 20 DevOps scripts

1) Auto-create Change Request from pipeline build (Business Rule):

// Business Rule — after insert on sn_devops_pipeline_execution
(function createCR(current) {
 var cr = new GlideRecord('change_request');
 cr.initialize();
 cr.short_description = 'Pipeline ' + current.pipeline.getDisplayValue() + ' build ' + current.number;
 cr.description = 'Auto-raised by DevOps. Pipeline: ' + current.pipeline_url;
 cr.type = 'normal';
 cr.cmdb_ci = current.pipeline.application_service;
 cr.requested_by = current.triggered_by;
 cr.assignment_group = current.pipeline.assignment_group;
 var crId = cr.insert();

var bridge = new GlideRecord(‘sn_devops_change_request’);
bridge.initialize();
bridge.pipeline_execution = current.sys_id;
bridge.change_request = crId;
bridge.insert();
})(current);

2) Policy-based auto-approval for Standard Changes:

// Script Include: DevOpsPolicyEngine.evaluate
var DevOpsPolicyEngine = Class.create();
DevOpsPolicyEngine.prototype = {
 initialize: function() {},

evaluate: function(crSysId) {
var cr = new GlideRecord(‘change_request’);
if (!cr.get(crSysId)) return false;

var policy = new GlideRecord(‘sn_devops_change_policy’);
policy.addQuery(‘active’, true);
policy.orderBy(‘order’);
policy.query();
while (policy.next()) {
if (this._matches(cr, policy)) {
cr.type = ‘standard’;
cr.approval = ‘approved’;
cr.state = -3;
cr.work_notes = ‘Auto-approved by policy: ’ + policy.name;
cr.update();
return true;
}
}
return false;
},

_matches: function(cr, policy) {
return GlideFilter.checkRecord(cr, policy.condition);
},

type: ‘DevOpsPolicyEngine’
};

3) Pipeline status sync to CR (snDevOpsChange Jenkinsfile snippet):

pipeline {
 agent any
 stages {
 stage('Build') {
 steps { snDevOpsStep() { sh 'mvn -B clean package' } }
 }
 stage('Test') {
 steps { snDevOpsStep() { sh 'mvn -B test' } }
 post { always { snDevOpsTestSummaries(testSummaries: [[name: 'JUnit', testType: 'JUnit']]) } }
 }
 stage('Change') {
 steps { snDevOpsChange(applicationName: 'HR Leave App', changeRequestDetails: '{"setCloseCode": true}') }
 }
 stage('Deploy') {
 steps { snDevOpsStep() { sh './deploy.sh prod' } }
 }
 }
}

4) Artifact-to-CI mapping (automatic CMDB update):

// Business Rule — after insert on sn_devops_artifact_version
(function mapToCi(current) {
 var ci = new GlideRecord('cmdb_ci_appl');
 ci.addQuery('name', current.artifact.name);
 ci.setLimit(1);
 ci.query();
 if (ci.next()) {
 current.cmdb_ci = ci.sys_id;
 current.update();
 gs.info('Artifact ' + current.version + ' linked to CI ' + ci.name);
 } else {
 gs.warn('No CI for artifact ' + current.artifact.name);
 }
})(current);

5) Test result aggregator (JUnit → Insights):

// Scripted REST resource: POST /api/x_acme/devops/junit
(function process(request, response) {
 var data = request.body.data;
 var summary = new GlideRecord('sn_devops_test_summary');
 summary.initialize();
 summary.pipeline_execution = data.executionId;
 summary.passed = data.passed;
 summary.failed = data.failed;
 summary.skipped = data.skipped;
 summary.total = data.total;
 var sid = summary.insert();

data.cases.forEach(function(c) {
var r = new GlideRecord(‘sn_devops_test_result’);
r.initialize();
r.test_summary = sid;
r.name = c.name;
r.classname = c.classname;
r.status = c.status;
r.duration = c.duration;
r.insert();
});

response.setStatus(201);
})(request, response);

6) Code quality gate (SonarQube → block deploy):

// Flow Designer subflow (Script step): block_if_sonar_failed
(function execute(inputs, outputs) {
 var sonar = new GlideRecord('sn_devops_security_result');
 sonar.addQuery('pipeline_execution', inputs.executionId);
 sonar.addQuery('tool_type', 'quality');
 sonar.addQuery('gate_status', 'FAILED');
 sonar.query();
 outputs.block = sonar.hasNext();
 outputs.reason = outputs.block ? 'Sonar quality gate FAILED' : 'OK';
})(inputs, outputs);

7) Vulnerability gate (Snyk/Veracode block on critical CVE):

// Business Rule — before update on sn_devops_pipeline_execution where step = deploy
(function gateCves(current) {
 var sec = new GlideAggregate('sn_devops_security_result');
 sec.addQuery('pipeline_execution', current.sys_id);
 sec.addQuery('severity', 'critical');
 sec.addAggregate('COUNT');
 sec.query();
 sec.next();
 var count = parseInt(sec.getAggregate('COUNT'), 10) || 0;
 if (count > 0) {
 current.state = 'blocked';
 current.block_reason = count + ' critical CVE(s) found';
 gs.addErrorMessage('Deploy blocked: ' + count + ' critical CVEs');
 }
})(current);

8) Rollback automation (auto-create rollback CR + trigger):

// Scheduled Job — detect failed deploy in last 15 min and auto-rollback
var failed = new GlideRecord('sn_devops_pipeline_execution');
failed.addQuery('state', 'failed');
failed.addQuery('step_type', 'deploy');
failed.addQuery('end_time', '>', gs.minutesAgo(15));
failed.addQuery('rollback_triggered', false);
failed.query();
while (failed.next()) {
 var cr = new GlideRecord('change_request');
 cr.initialize();
 cr.short_description = 'AUTO-ROLLBACK: ' + failed.pipeline.getDisplayValue();
 cr.type = 'emergency';
 cr.priority = 1;
 cr.cmdb_ci = failed.pipeline.application_service;
 cr.insert();

var ev = new sn_ih.IntegrationHubEvent(‘devops.rollback.trigger’);
ev.setPayload({ executionId: failed.sys_id.toString(), crSysId: cr.sys_id.toString() });
ev.send();

failed.rollback_triggered = true;
failed.update();
}

9) Feature flag toggle automation (LaunchDarkly via DevOps Config):

// Background script — toggle feature flag at deploy time
var ld = new sn_devops_config.LaunchDarklyClient();
ld.toggleFlag({
 project: 'leave-app',
 environment: 'prod',
 flag: 'new-balance-engine',
 state: true,
 reason: 'CR0010234 deploy completed'
});

10) Deployment frequency (DORA) calculator:

// Scheduled Job — daily 02:00
var ga = new GlideAggregate('sn_devops_pipeline_execution');
ga.addQuery('step_type', 'deploy');
ga.addQuery('state', 'success');
ga.addQuery('end_time', '>=', gs.daysAgoStart(1));
ga.addQuery('end_time', '<', gs.daysAgoStart(0));
ga.addAggregate('COUNT');
ga.groupBy('pipeline.application_service');
ga.query();
while (ga.next()) {
 var m = new GlideRecord('sn_devops_insight_metric');
 m.initialize();
 m.metric_type = 'deployment_frequency';
 m.application_service = ga.getValue('pipeline.application_service');
 m.value = ga.getAggregate('COUNT');
 m.period_start = gs.daysAgoStart(1);
 m.period_end = gs.daysAgoStart(0);
 m.insert();
}

11) Lead time for changes (DORA):

// Scheduled Job — daily
var rs = new GlideRecord('sn_devops_pipeline_execution');
rs.addQuery('step_type', 'deploy');
rs.addQuery('state', 'success');
rs.addQuery('end_time', '>=', gs.daysAgoStart(1));
rs.query();
while (rs.next()) {
 var firstCommit = new GlideRecord('sn_devops_commit');
 firstCommit.addQuery('pipeline_execution', rs.sys_id);
 firstCommit.orderBy('commit_time');
 firstCommit.setLimit(1);
 firstCommit.query();
 if (firstCommit.next()) {
 var leadMs = new GlideDateTime(rs.end_time).getNumericValue() -
 new GlideDateTime(firstCommit.commit_time).getNumericValue();
 var m = new GlideRecord('sn_devops_insight_metric');
 m.initialize();
 m.metric_type = 'lead_time_for_changes';
 m.application_service = rs.pipeline.application_service;
 m.value = leadMs / 1000 / 60; // minutes
 m.execution = rs.sys_id;
 m.insert();
 }
}

12) Change failure rate (DORA):

// Scheduled Job — daily
var total = new GlideAggregate('change_request');
total.addQuery('closed_at', '>=', gs.daysAgoStart(1));
total.addAggregate('COUNT');
total.groupBy('cmdb_ci');
total.query();

var failed = {};
var f = new GlideAggregate(‘change_request’);
f.addQuery(‘closed_at’, ‘>=’, gs.daysAgoStart(1));
f.addQuery(‘close_code’, ‘IN’, ‘unsuccessful,backed_out’);
f.addAggregate(‘COUNT’);
f.groupBy(‘cmdb_ci’);
f.query();
while (f.next()) {
failed[f.getValue(‘cmdb_ci’)] = parseInt(f.getAggregate(‘COUNT’), 10);
}

while (total.next()) {
var ci = total.getValue(‘cmdb_ci’);
var t = parseInt(total.getAggregate(‘COUNT’), 10);
var fr = (failed[ci] || 0) / t;
var m = new GlideRecord(‘sn_devops_insight_metric’);
m.initialize();
m.metric_type = ‘change_failure_rate’;
m.application_service = ci;
m.value = fr;
m.insert();
}

13) MTTR for failures (DORA):

// Scheduled Job — daily
var inc = new GlideRecord('incident');
inc.addQuery('caused_by_change.isNotEmpty', true);
inc.addQuery('resolved_at', '>=', gs.daysAgoStart(1));
inc.query();
while (inc.next()) {
 var openMs = new GlideDateTime(inc.resolved_at).getNumericValue() -
 new GlideDateTime(inc.opened_at).getNumericValue();
 var m = new GlideRecord('sn_devops_insight_metric');
 m.initialize();
 m.metric_type = 'mttr';
 m.application_service = inc.business_service;
 m.value = openMs / 1000 / 60;
 m.incident = inc.sys_id;
 m.insert();
}

14) Pull Request → CR linker (multiple PRs to one CR):

// Business Rule — after insert on sn_devops_pull_request (merged)
(function linkPrToCr(current) {
 if (current.state != 'merged') return;
 var cr = new GlideRecord('change_request');
 cr.addQuery('cmdb_ci', current.repository.application_service);
 cr.addQuery('state', 'IN', '-3,-2,-1,0'); // open / approved
 cr.orderByDesc('sys_created_on');
 cr.setLimit(1);
 cr.query();
 if (cr.next()) {
 var link = new GlideRecord('sn_devops_pr_cr_link');
 link.initialize();
 link.pull_request = current.sys_id;
 link.change_request = cr.sys_id;
 link.insert();
 cr.work_notes = 'Linked PR #' + current.number + ' (' + current.title + ')';
 cr.update();
 }
})(current);

15) Multi-stage approval orchestrator:

// Subflow trigger: pipeline reaches "approval-gate" step
// Script step: route to right approver groups based on environment + risk
(function execute(inputs, outputs) {
 var cr = new GlideRecord('change_request');
 cr.get(inputs.crSysId);
 var groups = [];
 if (cr.risk == '2') groups.push('CAB');
 if (cr.cmdb_ci.support_group) groups.push(cr.cmdb_ci.support_group.toString());
 if (cr.business_service.environment == 'prod') groups.push('Production Owners');
 outputs.approverGroups = groups.join(',');
})(inputs, outputs);

16) Slack/Teams notification for stuck pipelines:

// Scheduled Job — every 15 min
var stuck = new GlideRecord('sn_devops_step_execution');
stuck.addQuery('state', 'in_progress');
stuck.addQuery('start_time', '<', gs.hoursAgo(1));
stuck.query();
while (stuck.next()) {
 var payload = {
 text: ':warning: Pipeline step stuck: ' + stuck.name +
 ' in ' + stuck.pipeline_execution.getDisplayValue() +
 ' (' + stuck.external_url + ')'
 };
 var r = new sn_ws.RESTMessageV2();
 r.setEndpoint(gs.getProperty('x_acme.slack.webhook'));
 r.setHttpMethod('POST');
 r.setRequestHeader('Content-Type', 'application/json');
 r.setRequestBody(JSON.stringify(payload));
 r.execute();
}

17) Auto-link incident to deployment (post-deploy issues):

// Business Rule — after insert on incident
(function linkInc(current) {
 var cutoff = new GlideDateTime();
 cutoff.addSeconds(-3600); // last hour
 var dep = new GlideRecord('sn_devops_pipeline_execution');
 dep.addQuery('step_type', 'deploy');
 dep.addQuery('state', 'success');
 dep.addQuery('pipeline.application_service', current.business_service);
 dep.addQuery('end_time', '>', cutoff);
 dep.orderByDesc('end_time');
 dep.setLimit(1);
 dep.query();
 if (dep.next()) {
 current.caused_by_change = dep.change_request.toString();
 current.work_notes = 'Auto-linked to recent deployment: ' + dep.pipeline.getDisplayValue();
 current.update();
 }
})(current);

18) Pipeline ownership audit:

// Background script — find pipelines with no owner or stale owner
var p = new GlideRecord('sn_devops_pipeline');
p.query();
var orphans = [];
while (p.next()) {
 if (p.owner.nil()) { orphans.push(p.name + ' :: NO_OWNER'); continue; }
 var u = new GlideRecord('sys_user');
 if (u.get(p.owner) && !u.active) orphans.push(p.name + ' :: INACTIVE_OWNER ' + u.user_name);
}
gs.info('Pipelines needing owner attention:\n' + orphans.join('\n'));

19) Service dependency-aware deployment ordering:

// Script Include: DevOpsDependencyOrdering.next
var DevOpsDependencyOrdering = Class.create();
DevOpsDependencyOrdering.prototype = {
 initialize: function() {},

next: function(releaseSysId) {
var crs = new GlideRecord(‘change_request’);
crs.addQuery(‘target_release’, releaseSysId);
crs.addQuery(‘state’, ‘-2’); // approved
crs.query();
var byCi = {};
while (crs.next()) byCi[crs.cmdb_ci.toString()] = crs.sys_id.toString();

var ordered = [];
var seen = {};
var self = this;
Object.keys(byCi).forEach(function(ci) { self._walk(ci, byCi, seen, ordered); });
return ordered;
},

_walk: function(ci, byCi, seen, ordered) {
if (seen[ci]) return;
seen[ci] = true;
var rel = new GlideRecord(‘cmdb_rel_ci’);
rel.addQuery(‘child’, ci);
rel.addQuery(‘type.name’, ‘Depends on::Used by’);
rel.query();
while (rel.next()) this._walk(rel.parent.toString(), byCi, seen, ordered);
if (byCi[ci]) ordered.push(byCi[ci]);
},

type: ‘DevOpsDependencyOrdering’
};

20) Cost-aware deployment recommendation:

// Background script — recommend deploy window by cloud cost (mock)
var costs = {
 aws_us_east_1: { peak: 1.0, off: 0.6 },
 aws_eu_west_2: { peak: 1.2, off: 0.7 }
};
var region = 'aws_us_east_1';
var now = new Date();
var hour = now.getUTCHours();
var isPeak = hour >= 12 && hour < 22;
var pricing = isPeak ? costs[region].peak : costs[region].off;
var saving = isPeak ? '0%' : Math.round((1 - costs[region].off / costs[region].peak) * 100) + '%';
gs.info('Region ' + region + ' pricing factor: ' + pricing + ' (saving vs peak: ' + saving + ')');
gs.info('Recommendation: ' + (isPeak ? 'Defer if non-urgent' : 'Deploy now — cost optimal'));

عشرون سكربتًا جاهزة للنسخ — كل واحد منها يحل مشكلة شائعة في الإنتاج.

17.9 DORA Metrics dashboard

Out-of-box dashboard structure:

Tab 1 — Velocity
 ├─ Deployment Frequency (per service, per env)
 ├─ Lead Time for Changes (median, p95)
Tab 2 — Stability
 ├─ Change Failure Rate (%)
 └─ MTTR (minutes, median)
Tab 3 — Quality
 ├─ Test pass rate trend
 ├─ Code coverage trend
 └─ Open critical vulns
Tab 4 — Adoption
 ├─ Auto-approval ratio per policy
 ├─ Pipelines with no CR mapping
 └─ Tools without traffic in 7 days

Recommended filters: Application Service, Environment, Team, Tool. — أضف فلتر فريق + بيئة لكل ودجت — بدونها الأرقام غير قابلة للمقارنة بين الفرق.

§18 — From-Scratch Implementation (Day 0 → Day 30)

A 30-day pragmatic DevOps Change Velocity rollout.
خطة بناء DevOps من الصفر في 30 يومًا — كل يوم له مخرجات وتسليم.

18.1 Day 0 — Maturity assessment & branching strategy

DevOps maturity self-assessment (score each 1–5):

Tool inventory:

Branching strategy review (pick one):

- Trunk-based (recommended for high velocity):
 main → feature → PR → merge → CI → deploy
- GitFlow (regulated industries):
 main, develop, feature/*, release/*, hotfix/*
- Release branches (long-lived versions):
 main, release/v1.x, release/v2.x

Success criteria (must agree before Day 1):

- One signed-off pilot team with one application service
- DevOps Change Velocity entitlement confirmed in contract
- Service accounts approved by security team
- Webhook egress allow-listed at network layer
- KB owners identified for runbooks

لا تنتقل ليوم 1 قبل الحصول على توقيع الراعي + تأكيد ترخيص DevOps.

⚠️ DevOps Change Velocity is a paid add-on — confirm SKU before committing to a date.

18.2 Day 1 — Activate DevOps plugin

Navigate: All > System Definition > Plugins
Activate (in dev/clone first):
- com.snc.devops (DevOps Change Velocity)
- com.snc.devops_change_acceleration (Policy engine)
- com.snc.devops.insights (DORA)
- com.snc.devops_config (Feature flags / config-as-code)
- com.snc.devops.integrationhub (IH spokes glue)

Then install spokes from Store (only those needed):
- ServiceNow IntegrationHub Installer
- Jenkins Spoke / GitHub Spoke / GitLab Spoke / Azure DevOps Spoke / Jira Spoke

⚠️ Activate on a dev or clone first. Capture in Update Set “DevOps_Activation_v1”.

18.3 Day 2–3 — Roles & groups

Create with:

Navigate: All > User Administration > Groups > New
Name: DevOps Admins
Roles: sn_devops.admin, admin
Members: <named users>

حدّد المالكين قبل تركيب أي أداة — أكبر سبب لفشل المشاريع هو غياب الـ owner.

18.4 Day 4–5 — Tool registrations (GitHub, Jenkins, Jira)

GitHub registration:

1) GitHub > Organization > Settings > Developer settings > GitHub Apps > New
2) Create app "ServiceNow DevOps Bridge"
3) Permissions: contents read, pull requests read, statuses write, webhooks write
4) Install app on target repos
5) ServiceNow: All > DevOps > Administration > Tools > New
 - Name: GitHub Enterprise — Engineering Org
 - Tool Type: SCM
 - Webhook URL: copy and paste into GitHub app config
 - Webhook Secret: generate, paste into both sides
6) Test by pushing a commit → Webhook Inspector should show event

Jenkins registration:

1) Jenkins > Manage Plugins > Available > "ServiceNow DevOps" > Install
2) Restart Jenkins
3) Manage Jenkins > System > ServiceNow DevOps section
 - Instance URL: https://<inst>.service-now.com
 - Tool ID: <created in ServiceNow next step>
 - Credentials: service account user/pass
4) ServiceNow: All > DevOps > Administration > Tools > New
 - Name: Jenkins — Eng Controller
 - Tool Type: orchestration
 - URL: https://jenkins.example
5) Copy Tool ID into Jenkins config
6) Save; "Test Connection" — must be green

Jira registration:

1) Jira > Settings > Apps > Manage apps > ServiceNow DevOps
2) Configure: ServiceNow URL + service account
3) ServiceNow: Tools > New
 - Name: Jira Cloud — Engineering
 - Tool Type: planning
4) Map Jira project keys to Application Services
5) Push a Jira issue transition → confirm event in Webhook Inspector

الصور تُذكر في الـ runbook الداخلي عند كل خطوة — هنا الخطوات فقط.

18.5 Day 6–7 — First pipeline integration (Jenkins → ServiceNow)

Pick a low-risk Jenkins job (e.g. docs site build).
Edit Jenkinsfile:

pipeline {
 agent any
 stages {
 stage('Build') { steps { snDevOpsStep() { sh 'make build' } } }
 stage('Test') { steps { snDevOpsStep() { sh 'make test' } }
 post { always { snDevOpsTestSummaries(testSummaries: [[name: 'JUnit', testType: 'JUnit']]) } } }
 stage('Deploy') { steps { snDevOpsStep() { sh 'make deploy' } } }
 }
}

Run the job 3 times. Verify in ServiceNow:
- Pipeline visible in DevOps > Pipelines
- Each run appears in Pipeline Executions
- Step states transition correctly
- Test summary populated

18.6 Day 8–9 — Auto-create CR from pipeline

Edit Jenkinsfile to add Change stage:

stage('Change') {
 steps {
 snDevOpsChange(applicationName: 'HR Leave App',
 changeRequestDetails: '{"setCloseCode": true, "type": "normal"}')
 }
}

1) Trigger pipeline
2) Pipeline pauses at Change stage
3) ServiceNow: new CR appears under change_request
4) Approve manually
5) Jenkins resumes; deploy starts
6) After deploy, CR auto-closes with close_code from pipeline

الـ snDevOpsChange هو الحدث الذي يربط البايبلاين بـ CR — تأكد أنه قبل خطوة Deploy.

18.7 Day 10–11 — Policy-based approval for Standard Changes

1) All > DevOps > Change > Change Policies > New
2) Name: "Docs Site Standard Auto-Approve"
3) Conditions (all must match):
 - Repository: docs-site
 - Branch: main
 - Test pass rate: >= 95
 - Code coverage: >= 70
 - Open critical CVEs: 0
4) Action:
 - Set Change Type: Standard
 - Set State: Approved
 - Add work note: "Auto-approved by policy docs-site-std"
5) Order: 100 · Active: true
6) Test: trigger 5 builds; verify auto-approve rate

⚠️ Always test policies in non-prod first — a bad condition can auto-approve everything.

18.8 Day 12 — Multi-tool aggregation (GitHub + Jenkins + Jira)

Goal: Single CR shows commit, build, test, and Jira issues together.

1) In Jenkinsfile, ensure environment exposes BUILD_TAG and GIT_COMMIT
2) snDevOpsStep auto-correlates GIT_COMMIT to sn_devops_commit
3) Jira spoke maps issue key from commit message (e.g. “ABC-123: fix”)
4) Open the auto-raised CR; related lists show:
- Pipeline execution
- Linked commits
- Linked PRs
- Linked Jira issues
- Test summary
- Security findings

18.9 Day 13–14 — Artifact tracking + CMDB mapping

1) Register artifact registry:
 All > DevOps > Artifacts > Registries > New
 - Name: JFrog — Prod
 - Type: artifactory
 - URL: https://artifactory.example
2) Pipeline publishes artifact; snDevOpsArtifact records version
3) Business rule (script 4 in §17) maps artifact to cmdb_ci
4) Verify: open Artifact Version → CI link populated
5) Verify: open CI → related list "Recent Artifacts" shows latest 5

18.10 Day 15 — Test result integration (JUnit, Cypress)

1) JUnit: snDevOpsTestSummaries collects test-reports/junit/*.xml
2) Cypress: add Mochawesome reporter; convert JSON to JUnit XML
3) For custom test tools, use Scripted REST (§17 script 5)
4) Verify summaries in Pipeline Execution
5) Build PA indicator: avg pass rate per service per week

18.11 Day 16 — Code quality + security gates

1) Connect SonarQube:
 - Tools > New: Tool Type = quality
 - Sonar webhook → /api/sn_devops/v1/devops/tool/sonar
2) Connect Snyk:
 - Tools > New: Tool Type = security
 - Add Snyk webhook → /api/sn_devops/v1/devops/tool/snyk
3) Build quality+security policy that blocks deploy when:
 - sonar.gate = FAILED OR snyk.critical > 0
4) Test by introducing a known critical vuln; confirm CR blocked
5) Document override path (manual emergency CR + dual approval)

18.12 Day 17–18 — Rollback automations

1) Build rollback subflow:
 Trigger: change_request update → close_code = unsuccessful
 Actions:
 - Create rollback CR (type emergency)
 - Call IH event devops.rollback.trigger
 - Notify on-call SRE via Slack
2) In Jenkinsfile, add stage('Rollback') that reads CR sys_id from event
3) Argo / Jenkins pulls previous artifact version and redeploys
4) Test by deliberately breaking a deploy in staging
5) Confirm rollback completes and CR auto-closes with cause

اختبر rollback مرتين في staging قبل أن يصل لـ prod — لا توجد فرصة ثانية.

18.13 Day 19 — Feature flags (LaunchDarkly integration)

1) DevOps Config > Connections > New: LaunchDarkly
2) API key stored in sys_credentials
3) Map projects/environments to ServiceNow CIs
4) In pipeline, after deploy: call snDevOpsConfigSetFlag
5) Build dashboard widget: open flags per service
6) Decommission stale flags after 90 days (script automated)

18.14 Day 20 — DORA metrics dashboard

1) Enable Insights collectors:
 - Deployment Frequency
 - Lead Time for Changes
 - Change Failure Rate
 - MTTR
2) Populate 7 days of historic data via background script
3) Open Insights Dashboards > DORA Metrics
4) Add filters: Application Service, Environment
5) Share with engineering leadership; collect feedback

18.15 Day 21 — Service Graph integration (deployment-aware CMDB)

1) Run Service Graph Connector for GitHub:
 - Maps repos → Application Services
2) Run Service Graph for Jenkins/Argo:
 - Maps pipelines → CIs
3) Verify cmdb_ci_appl rows have:
 - .source_url (Git repo)
 - .build_pipeline (Jenkins)
 - .deployment_pipeline (Argo)
4) Visualize in Service Map: see code → build → deploy → runtime

18.16 Day 22 — Notification rules (Slack/Teams/Email)

1) Build notification:
 - Trigger: change_request created from pipeline
 - Recipients: pipeline owner + assignment group
 - Channel: Slack via webhook (sys_properties holds URL)
2) Build escalation:
 - Stuck step > 1 hour → notify SRE on-call
3) Build digest:
 - Daily 18:00 → release manager: today's deploys + failures
4) Test each notification path before go-live

18.17 Day 23 — Custom dashboards (release health, pipeline health, MTTR)

1) Build Performance Analytics workspace "Release Health"
2) Widgets:
 - Pipelines passing vs failing (last 7d)
 - Mean stage duration trend
 - Top 5 noisy steps (most retries)
 - Change failure rate by team
 - MTTR by service
3) Schedule weekly email of dashboard PDF to leadership

18.18 Day 24 — PA Indicators

Build each in PA > Indicators > New.

18.19 Day 25 — Now Assist (deployment risk score, change suggestions)

Pre-req: Now Assist for DevOps subscription
1) Enable plugin com.snowflake.nowassist.devops
2) Configure model assignments:
 - CR risk scorer
 - Deployment summarizer
3) On CR form, "Risk" panel shows AI-generated risk + reasoning
4) On Pipeline Execution form, "Summary" shows one-paragraph deploy summary
5) Train approvers: AI score is advisory, not deterministic

18.20 Day 26 — ATF (DevOps workflow tests)

1) All > Automated Test Framework > Tests > New: "DevOps Auto-Approve Flow"
2) Steps:
 - Create test pipeline execution via REST
 - Simulate test results 100% pass
 - Assert CR raised with type=normal
 - Wait for policy evaluation
 - Assert CR auto-approved within 30s
3) Build "Rollback Trigger" test
4) Add tests to nightly suite; failures notify DevOps Admins

18.21 Day 27 — Training (devs, DevOps engineers, change board)

Training tracks:
- Developer (45 min): how to read a CR, why builds wait, how to fix a failed gate
- DevOps Engineer (3 hours): tools, pipelines, policies, troubleshooting
- Change Manager (90 min): policy review, auto-approval governance, override path
- Auditor (60 min): how to pull change velocity report, evidence locations
- SRE (2 hours): rollback runbook, incident-to-deploy linkage
Materials: Loom videos + KB articles + one printed cheat sheet per role

18.22 Day 28 — Pilot with 1 team

1) Pick one engineering team with one application service
2) Enable DevOps end-to-end for them only (limit policies by repo)
3) Daily 15-min stand-up with team for 1 week
4) Track: time saved per CR, auto-approval rate, blocker count
5) Capture testimonials + screenshots for rollout deck

18.23 Day 29 — Rollout planning

1) Rank remaining teams by readiness (CI maturity, test coverage)
2) Wave plan:
 - Wave 1 (week 1–2): 3 teams similar to pilot
 - Wave 2 (week 3–4): 5 mid-maturity teams
 - Wave 3 (month 2): rest
3) Per-team onboarding checklist:
 - Tools registered
 - Pipelines mapped to services
 - Policies tuned to team risk
 - Owners trained
4) Communications plan: kick-off email, Slack channel, office hours twice weekly

18.24 Day 30 — Go-Live + Hypercare

Go-live checklist:
- Update Set promoted to prod with sign-off
- Webhook secrets rotated and stored in vault
- Service accounts confirmed least-privilege
- Rollback runbook tested in prod-like environment
- On-call DevOps SRE rota live
- Monitoring dashboards green for 48h
- Sponsor sign-off recorded

Hypercare (2 weeks):
- Daily 15-min sync with pilot teams
- Weekly DORA review with leadership
- Burn down policy tuning backlog
- Capture lessons learnt for next wave

Exit hypercare when:
- Auto-approval ratio > 60% sustained
- Change failure rate within target band
- < 3 high-severity DevOps tickets per week
- Engineering NPS for DevOps > +20 — نهاية اليوم 30 ليست نهاية المشروع — هي بداية تشغيل مستدام. استمر بتحسين السياسات شهريًا ومراجعة DORA ربعيًا.

⚠️ Do not declare success until two full release cycles pass without manual workaround.

Appendix A — Jenkinsfile (full example)

pipeline {
 agent any
 options {
 timestamps()
 timeout(time: 60, unit: 'MINUTES')
 }
 environment {
 SN_INSTANCE = 'https://acme.service-now.com'
 APP_NAME = 'HR Leave App'
 }
 stages {
 stage('Checkout') {
 steps { checkout scm }
 }
 stage('Build') {
 steps {
 snDevOpsStep() {
 sh 'mvn -B clean package'
 }
 }
 }
 stage('Unit tests') {
 steps {
 snDevOpsStep() { sh 'mvn -B test' }
 }
 post {
 always {
 snDevOpsTestSummaries(testSummaries: [[name: 'JUnit', testType: 'JUnit']])
 }
 }
 }
 stage('Quality') {
 steps {
 snDevOpsStep() {
 sh 'mvn sonar:sonar -Dsonar.projectKey=hr-leave-app'
 }
 }
 }
 stage('Security') {
 steps {
 snDevOpsStep() {
 sh 'snyk test --severity-threshold=critical'
 }
 }
 }
 stage('Artifact') {
 steps {
 snDevOpsArtifact(artifactsPayload: """{
 "artifacts": [{
 "name": "hr-leave-app",
 "version": "${BUILD_NUMBER}",
 "semanticVersion": "1.0.${BUILD_NUMBER}",
 "repositoryName": "hr-leave-app-releases"
 }],
 "branchName": "${BRANCH_NAME}"
 }""")
 }
 }
 stage('Change') {
 steps {
 snDevOpsChange(applicationName: "${APP_NAME}",
 changeRequestDetails: '{"setCloseCode":true,"attributes":{"chg_model":"prb_chg_model_normal"}}')
 }
 }
 stage('Deploy') {
 steps {
 snDevOpsStep() {
 sh './deploy.sh prod'
 }
 }
 }
 stage('Verify') {
 steps {
 snDevOpsStep() {
 sh './smoke-tests.sh prod'
 }
 }
 }
 }
 post {
 failure {
 snDevOpsStep() {
 sh 'echo "Deployment failed — rollback will be triggered by ServiceNow"'
 }
 }
 }
}

Appendix B — GitHub Actions workflow (full example)

name: hr-leave-app-pipeline
on:
 push:
 branches: [ main, release/** ]
 pull_request:
 branches: [ main ]

permissions:
contents: read
pull-requests: write

jobs:
build-test-deploy:
runs-on: ubuntu-latest
env:
SN_INSTANCE_URL: ${{ secrets.SN_INSTANCE_URL }}
SN_DEVOPS_USER: ${{ secrets.SN_DEVOPS_USER }}
SN_DEVOPS_PASS: ${{ secrets.SN_DEVOPS_PASS }}
SN_DEVOPS_TOOL_ID: ${{ secrets.SN_DEVOPS_TOOL_ID }}
steps:
- name: Checkout
uses: actions/checkout@v4

• name: ServiceNow DevOps — Register Build
  uses: ServiceNow/servicenow-devops-build@v4
  with:
  devops-integration-user-name: ${{ env.SN_DEVOPS_USER }}
  devops-integration-user-password: ${{ env.SN_DEVOPS_PASS }}
  instance-url: ${{ env.SN_INSTANCE_URL }}
  tool-id: ${{ env.SN_DEVOPS_TOOL_ID }}
  context-github: ${{ toJSON(github) }}
  job-name: ‘Build’

• name: Build
  run: ./gradlew build

• name: Test
  run: ./gradlew test

• name: ServiceNow DevOps — Test Summaries
  uses: ServiceNow/servicenow-devops-test-report@v4
  with:
  devops-integration-user-name: ${{ env.SN_DEVOPS_USER }}
  devops-integration-user-password: ${{ env.SN_DEVOPS_PASS }}
  instance-url: ${{ env.SN_INSTANCE_URL }}
  tool-id: ${{ env.SN_DEVOPS_TOOL_ID }}
  context-github: ${{ toJSON(github) }}
  xml-report-filename: build/test-results/test/TEST-*.xml
  job-name: ‘Test’

• name: SonarQube scan
  run: ./gradlew sonarqube

• name: Snyk security scan
  uses: snyk/actions/gradle@master
  with:
  command: test
  args: –severity-threshold=critical

• name: ServiceNow DevOps — Register Artifact
  uses: ServiceNow/servicenow-devops-register-artifact@v4
  with:
  devops-integration-user-name: ${{ env.SN_DEVOPS_USER }}
  devops-integration-user-password: ${{ env.SN_DEVOPS_PASS }}
  instance-url: ${{ env.SN_INSTANCE_URL }}
  tool-id: ${{ env.SN_DEVOPS_TOOL_ID }}
  context-github: ${{ toJSON(github) }}
  job-name: ‘Artifact’
  artifacts: |
  [
  {
  “name”: “hr-leave-app”,
  “version”: “${{ github.run_number }}”,
  “semanticVersion”: “1.0.${{ github.run_number }}”,
  “repositoryName”: “hr-leave-app-releases”
  }
  ]

• name: ServiceNow DevOps — Change Request
  uses: ServiceNow/servicenow-devops-change@v4
  with:
  devops-integration-user-name: ${{ env.SN_DEVOPS_USER }}
  devops-integration-user-password: ${{ env.SN_DEVOPS_PASS }}
  instance-url: ${{ env.SN_INSTANCE_URL }}
  tool-id: ${{ env.SN_DEVOPS_TOOL_ID }}
  context-github: ${{ toJSON(github) }}
  job-name: ‘Change’
  change-request: |
  {
  “setCloseCode”: “true”,
  “attributes”: {
  “short_description”: “Deploy HR Leave App build ${{ github.run_number }}”,
  “description”: “Auto-raised from GitHub Actions”,
  “chg_model”: “prb_chg_model_normal”
  }
  }

• name: Deploy to production
  run: ./deploy.sh prod

• name: Smoke test
  run: ./smoke.sh prod

Appendix C — Quick command reference

احفظ هذا الجدول — معظم العمليات اليومية تبدأ من هنا.

End of DevOps module.
نهاية ملف الـ DevOps.

§19 — CIS-DevOps Exam & Z-to-Hero Feature Labs (20 inline Q)

19.1 Z-to-Hero Feature Labs (10 labs)

19.2 20 DevOps Practice Questions

1. DevOps Change Velocity table: A. sn_devops_change_request ext of change_request | B. Independent | C. None | D. Random — Answer: A
2. DORA metrics: A. Deployment Frequency, Lead Time for Changes, MTTR, Change Failure Rate | B. Just speed | C. Random | D. None — Answer: A
3. Pipeline ingestion: A. Inbound webhook from Jenkins/Azure/GitHub | B. Polling | C. Manual | D. None — Answer: A
4. Auto-CR creation: A. From pipeline event + risk assessment | B. Manual only | C. Random | D. None — Answer: A
5. DevOps Config: A. Track config-as-code via Tool | B. Manual | C. Random | D. None — Answer: A
6. Velocity vs Stability tradeoff: A. Optimize both via gates | B. Speed-only | C. Random | D. None — Answer: A
7. CR Risk Score inputs: A. Files changed, blast radius, deploy frequency, prior failures | B. Random | C. Manual | D. None — Answer: A
8. Standard Change template: A. Pre-approved, low-risk, frequent | B. Always emergency | C. Random | D. None — Answer: A
9. ServiceNow SDK + Now Pipelines: A. CI/CD for ServiceNow apps | B. None | C. Random | D. Manual — Answer: A
10. Tool table: A. sn_devops_tool (Jenkins, GitHub, Sonar, etc.) | B. Random | C. None | D. Manual — Answer: A
11. Insights dashboard: A. Engineering productivity by team/pipeline/service | B. None | C. Random | D. Manual — Answer: A
12. CR auto-approval gate: A. Test pass + coverage threshold + security scan clean | B. Manual only | C. Random | D. None — Answer: A
13. Pipeline event types: A. Build, Test, Deploy, Approval, Rollback | B. Just deploy | C. Random | D. None — Answer: A
14. Source of Truth for pipeline metadata: A. ServiceNow CMDB (cmdb_ci_devops_) | B. Excel | C. Random | D. None — Answer: A*
15. Multi-tool aggregation: A. One CR shows all tool stages | B. Multiple CRs | C. Random | D. None — Answer: A
16. Deploy to ServiceNow itself: A. Update Sets via Now Pipelines or REST | B. Manual only | C. Random | D. None — Answer: A
17. Rollback workflow: A. CR with category=rollback linked to original | B. Random | C. None | D. Manual — Answer: A
18. Value Stream metric: A. End-to-end flow time per change | B. Just lead time | C. Random | D. None — Answer: A
19. Release coordination: A. Multiple linked CRs + dependency map | B. Single CR | C. Random | D. None — Answer: A
20. ChatOps integration: A. Teams/Slack bot for CR approval | B. Email only | C. Random | D. None — Answer: A

19.3 What “Hero” looks like for DevOps

Reduces CR cycle 50%+; auto-CR from pipelines; risk-aware approval gates; DORA dashboard for execs; multi-tool ingestion; ChatOps approvals; Now Pipelines for SN apps; value-stream-driven improvements.

Hands-on Labs — مختبرات عملية

Step-by-step labs for PDI / Australia instance. Replace {BASE} with your instance URL.
مختبرات خطوة بخطوة — استبدل {BASE} برابط نسختك.

1. Lab 1 — Tool integration / تكامل أداة — Register Jenkins/GitHub at {BASE}/sn_devops_tool_integration_list.do.

2. Lab 2 — Pipeline mapping / ربط pipeline — Map repo branch to ServiceNow application.

3. Lab 3 — Standard Change template / قالب Standard — Create auto-approved template at {BASE}/std_change_record_producer_list.do.

4. Lab 4 — Pipeline run / تشغيل pipeline — Trigger build; verify CR auto-created.

5. Lab 5 — Quality gate / بوابة جودة — Block deploy on failed SonarQube/Snyk scan.

6. Lab 6 — Artifact → CI / Artifact → CI — Confirm deployment updates cmdb_ci.

7. Lab 7 — DORA dashboard / لوحة DORA — Review metrics at {BASE}/$pa_dashboard.do.

Best Practices — أفضل الممارسات

Deployment Checklist — قائمة النشر

Direct URLs Quick Reference