In this project you will build a complete, production-style CI/CD pipeline from scratch on AWS. A developer commits code to GitHub, Jenkins picks it up automatically, runs tests, scans code quality with SonarQube, builds a Docker image, scans it for vulnerabilities with Trivy, pushes it to Amazon ECR, and deploys it to an EC2 server — all fully automated without a single manual step after the git push. This is the pipeline stack that powers most Indian product companies that are not yet on Kubernetes — companies like mid-stage SaaS startups, fintech teams using EC2-based deployments, and backend teams at scale-ups. Developer (git push) | v GitHub Repo | Webhook trigger v +-------------------------------------------------+ | Jenkins Pipeline | | | | Stage 1: Checkout Code | | Stage 2: Run Unit Tests (Maven/npm) | | Stage 3: SonarQube Analysis | | Stage 4: Quality Gate (pass/fail) | | Stage 5: Build Docker Image | | Stage 6: Trivy Image Scan | | Stage 7: Push to AWS ECR | | Stage 8: Deploy to EC2 (SSH) | +-------------------------------------------------+ | v AWS ECR Registry | v Production EC2 Server (Docker container running)
Without a CI/CD pipeline, deploying means: SSH into the server, pull the latest code, run the build manually, restart the application, and hope nothing broke. Every deployment is different. There is no code quality check. There is no security scan. There is no record of what was deployed and when. This project solves all of that. Every git push triggers the full pipeline automatically. Code that does not pass tests or SonarQube quality gate does not get deployed. Every deployment is logged in Jenkins. Container images are scanned before being pushed. The entire history is auditable.
### Step 1: Provision the Infrastructure on AWS You will need three EC2 instances: * **Jenkins Server** — `t2.medium` (Jenkins needs 2GB RAM minimum) * **SonarQube Server** — `t2.medium` (SonarQube is memory-intensive) * **Deployment Server** — `t2.micro` (runs the final Docker container) ```bash ## Launch via AWS CLI (or use the console) ## Jenkins Server aws ec2 run-instances \ --image-id ami-0f5ee92e2d63afc18 \ --instance-type t2.medium \ --key-name your-key-pair \ --security-group-ids sg-xxxxxxxx \ --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=jenkins-server}]' ## SonarQube Server aws ec2 run-instances \ --image-id ami-0f5ee92e2d63afc18 \ --instance-type t2.medium \ --key-name your-key-pair \ --security-group-ids sg-xxxxxxxx \ --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=sonarqube-server}]' ``` **Security group rules needed:** * Jenkins server: port 22 (SSH), port 8080 (Jenkins UI) — from your IP * SonarQube server: port 22 (SSH), port 9000 (SonarQube UI) — from Jenkins server IP and your IP * Deployment server: port 22 (SSH), port 80 (app) — from Jenkins server IP and internet > ⚠️ **Security:** Never open port 8080 (Jenkins) or 9000 (SonarQube) to `0.0.0.0/0`. Restrict to your office IP or use a bastion host. These UIs have no rate limiting on login attempts by default. ### Step 2: Install and Configure Jenkins SSH into the Jenkins server: ```bash ssh -i your-key.pem ubuntu@JENKINS_SERVER_IP ## Update and install Java (Jenkins requires Java 17) sudo apt update && sudo apt upgrade -y sudo apt install -y openjdk-17-jdk java -version ## Add Jenkins repository and install curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | \ sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \ https://pkg.jenkins.io/debian-stable binary/ | \ sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null sudo apt update && sudo apt install -y jenkins ## Start Jenkins sudo systemctl start jenkins sudo systemctl enable jenkins sudo systemctl status jenkins ## Get the initial admin password sudo cat /var/lib/jenkins/secrets/initialAdminPassword ## Install Docker (Jenkins will build images) sudo apt install -y docker.io sudo systemctl start docker sudo usermod -aG docker jenkins # Allow Jenkins to run Docker commands sudo chmod 777 /var/run/docker.sock # Fix socket permissions ## Install AWS CLI curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip unzip awscliv2.zip && sudo ./aws/install ## Install Trivy for container scanning sudo apt-get install wget apt-transport-https gnupg lsb-release -y wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add - echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | \ sudo tee -a /etc/apt/sources.list.d/trivy.list sudo apt update && sudo apt install -y trivy trivy --version ``` **Open Jenkins UI** at `http://JENKINS_SERVER_IP:8080`, paste the initial password, and install the suggested plugins. Then install these additional plugins via **Manage Jenkins -> Plugins -> Available**: * SonarQube Scanner * Docker Pipeline * Amazon ECR * SSH Agent * Pipeline Utility Steps ### Step 3: Install and Configure SonarQube SSH into the SonarQube server: ```bash ssh -i your-key.pem ubuntu@SONARQUBE_SERVER_IP ## SonarQube requires Java 17 and specific system settings sudo apt update && sudo apt install -y openjdk-17-jdk ## Required system settings for Elasticsearch (used internally by SonarQube) sudo sysctl -w vm.max_map_count=262144 sudo sysctl -w fs.file-max=65536 echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.conf ## Install PostgreSQL as the SonarQube database sudo apt install -y postgresql postgresql-contrib sudo systemctl start postgresql ## Create the SonarQube database and user sudo -u postgres psql -c "CREATE USER sonarqube WITH PASSWORD 'sonarqube_password';" sudo -u postgres psql -c "CREATE DATABASE sonarqube OWNER sonarqube;" ## Download and install SonarQube wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-10.4.1.88267.zip unzip sonarqube-10.4.1.88267.zip sudo mv sonarqube-10.4.1.88267 /opt/sonarqube ## Create a dedicated user (SonarQube cannot run as root) sudo useradd -r -s /bin/false sonarqube sudo chown -R sonarqube:sonarqube /opt/sonarqube ## Configure database connection sudo sed -i 's/#sonar.jdbc.username=/sonar.jdbc.username=sonarqube/' \ /opt/sonarqube/conf/sonar.properties sudo sed -i 's/#sonar.jdbc.password=/sonar.jdbc.password=sonarqube_password/' \ /opt/sonarqube/conf/sonar.properties sudo sed -i 's|#sonar.jdbc.url=jdbc:postgresql|sonar.jdbc.url=jdbc:postgresql|' \ /opt/sonarqube/conf/sonar.properties ## Create systemd service cat <<EOF | sudo tee /etc/systemd/system/sonarqube.service [Unit] Description=SonarQube service After=syslog.target network.target [Service] Type=forking ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop User=sonarqube Group=sonarqube Restart=always LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF sudo systemctl daemon-reload sudo systemctl start sonarqube sudo systemctl enable sonarqube ## Wait 2-3 minutes for SonarQube to start, then access: ## http://SONARQUBE_SERVER_IP:9000 ## Default credentials: admin / admin (change immediately) ``` **Generate a SonarQube token for Jenkins:** 1. Log into SonarQube UI -> My Account -> Security -> Generate Token 2. Name: `jenkins-token`, Type: Global Analysis Token 3. Copy the token — you will add it to Jenkins credentials ### Step 4: Configure Jenkins Credentials and Connections In Jenkins UI, go to **Manage Jenkins -> Credentials -> Global -> Add Credentials**: | ID | Kind | Value | |---|---|---| | `sonarqube-token` | Secret text | SonarQube analysis token | | `aws-ecr-credentials` | AWS Credentials | IAM access key with ECR permissions | | `deployment-server-ssh` | SSH Username with private key | SSH key for deployment EC2 | **Configure SonarQube in Jenkins:** Manage Jenkins -> System -> SonarQube servers: * Name: `sonarqube` * Server URL: `http://SONARQUBE_SERVER_IP:9000` * Server authentication token: select `sonarqube-token` **Create the ECR repository:** ```bash ## Create ECR repository for the application aws ecr create-repository \ --repository-name my-app \ --region ap-south-1 \ --image-scanning-configuration scanOnPush=true ## Note the repository URI — you will use it in the Jenkinsfile ## Format: ACCOUNT_ID.dkr.ecr.ap-south-1.amazonaws.com/my-app ``` ### Step 5: Write the Production Jenkinsfile Create this `Jenkinsfile` at the root of your application repository: ```groovy pipeline { agent any environment { AWS_REGION = 'ap-south-1' ECR_REGISTRY = 'YOUR_ACCOUNT_ID.dkr.ecr.ap-south-1.amazonaws.com' IMAGE_NAME = 'my-app' IMAGE_TAG = "${BUILD_NUMBER}-${GIT_COMMIT.take(8)}" DEPLOY_SERVER = 'DEPLOYMENT_SERVER_IP' SONAR_PROJECT_KEY = 'my-app' } stages { stage('Checkout') { steps { checkout scm echo "Building commit: ${GIT_COMMIT}" } } stage('Unit Tests') { steps { sh 'mvn test -q' // Replace with npm test, pytest, etc. } post { always { junit 'target/surefire-reports/**/*.xml' } } } stage('SonarQube Analysis') { steps { withSonarQubeEnv('sonarqube') { sh ''' mvn sonar:sonar \ -Dsonar.projectKey=${SONAR_PROJECT_KEY} \ -Dsonar.projectName="My Application" \ -Dsonar.sources=src/main \ -Dsonar.tests=src/test ''' } } } stage('Quality Gate') { steps { timeout(time: 5, unit: 'MINUTES') { waitForQualityGate abortPipeline: true // Fail pipeline if quality gate fails } } } stage('Build Docker Image') { steps { sh "docker build -t ${IMAGE_NAME}:${IMAGE_TAG} ." sh "docker tag ${IMAGE_NAME}:${IMAGE_TAG} ${ECR_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}" sh "docker tag ${IMAGE_NAME}:${IMAGE_TAG} ${ECR_REGISTRY}/${IMAGE_NAME}:latest" } } stage('Trivy Security Scan') { steps { sh ''' trivy image \ --severity CRITICAL,HIGH \ --exit-code 1 \ --no-progress \ ${IMAGE_NAME}:${IMAGE_TAG} ''' } post { always { sh "trivy image --format json --output trivy-report.json ${IMAGE_NAME}:${IMAGE_TAG} || true" archiveArtifacts artifacts: 'trivy-report.json' } } } stage('Push to ECR') { steps { withCredentials([aws(credentialsId: 'aws-ecr-credentials', accessKeyVariable: 'AWS_ACCESS_KEY_ID', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY')]) { sh ''' aws ecr get-login-password --region ${AWS_REGION} | \ docker login --username AWS --password-stdin ${ECR_REGISTRY} docker push ${ECR_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} docker push ${ECR_REGISTRY}/${IMAGE_NAME}:latest ''' } } } stage('Deploy to EC2') { when { branch 'main' // Only deploy from main branch } steps { sshagent(['deployment-server-ssh']) { sh ''' ssh -o StrictHostKeyChecking=no ubuntu@${DEPLOY_SERVER} " aws ecr get-login-password --region ${AWS_REGION} | \ docker login --username AWS --password-stdin ${ECR_REGISTRY} && \ docker pull ${ECR_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} && \ docker stop my-app || true && \ docker rm my-app || true && \ docker run -d \ --name my-app \ --restart unless-stopped \ -p 80:8080 \ ${ECR_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} " ''' } } } } post { success { echo "Pipeline succeeded — ${IMAGE_NAME}:${IMAGE_TAG} deployed" } failure { echo "Pipeline failed — check logs above" } always { sh "docker rmi ${IMAGE_NAME}:${IMAGE_TAG} || true" // Clean up local image } } } ``` > 🔴 **Common Mistake:** Using `latest` as the only image tag. The pipeline above tags images with both `BUILD_NUMBER-GIT_SHA` and `latest`. Always keep the specific tag — it is how you roll back. If you push only `latest`, you cannot redeploy a previous version without rebuilding it. ### Step 6: Configure the GitHub Webhook In your GitHub repository, go to **Settings -> Webhooks -> Add webhook**: * Payload URL: `http://JENKINS_SERVER_IP:8080/github-webhook/` * Content type: `application/json` * Events: **Just the push event** In Jenkins, create a new **Pipeline** job: * Pipeline: **Pipeline script from SCM** * SCM: Git * Repository URL: your GitHub repo URL * Credentials: add your GitHub PAT * Branch: `*/main` * Script Path: `Jenkinsfile` * Check: **GitHub hook trigger for GITScm polling**
```bash ## 1. Trigger the pipeline with a git push git commit --allow-empty -m "trigger: test pipeline" git push origin main ## Watch Jenkins UI at http://JENKINS_SERVER_IP:8080 ## 2. Verify all stages passed in Jenkins Blue Ocean view ## Navigate to http://JENKINS_SERVER_IP:8080/blue ## 3. Check SonarQube results ## Navigate to http://SONARQUBE_SERVER_IP:9000 ## Your project should appear with quality gate PASSED ## 4. Verify the image was pushed to ECR aws ecr describe-images \ --repository-name my-app \ --region ap-south-1 \ --query 'imageDetails[*].[imagePushedAt,imageTags]' \ --output table ## 5. Verify the application is running on the deployment server ssh -i your-key.pem ubuntu@DEPLOYMENT_SERVER_IP "docker ps" ## Expected: my-app container running curl http://DEPLOYMENT_SERVER_IP ## Expected: Your application's response ## 6. Test Quality Gate blocking — introduce a vulnerability ## Add a hardcoded password to a source file: ## String password = "admin123"; ## Commit and push — the SonarQube stage should fail and block deployment git commit -am "test: quality gate block" git push ## Expected: Pipeline fails at Quality Gate stage, no deployment occurs ```
In this project you will build a complete, production-style CI/CD pipeline from scratch on AWS. A developer commits code...
Without a CI/CD pipeline, deploying means: SSH into the server, pull the latest code, run the build manually, restart th...
Step 1: Provision the Infrastructure on AWS You will need three EC2 instances: Jenkins Server — t2.medium (Jenkins needs...
...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.