Your Site's RSS Feed

Deploying Azure Log Analytics Solution Using Terraform

Sat, 16 Apr 2022 00:00:00 GMT

If it's needed to deploy Azure Log Analytics Solution using Terraform, you can use the following Terraform structure,

resource "azurerm_resource_group" "rg" {
  name     = "rg-sample"
  location = "westus3"
}

resource "azurerm_log_analytics_workspace" "workspace" {
  name                = "alaw-sample"
  location            = "${azurerm_resource_group.rg.location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  sku                 = "Free"
}

resource "azurerm_log_analytics_solution" "solution" {
  solution_name         = "ContainerInsights"
  location              = "${azurerm_resource_group.rg.location}"
  resource_group_name   = "${azurerm_resource_group.rg.name}"
  workspace_resource_id = "${azurerm_log_analytics_workspace.workspace.id}"
  workspace_name        = "${azurerm_log_analytics_workspace.workspace.name}"

  plan {
    publisher = "Microsoft"
    product   = "OMSGallery/ContainerInsights"
  }
}

It's possible to deploy an Azure Resource Group, an Azure Log Analytics Workspace and an Azure Log Analytics Solution (for Container Monitoring) using the following command,

terraform init
terraform apply -auto-approve

Unfortunately, list of Azure Log Analytics Solutions is not available in the Terraform provider documentation 😟

Official Terraform Resource Provider page for Azure Log Analytics Solution displays only ContainerInsights example, azurerm_log_analytics_solution

If there is already an Azure Subscription, that has Azure Log Analytics Solutions installed, and it's needed to create a Terraform Module out of it, following command will list the installed Azure Log Analytics Solutions,

# List all log-analytics solutions in the current subscription
az monitor log-analytics solution list

# List all log-analytics solutions for a specific subscription
az monitor log-analytics solution list --subscription ${SUBSCRIPTION_ID}

# List all log-analytics solutions in a resource group
az monitor log-analytics solution list --resource-group ${RESOURCE_GROUP_NAME}

When the command above is executed, it will list the installed Azure Log Analytics Solutions, and it's possible to create a Terraform Module using the output of the command,

resource "azurerm_log_analytics_solution" "solution" {
  solution_name         = "{Product name from the command output}"
  location              = "${azurerm_resource_group.rg.location}"
  resource_group_name   = "${azurerm_resource_group.rg.name}"
  workspace_resource_id = "${azurerm_log_analytics_workspace.workspace.id}"
  workspace_name        = "${azurerm_log_analytics_workspace.workspace.name}"

  plan {
    publisher = "{Publisher name from the command output}"
    product   = "{Product name from the command output}"
  }
}

Note: Before running Azure CLI commands, Log Analytics Solution Extension should be installed; az extension add --name log-analytics-solution

What makes a good developer?

Tue, 24 Aug 2021 00:00:00 GMT

The world is moving towards a more technical future and in order to keep up with the trend, more and more people are looking to learn how to code. Coding, engineering, design and development are some of the most in demand skills in the job market today and learning how to code can help you become more employable in the real world.

Curiosity

Remember that your job does not just stop at writing code. It’s important to know how to ask the right questions when your code isn’t working. If you have more than one idea or solution, try them all. When you’re working on a project, you're going to have questions. You should feel comfortable asking your superiors, colleagues or clients questions about the project. In order to be a good developer, you need to be willing to experiment and explore

Once you have the information and you know what you’re doing, it’s time to get started. You may have the required skills and you may be able to right the wrongs. But there will be cases where you need to ask for help. As a developer, you need to be able to not be afraid to do it. Remember to pause and ask "why?" when something doesn’t look right, and keep digging until you find the answer.

Analytical mindset

A developer has many skills, but the two most important ones are understanding the system and understanding the details. At the system-level, it's important to understand how the systems you're working with are put together. You need to have a deep knowledge of the topics you're working with in order to understand how they relate to the big picture. You also need to be able to balance thinking about the big picture and the details.

A developer should be someone who can think critically. A developer should have the necessary systems-level thinking skills to be able to look at a problem and come up with a plan to solve it. Even if you only have an idea about how things should work, you need to be able to think about the big picture and the small details of what that means. This type of job requires a lot of things that might not be your expertise in your daily life.

Creativity

To me, creativity is the most important skill that a programmer can have. When you are faced with a problem, it is often the creative programmer that can come up with the best solution.

Creativity is the ability to come up with new and interesting ideas. It is the power to see old things in new ways. It lies at the heart of innovation and is a driving force behind many great discoveries. It usually involves seeing relationships or connections among things that others do not notice.

Programmers tend to be analytical and logical. The best programmers I know tend to be great at solving problems in different ways. They can break down a problem into its simplest parts and then work their way up to the final, full solution. This type of thinking is very important in programming because there are many different ways to solve a problem.

Informed decisions

A modern developer takes a lot of responsibility. They are required to keep up with the latest changes in their areas of expertise, be it language, frameworks, or libraries. This does not mean that they have to constantly be on the hype train or jump on every new library that becomes popular in the blogs or social media. A good developer is informed, so they make informed decisions and guide the project towards the correct and future-proof direction.

It is impossible to know everything. It is important to choose your area of expertise and focus on that. When working on a team, it is important to know what you do not know, but equally important to know what your peers do know. Since most software projects are a team effort, it is important that each member of the team can complement the knowledge of their peers. This ensures that the team as a whole has a solid foundation of knowledge.

Adaptability

For developers, jumping from one project to another is part of the job. You need to have the ability to switch contexts or the scope of a project unexpectedly, and then go back to pick up where you left off. You should always be prepared to adapt and respond to issues when they arise. This ability is important in the tech industry in particular because things can change quickly. Sometimes they change on a dime. The tech industry is unpredictable, so having the ability to adjust to the needs of your business

In the realm of tech, there are very few projects that have steps that are able to be completed one after the other without adapted or changing. As a developer, it’s important to have the ability to switch contexts or the scope of a project unexpectedly, and then go back to pick up where you left off. This is a skill that every developer should develop, because the tech industry can be unpredictable, so being able to adjust to the needs of your business or product when they change is

Knowledge

Knowledge is crucial to being able to apply your creativity. An understanding of the why? behind how you are solving a problem is the difference between a good programmer and a great one. You should not only understand the syntax of the language you are using but also understand how and why it works the way it does.

Efficiency

Programming is the practice of designing and building an application, software, or website with programming language. Programmers need to be able to determine what the problem is and then find a solution to it. A programmer must think logically and clearly so they can create logical codes. A good programmer must be able to write programs fast and efficiently. A good programmer must have time management skills. Programs can be tested and fixed several times, but a good programmer must have a work plan.

Testing

Testing is a critical part of the software development lifecycle. Whether it’s done by a developer, an engineer, or a QA team, testing is incredibly important to create high-quality, bug-free products. Whether you’re building an app, a website, or a piece of hardware, testing is essential to ensure the quality of your product.

Communication

We often work in teams, but communication is a problem. We are all different and we don't all use the same words to express ourselves. It's hard to find a common language and things can get mixed up. We need to know the language of each person and we need to learn to listen. We need to be aware when someone is saying something, or we may miss what they are saying, or they may miss what we are saying.

How to disable Message of the Day prompt?

Tue, 17 Aug 2021 00:00:00 GMT

The abbreviation "motd" stands for "message of the day", and this file has been traditionally used for exactly that (it requires much less disk space than mail to all users).

The contents of /run/motd are displayed by login after a successful login but just before it executes the login shell.

- Ubuntu Manpage for Message of the Day

The /run/motd file is a text file with contents that can be displayed to all users logging in on the system. The content of the file is usually a system message, which can be used to convey information about the current state of the system to users logging in.

Execute the following command on a Terminal window to display the content of /run/motd;

cat /run/motd

This file is used to display system information, to alert users of system-wide changes, or to advertise information. In some cases, system administrators might want to display specific message to users after successful login.

To disable the "message of the day" prompt, execute the following command on a Terminal window and create a blank .hushlogin file at the root folder of the current user;

touch ~/.hushlogin

References

Ubuntu Message of the Day ManPage

How to find and delete files by extension

Tue, 10 Aug 2021 00:00:00 GMT

Creating, modifying and deleting files are everyday tasks performed in any operating system mostly used by System Administrators.

Linux operating system provides users an easy way to delete files and directories. This can be done using the rm command.

The rm command is the remove command in the Linux operating system. It can be used to delete files, directories and links.

System Administrator (sysadmins), often need to clean things up to keep the system clean. This cleaning process may take quite a while, including finding and deleting files recursively in a folder structure.

One method for removing batches of files based on file extensions, or filename patterns, is the find command.

The find command will search for all files and directories within a specific directory, and possibly even within nested subdirectories.

To begin, you will need to identify the specific directory that you want to search, such as;

cd "/root_folder"

If you wanted to search for all files with a ".log" extension, you would execute;

find . -name "*.log" -type f

With the " . " we indicate that it has to search from the directory in which we are, and, with the "*.log" we tell it to search for all files with the ".log" extension.

Finally, to delete the files, just add the -delete parameter at the end of the command;

find . -name "*.log" -type f -delete

Good practices for creating Pull Requests

Tue, 03 Aug 2021 00:00:00 GMT

Before Git, the open source community used mailing lists for collaboration. Because of this, people had to write very long emails in order to justify their work. This caused a lot of friction because people had to spend a lot of time to write long emails and wait for a response from someone else.

With Git, your changes are local and you can push changes as soon as they are ready. This leads to more efficient and focused discussions.

Pull Requests are a relatively new tool that allow teams and developers to share, discuss, and merge code. They are a great tool for collaboration and communication. Pull Requests allow a developer to share their work with the team in progress, rather than showing up to a meeting with a completed feature or bug fix.

This way, the team can review, discuss, and comment on the proposed changes. Pull Requests also allow a developer to get feedback on their code from peers. This is one of the best indicators of the health of a team.

Pull Requests are a great way to introduce peer code review into the development process. The concept is simple: developers can request that a code change be pulled into the main branch of a project.

This differs from a patch which is a piece of code that is sent by email to a project maintainer, who then reviews the code and either applies the patch or not. The great thing about pull requests is that they make it easy to discuss code with others.

Pull requests are also a way to consolidate code reviews, discussions, and commits. They help ensure that the team's codebase is easy to work with, and they make it easier to keep track of who did what. By having everyone's code changes reviewed before they're merged into the project's codebase, you can ensure that the code that gets merged is clean and follows the project's guidelines. This helps the team avoid bugs, and it also helps avoid situations where one person's changes break another person's code.

One of the other advantages of creating Pull Requests is; it's helping drastically improve the quality of code and documentation. As the quality improves, the project team also gains more confidence and therefore has more time to solve more quality problems.

Pull requests really help to share knowledge and adjust peoples thinking. Many people know more parts of the code than before, and learn some skills from each other.

How to create a Pull Request

Creating a Pull Request is so trivial.

First we need a service that hosts git repositories, such as, Github, Azure DevOps, Bitbucket, Gitlab, etc.

git checkout -b <branch-name>

After checkout to the intended branch, developer will make the changes, and create a commit;

git add .

git commit -m "fix(shopping-cart): sum of the items prices in the shopping cart"

Now we need to push the changes to the remote repository;

git push -u origin <branch-name>

It's time to create the Pull Request, GitHub CLI has a command to do this, gh pr create

gh pr create --title "shopping cart grand total fix" --body "sum of the items prices in the shopping cart calculated correct again 🥳" --reviewer "polatengin"

Good practices for Pull Requests

There are some tips and good practices that we can use when creating Pull Requests, one of the most important parts of developers' lives.

Review your own Pull Request

It is important that you review your pull request before asking someone else to review it. That is because you may miss important things due to your familiarity with the code that you have written.

Also, check out the guidelines below to make sure that your code is ready for review, such as, tests, documentation and formatting.

Sometimes, people have trouble reviewing their own code. It's a totally normal thing! We all have blind spots. When people start contributing to open source projects, they often need someone else to review their work before they submit it. This is because they are not used to the style and often don't catch their own mistakes. Even experienced developers frequently need review!

Make sure the code builds and works

When you are going to contribute to an open source project, please do all testing on your local machine. Yes, it works on my machine is the worst excuse for not testing. It is unreliable and leads to frustration. If it doesn't work on your machine, it's unlikely to work on other machines as well.

If you don't have access to a machine that runs the software you're working with, it's your responsibility to find one — or build a virtual machine using a service like Vagrant. There's no excuse for not testing.

It is also important to test the code on different environments as much as possible before sending a Pull Request. An important thing to check is that the page is working and that user interaction is possible.

Make small Pull Requests (and often)

We value quickness and agility in our team. In order to achieve this, we need effective communication and reviews in order to be able to close Pull Requests quickly. We do not enjoy when people do not follow the team's process and raise comments in an asynchronous manner.

The goal of review should be to get code merged into the main branch as quickly as possible. As soon as the code is merged, it is out of the reviewers' hands. Reviewers should be focused on the code in front of them, not distracted by discussions about unrelated topics.

When we first started to use this way of working we saw that when we had a team member with a really big pull request it took a bit longer time to get the pull request reviewed and merged. After a while we learned that we should never create a pull request which contained more than one day of work from a team member. Instead we should split the story into smaller stories and do a pull request for each one. This also helps us to get the work done faster because we don't wait for all the feedback before we start working on the next story.

Smaller commits lead to more frequent deployments and a greater feeling of progress. It also makes bugs easier to track down, since there are fewer changes to the codebase.

The smaller the Pull Requests, the better.

Write a good description

The description written in the pull request is one of the most important and useful parts of reviewing it. If someone reviewing the code sees that hundreds of lines have been added, dozens of lines deleted, and multiple files have been renamed, they probably don't know the reason for these changes, so in order to help them review your code, please leave a good description to explain what you are doing.

Writing a complete description in the pull request will help provide context for the review.

Automate everything possible

Automation is a key part of good DevOps practice. Ideally, automation leads to an increase in quality and consistency. It also helps to reduce the amount of time it takes to deploy new code to production.

When you have automated steps in place, you can easily deploy your app and move on to the next task.

There are some steps that can easily automated;

Tests: Tests should run automatically whenever a Pull Request is opened or updated. The branch should not be merged unless the tests pass.
Documentation: Maintaining documentation is a very tedious process, so the more automated the process, the better. For example, if we are developing a documented API, we can automatically generate it as OpenAPI.

To generate documentation from code, we can use a code generator, such as Angular's ngDoc tool or Swagger's code generator. These tools can generate documentation based on comments in the source code. You can then use a tool such as Swagger UI to display the documentation.

Formatting the code: Checking the formatting of the code can also be automated and merging can be blocked if the format is not correct. Check the documentation format to avoid comments on the Pull Request, such as "this line is too long", "this method is too large", "this class is not well formatted", "variable name doesn't follow naming convention".

Create, Host and Share Docker Images with GitHub Packages

Tue, 30 Mar 2021 00:00:00 GMT

After the rise of cloud, and rise of microservices approaches, modular, plug-and-play and containerized applications are getting more attention.

So, literally hundreds of thousands software engineers are working on projects that built with multiple containerized pieces, all around the world.

If you're using GitHub and wanted a place to store and serve Docker Images for your project, you can easily go and use GitHub Packages, it's free ^(*) and super easy to use 🥳

You can have package registries for your;

Docker Images
npm packages
Maven packages
Gradle packages
NuGet packages
Ruby Gems

^(*): Actually, it's not free (🤦‍♂️) if you want to store more than 500MBs

Plan	Storage
Free	500 MB
Pro	2 GB
Team	2 GB
Enterprise	50 GB
Additional Storage	$0.25 per GB

In this post you can find a guideline to publish, serve and pull a Docker Image to GitHub Packages.

Creating the repo and Dockerfile

Let's create a repo on GitHub first;

After creating the repo, clone it and open it with your favorite editor (mine is Visual Studio Code 😎)

git clone https://github.com/polatengin/copenhagen.git

code copenhagen/

Now we need to create a Dockerfile and build the image we need in the project. (Example Dockerfile can be found on Copenhagen repo, Dockerfile)

FROM ubuntu:latest

LABEL org.opencontainers.image.source=https://github.com/polatengin/copenhagen

Logging in to GitHub Container Registry (a.k.a. GitHub Packages)

Navigate to Personal Access Tokens under GitHub Account Settings, and create a new one, just like below;

* Don't forget to set write:packages permission

It'll display the PAT, copy it to the clipboard.

Run the following command to login via docker cli;

echo ${GITHUB_PAT} | docker login ghcr.io -u ${GITHUB_USERNAME} --password-stdin

* Don't forget to set your username and PAT to the GITHUB_USERNAME and GITHUB_PAT variables

Building and Pushing Docker Image to GitHub Container Registry (a.k.a. GitHub Packages)

It's time to build, tag and push the Docker Image to the GitHub Container Registry (a.k.a. GitHub Packages);

docker build -t copenhagen .

docker tag copenhagen ghcr.io/polatengin/copenhagen:latest

docker push ghcr.io/polatengin/copenhagen:latest

After the Docker Image has been pushed, we can consume it in our projects;

We can see the packages in out GitHub Account, by navigating to the Packages tab (like, https://github.com/polatengin?tab=packages)

Using images from GitHub Container Registry (a.k.a. GitHub Packages)

After login to GitHub Container Registry, we can easily pull and consume images;

echo ${GITHUB_PAT} | docker login ghcr.io -u ${GITHUB_USERNAME} --password-stdin

docker run -d -it ghcr.io/polatengin/copenhagen:latest

* Don't forget to set your username and PAT to the GITHUB_USERNAME and GITHUB_PAT variables

So, it's possible (and relatively easy) to use GitHub Packages, within your CI/CD orchestration platform (such as GitHub Actions), or Kubernetes platform (such as, Azure Kubernetes Service (AKS))

References

How to troubleshoot AKS (Kubernetes) pods

Tue, 23 Mar 2021 00:00:00 GMT

Guidelines' are always help people to ensure if they made necessary checks before going deep dive.

If your team faces the same issue more than several times, it's good to keep it recorded. So, next time, your response time will be significantly shorter. That leads happy customers 🥳

In this post you can find a guideline to determine the underlying issue when there is an issue with a pod.

Check if cluster is healthy

First thing first, let's check if the nodes are healthy. Run the following command and wait to see if all the nodes are in Ready status;

kubectl get nodes

If some of the nodes are not in the Ready status, that means those nodes (or VMs if you will) are not healthy.

You can find the issue that causes the node to fail, by executing the following command;

kubectl describe node <NODE_NAME>

If nodes are Ready, check the logs, by executing the following commands;

# On Master node
cat /var/log/kube-apiserver.log # Display API Server logs
cat /var/log/kube-scheduler.log # Display Scheduler logs
cat /var/log/kube-controller-manager.log # Display Replication Manager logs

# On Worker nodes
cat /var/log/kubelet.log # Display Kubelet logs
cat /var/log/kube-proxy.log # Display KubeProxy logs

If nodes are healthy, continue with checking pods;

Check if pods are healthy

Let's list the pods;

kubectl get pods

If you're seeing some pods are not in Running state, that means, we need to focus on those pods.

Let's run the following command to see if there is a metadata issue;

kubectl describe pod <POD_NAME>

Check the Status, Reason and Message fields first.

In the below example, we can clearly see that nodes doesn't have enough memory to run the pod.

Eviction Reasons

MemoryPressure: Available memory on the node has satisfied an eviction threshold
DiskPressure: Available disk space and inodes on either the node's root filesystem or image filesystem has satisfied an eviction threshold
PIDPressure: Available processes identifiers on the (Linux) node has fallen below an eviction threshold

If there is no issue with the Status, Reason and Message fields, check the Image field.

Check Pod Image is correct

Somehow, your CI/CD Pipeline may not be able to push the new image to the Container Registry, but update the Kubernetes Pod Metadata, so, Kubernetes cannot fetch the new image and ...will fail.

If the image data is correct, check the integrity of Pod Metadata

Check Pod Metadata Integrity

Since the pod is not running properly, let's delete it safely and validate the Pod Metadata first, by executing the following command;

kubectl apply --validate -f deploy.yaml

If there is an issue with the metadata, --validate option detects the issue before applying it to the Kubernetes.

If everything up to this point is fine, that means, pod is running, it's time to check the logs

Check Logs of the Running Pod

Run the following command to check the logs of the running pod;

kubectl get pods

kubectl logs <POD_NAME>

If you don't spot any issue with the logs of the pod, connect to the pod and check the system in the pod;

Connect to a Shell in the Container

To get a shell to the running container, execute the running command;

kubectl exec -ti <POD_NAME> -- bash

If the running pod doesn't have bash, use sh instead of bash, use the following command;

kubectl exec -ti <POD_NAME> -- sh

References

Developing a simple Terratest Module for Azure Container Registry (ACR)

Tue, 16 Mar 2021 00:00:00 GMT

Terratest is a Go library that makes it easier to write automated tests for the infrastructure code. Terratest provides us a variety of helper functions and patterns to test our infrastructure easily.

In this post, we're gonna develop a simple Terratest Module for Azure Container Registry (ACR)

We're gonna follow the steps below;

Fork the official repo
Add the module
Run the CI workflow to test the module

Forking the official repo

Let's go to the official Terratest repo (https://github.com/gruntwork-io/terratest)

Fork it to your own GitHub account.

And clone it to your development environment.

If you're a Microsoft employee, let aztfmod team (or my manager Richard Guthrie, or my fellow team-mate Hattan Shobokshi or myself - Engin Polat) know, so you can directly use aztfmod/terratest repo.

We're gonna use GitHub Codespaces throughout this blog post.

Adding the module

First thing first, let's run the following command to download missing dependencies;

go mod tidy

Now we're ready to add the module, to do that, create a file in the modules/ folder;

touch modules/azure/acr.go

There is a subfolder for Azure related modules, modules/azure

Let's put the module into the azure package by adding the following line at the beginning of the file;

package azure

Start simple by creating a function that returns the *containerregistry.Registry structure, for given Azure Subscription Id, Resource Group Name and ACR (Azure Container Registry) Name;

func GetACRClient(t *testing.T, resourceName string, resGroupName string, subscriptionID string) *containerregistry.Registry {
}

This function will allow Terratest users to get the underlying Registry structure easily.

In the future we might need to add more functions, so let's create an internal facing function that has the same signature (accepts the same parameters and return the same type);

func getACRClientE(resourceName string, resGroupName string, subscriptionID string) (*containerregistry.Registry, error) {
  subscriptionID, err := getTargetAzureSubscription(subscriptionID)
  if err != nil {
    return nil, err
  }

  managedServicesClient := containerregistry.NewRegistriesClient(subscriptionID)
  authorizer, err := NewAuthorizer()

  if err != nil {
    return nil, err
  }

  managedServicesClient.Authorizer = *authorizer

  resource, err := managedServicesClient.Get(context.Background(), resGroupName, resourceName)
  if err != nil {
    return nil, err
  }

  return &resource, nil
}

In this function, first, we're checking if the provided subscription is valid;

subscriptionID, err := getTargetAzureSubscription(subscriptionID)
if err != nil {
  return nil, err
}

Then creating the Authorizer and setting it as the Authorizer of the underlying service client;

managedServicesClient := containerregistry.NewRegistriesClient(subscriptionID)
authorizer, err := NewAuthorizer()

if err != nil {
  return nil, err
}

managedServicesClient.Authorizer = *authorizer

If everything went well up until this point, we can get the Container Registry instance by calling the Get() method of the client;

resource, err := managedServicesClient.Get(context.Background(), resGroupName, resourceName)
if err != nil {
  return nil, err
}

return &resource, nil

As a go rule, if the function starts with lower-case character, it means the function is private and cannot be accessible from outside of the package, but if the function starts with upper-case character, it means the function is public and can be accessible from outside of the package.

This is the main difference between the following function signatures;

func GetACRClient(t *testing.T, resourceName string, resGroupName string, subscriptionID string) *containerregistry.Registry {
}

func getACRClientE(resourceName string, resGroupName string, subscriptionID string) (*containerregistry.Registry, error) {
}

Also, if function returns both the structure and an error, function name usually ends with capital E character, to make it easier to see the function may return an Error object;

func getACRClientE(resourceName string, resGroupName string, subscriptionID string) (*containerregistry.Registry, error) {
}

After having the private function, we can call it from the public function;

func GetACRClient(t *testing.T, resourceName string, resGroupName string, subscriptionID string) *containerregistry.Registry {
  resource, err := getACRClientE(resourceName, resGroupName, subscriptionID)

  require.NoError(t, err)

  return resource
}

Running the CI workflow to test the module

If you have access to an Azure Subscription (get a free Azure Subscription from: Create your Azure free account today) you can run the CI workflow on it. If you don't have access to an Azure Subscription, you can run the CI workflow on aztfmod/terratest repo.

Running CI workflow on your own Azure Subscription

Official Terratest repo and aztfmod/terratest fork has a GitHub Action in .github/workflows/ci.yml file.

If you fork one of those repos, .github/workflows/ci.yml file will be exist in your repo too.

CI workflow needs the following Secrets set up in the repo Settings;

AZURE_CREDENTIALS

To have permission to the Azure Subscription, to be able to provision "terraform script under test", you need to create a Service Principal by executing the following script in your Terminal;

You can generate the credential by executing following script;

az ad sp create-for-rbac --sdk-auth --name terratest_workflow

Output of this command is in json format, copy and paste it in a Secret, named, AZURE_CREDENTIALS

PAT

When the CI workflow finishes, it creates a comment in the PR, it's working against.

CI workflow uses PAT to create the comment.

To create the PAT, go to Personal Access Tokens page under Developer Settings menu.

Copy the generated PAT, and paste it in a Secret, named, PAT

Your for is ready to run the tests now, you can use the same guideline in the next section.

Running CI workflow on aztfmod/terratest repo

Since the CI workflow has workflow_dispatch trigger, you can go to GitHub Actions page and manually run the workflow.

Actors of the workflow

Source repo: official terratest repo (gruntwork-io/terratest)
Forked repo: (e.g., aztfmod/terratest, polatengin/terratest)
PR: created from forked repo, against source repo (gruntwork-io/terratest/pull/{PR NUMBER})

Flow of the workflow

Developer fork the source repo
Developer creates a branch in the forked repo
Developer does the development 😎
Developer creates a PR
Developer triggers the CI workflow manually with the following values;
- Repo: name of the forked repo (e.g. xyz/terratest)
- Branch: branch name on the forked repo (e.g. feature/adding-some-important-module)
- Target repository: home of the target_pr, which is the source repo (gruntwork-io/terratest)
- Target PR: pr number on the source repo (e.g. 14, 25, etc.)

References

How to remove file from git history

Tue, 09 Mar 2021 00:00:00 GMT

When we commit a file to a git system (such as, Azure DevOps or GitHub) it shows up in the folder structure of the repo.

And also, there is a record created in the underlying history structure.

We can delete the file that needs to be confidential, after we accidentally commit it.

But it'll stay in the commit history of the git system.

And anyone who can access the git system, can search through the commit history and find that file.

To delete a file from the git system, we can use the following command;

FILE_PATH_AND_NAME="src/backend/.env"

git rm -rf --cached --ignore-unmatch ${FILE_PATH_AND_NAME}

But, it'll not remove the file from the commit history. To remove it from the history as well, we can use the following command;

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch ${FILE_PATH_AND_NAME}" HEAD

This command will delete the file from the commit history and re-write the commit history without it.

It may take so long to complete, regarding the size of the commit history of the repo. To give a scale, it may need almost a full hour if you have 10K commits and wanted to remove a file from one of the first commits.

After the command finishes, we just need to commit the changed history, by executing the following command;

git push --force

References

Scenario : DevOps for Unity3D game projects

Tue, 02 Mar 2021 00:00:00 GMT

Game projects usually consist of a combination of more than one piece. Such as, game itself, a backend, a messaging platform, notification system, etc.

In this scenario, we'll create a game project with a .Net 5.0 backend and a Unity3D game project.

We'll create required infrastructure resources on one of the most advanced cloud platform; Microsoft Azure

And we'll store all the codes on GitHub and apply DevOps practices by creating couple of GitHub Actions in the project.

TL;DR, you can find the source code, GitHub Actions and Bash scripts in polatengin/sarajevo repo on GitHub

Let's start with creating the infrastructure on Microsoft Azure

Creating Infrastructure

First, to create infra.sh file in .iac folder, run the following command on terminal;

mkdir .iac && cd .iac

touch ./infra.sh

chmod +x ./infra.sh

chmod command will give executable capability to the infra.sh file, so we can run it by calling it (more details about chmod command can be found on chmod wikipedia page)

./infra.sh

First thing first, let's create couple of variables to hold common information for the infra.sh script;

PROJECT_NAME="sarajevo"
LOCATION="westeurope"

We can find the list of available locations by running the following command;

az account list-locations --output table

DisplayName               Name                 RegionalDisplayName
------------------------  -------------------  -------------------------------------
East US                   eastus               (US) East US
East US 2                 eastus2              (US) East US 2
South Central US          southcentralus       (US) South Central US
West US 2                 westus2              (US) West US 2
Australia East            australiaeast        (Asia Pacific) Australia East
Southeast Asia            southeastasia        (Asia Pacific) Southeast Asia
North Europe              northeurope          (Europe) North Europe
UK South                  uksouth              (Europe) UK South
West Europe               westeurope           (Europe) West Europe
Central US                centralus            (US) Central US
North Central US          northcentralus       (US) North Central US
West US                   westus               (US) West US
South Africa North        southafricanorth     (Africa) South Africa North
Central India             centralindia         (Asia Pacific) Central India
East Asia                 eastasia             (Asia Pacific) East Asia
Japan East                japaneast            (Asia Pacific) Japan East
Korea Central             koreacentral         (Asia Pacific) Korea Central
Canada Central            canadacentral        (Canada) Canada Central
France Central            francecentral        (Europe) France Central
Germany West Central      germanywestcentral   (Europe) Germany West Central
Norway East               norwayeast           (Europe) Norway East
Switzerland North         switzerlandnorth     (Europe) Switzerland North
UAE North                 uaenorth             (Middle East) UAE North
Brazil South              brazilsouth          (South America) Brazil South
Central US (Stage)        centralusstage       (US) Central US (Stage)
East US (Stage)           eastusstage          (US) East US (Stage)
East US 2 (Stage)         eastus2stage         (US) East US 2 (Stage)
North Central US (Stage)  northcentralusstage  (US) North Central US (Stage)
South Central US (Stage)  southcentralusstage  (US) South Central US (Stage)
West US (Stage)           westusstage          (US) West US (Stage)
West US 2 (Stage)         westus2stage         (US) West US 2 (Stage)
Asia                      asia                 Asia
Asia Pacific              asiapacific          Asia Pacific
Australia                 australia            Australia
Brazil                    brazil               Brazil
Canada                    canada               Canada
Europe                    europe               Europe
Global                    global               Global
India                     india                India
Japan                     japan                Japan
United Kingdom            uk                   United Kingdom
United States             unitedstates         United States
East Asia (Stage)         eastasiastage        (Asia Pacific) East Asia (Stage)
Southeast Asia (Stage)    southeastasiastage   (Asia Pacific) Southeast Asia (Stage)
Central US EUAP           centraluseuap        (US) Central US EUAP
East US 2 EUAP            eastus2euap          (US) East US 2 EUAP
West Central US           westcentralus        (US) West Central US
West US 3                 westus3              (US) West US 3
South Africa West         southafricawest      (Africa) South Africa West
Australia Central         australiacentral     (Asia Pacific) Australia Central
Australia Central 2       australiacentral2    (Asia Pacific) Australia Central 2
Australia Southeast       australiasoutheast   (Asia Pacific) Australia Southeast
Japan West                japanwest            (Asia Pacific) Japan West
Korea South               koreasouth           (Asia Pacific) Korea South
South India               southindia           (Asia Pacific) South India
West India                westindia            (Asia Pacific) West India
Canada East               canadaeast           (Canada) Canada East
France South              francesouth          (Europe) France South
Germany North             germanynorth         (Europe) Germany North
Norway West               norwaywest           (Europe) Norway West
Switzerland West          switzerlandwest      (Europe) Switzerland West
UK West                   ukwest               (Europe) UK West
UAE Central               uaecentral           (Middle East) UAE Central
Brazil Southeast          brazilsoutheast      (South America) Brazil Southeast

We need to create a Resource Group to put the project resources in it;

az group create --name "${PROJECT_NAME}-rg" --location "${LOCATION}"

A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group.

We can create an App Service Plan by running the following command;

az appservice plan create --resource-group "${PROJECT_NAME}-rg" --name "${PROJECT_NAME}-plan" --location "${LOCATION}" --sku "FREE"

An Azure App Service Plan provides the resources that an App Service App needs to run.

Only thing missing is the App Service resource itself, let's create it with the following command;

az webapp create --resource-group "${PROJECT_NAME}-rg" --plan "${PROJECT_NAME}-plan" --name "${PROJECT_NAME}-web"

Azure App Service is an HTTP-based service for hosting web applications, REST APIs, and mobile back ends. You can develop in your favorite language, be it .NET, .NET Core, Java, Ruby, Node.js, PHP, or Python. Applications run and scale with ease on both Windows and Linux-based environments.

Now, we have the infra that can run the backend project.

Creating Backend Project

To create the backend project, let's create a folder named backend, at the root of the solution;

mkdir backend && cd backend

We'll use dotnet cli to create a WebApi project;

dotnet new webapi

We created the backend project with the webapi template, alternatively we could use another template for the backend, such as, grpc, mvc, web, etc. More information about the templates can be found on dotnet new arguments page

Based on the genre of the game, we need to add code into the backend project, for the sake of simplicity of this post, we're gonna skip adding code into the backend project.

Creating Unity3D Game Project

To create the game project, let's create a folder named game, at the root of the solution;

mkdir game && cd game

We'll use Unity3D v2020.2.6.f1 to create the game project;

Based on the genre of the game, we can select one of the templates at the New Game Creation Dialog, at the time of writing this post, available options are;

2D
3D
High Definition RP
Universal Render Pipeline
Mobile 2D
Mobile 3D

After creating the game project, we need to actually develop the game, for the sake of simplicity of this post, we're gonna skip developing a game with Unity3D.

Creating GitHub Actions

We'll use GitHub Actions as the orchestrator of pipelines of this project.

Let's start with creating .github/workflows folder at the root of the solution;

mkdir .github/workflows && cd .github/workflows

Creating GitHub Action for .Net 5 Backend Project

Create a file named backend.yml in the .github/workflows folder

touch backend.yml

Open backend.yml file with the favorite code editor (in this post we'll use Visual Studio Code) and start with giving a name to our pipeline;

name: ci_backend

We need to define triggers for the pipeline, so pipeline can be start based on an event on GitHub;

on:
  workflow_dispatch:
  push:
    branches:
      - "main"
    paths:
      - "backend/**"

We're using 2 triggers;

workflow_dispatch, so we can start the pipeline on GitHub UI
push, so pipeline will automatically start when there is a commit to GitHub

But, since we have .iac, backend and game folders, and don2t want to build and deploy backend when there is a commit to game folder, we add couple of filters to the push trigger;

branches, so pipeline will start only if the main branch gets a commit
paths, so pipeline will start only if the listed folders have changed files

We can continue with adding jobs structure to the pipeline, so we can define what type of runner we'll use under-the-hood;

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:

GitHub provides cloud hosted runners for us, for free, to use them in our pipelines. There are a few flavors we can choose;

Windows: windows-latest, windows-2019
Ubuntu: ubuntu-latest, ubuntu-20.04, ubuntu-18.04, ubuntu-16.04
MacOS: macos-latest, macos-11.0, macos-10.15

Windows and Linux runners have the following hardware specs;

2-core CPU
7 GB of RAM memory
14 GB of SSD disk space

MacOS runners have the following hardware specs;

3-core CPU
14 GB of RAM memory
14 GB of SSD disk space

More details about the hosted runners can be found on GitHub-hosted runners page

First step will be checking out to the branch, so, the runner will have the latest code;

- uses: actions/checkout@v2

Then, we need to login to Azure CLI, so we can deploy the backend project code to Azure App Service resource (we provisioned the resource on Creating Infrastructure section)

- name: azure cli login
  uses: azure/login@v1
  with:
    creds: ${{ secrets.AZURE_CREDENTIALS }}

The runner, will need to login to Azure CLI when it's started, but we don't want to leak the credentials to our Azure Subscription

We need to create a Service Principal and store the Service Principal Object in a safe place that the runner can access.

To create the Service Principal, we'll use Azure CLI. Run the following command on your terminal;

az ad sp create-for-rbac --name "sarajevo-sp" --sdk-auth --role contributor

When the Service Principal Object created, it'll display on the terminal, like below;

{
  "clientId": "<GUID>",
  "clientSecret": "<GUID>",
  "subscriptionId": "<GUID>",
  "tenantId": "<GUID>",
  "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
  "resourceManagerEndpointUrl": "https://management.azure.com/",
  "activeDirectoryGraphResourceId": "https://graph.windows.net/",
  "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
  "galleryEndpointUrl": "https://gallery.azure.com/",
  "managementEndpointUrl": "https://management.core.windows.net/"
}

Copy the output and create a Secret on GitHub

Go to https://github.com/{ACCOUNT_NAME}/{REPO_NAME}/settings/secrets/actions
Add a new repository secret
Paste the output
Save

Please notice that the name of the secret should be the same with what we're using in the Azure CLI Login step;

- name: azure cli login
  uses: azure/login@v1
  with:
    creds: ${{ secrets.AZURE_CREDENTIALS }}

In the third step, we can set the .Net version to what we're using in the backend project;

- name: Setup DotNet Core
  uses: actions/setup-dotnet@v1
  with:
    dotnet-version: 5.0.x

In the fourth step, we can build and publish the backend project, so the upcoming steps can access the compiled version of it;

- name: build
  run: |
    cd backend

    dotnet restore
    dotnet build --no-restore
    dotnet publish --configuration release --output ./published

In the last step, we actually publish the backend project to the Azure App Service;

- name: Deploy to Azure
  uses: azure/webapps-deploy@v2
  with:
    app-name: "sarajevo-web"
    package: ./backend/published

Now, we're ready to start backend project pipeline, go to https://github.com/{ACCOUNT_NAME}/{REPO_NAME}/actions and click Run workflow button;

Creating GitHub Action for Unity3D Game Project

Create a file named game.yml in the .github/workflows folder

touch game.yml

Open game.yml file with the favorite code editor (in this post we'll use Visual Studio Code) and start with giving a name to our pipeline;

name: ci_game

We need to define triggers for the pipeline, so pipeline can be start based on an event on GitHub;

on:
  workflow_dispatch:
  push:
    branches:
      - "main"
    paths:
      - "game/**"

You can find more detailed explanation about the triggers at the Creating GitHub Action for .Net 5 Backend Project section of this post.

We can continue with defining the job;

jobs:
  build:
    runs-on: ubuntu-latest

This time, we'll do something different, before starting steps, we'll define strategy to add matrix to the game project pipeline;

strategy:
  fail-fast: false
  matrix:
    targetPlatform:
      - StandaloneOSX # Build a macOS standalone (Intel 64-bit).
      - StandaloneWindows # Build a Windows standalone.
      - StandaloneWindows64 # Build a Windows 64-bit standalone.
      - StandaloneLinux64 # Build a Linux 64-bit standalone.
      - iOS # Build an iOS player.
      - Android # Build an Android .apk standalone app.
      - WebGL # WebGL.

More detailed information about the strategy and matrix structures can be found on Workflow syntax for GitHub Actions page

Basically, we're telling GitHub Action to run the steps for the job parallely for each of the items in the targetPlatform variable in the matrix node.

So, instead of running steps once, GitHub Action will run the steps more than once (7 times for this example, because there are 7 items in the targetPlatform variable), but in parallel.

First step is the same first step we used in the backend pipeline;

- uses: actions/checkout@v2

In the second step, we'll build the Unity3D Game Project, for each of the items in the targetPlatform variable in the matrix;

- name: Unity - Builder
  uses: game-ci/unity-builder@v2.0-alpha-6
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  with:
    targetPlatform: ${{ matrix.targetPlatform }}
    projectPath: "game"

Building a Unity3D Game Project requires a Unity3D License, to get a license, we need to run the following GitHub Action steps one-time;

- id: getManualLicenseFile
  uses: game-ci/unity-request-activation-file@v2

- uses: actions/upload-artifact@v1
  with:
    name: ${{ steps.getManualLicenseFile.outputs.filePath }}
    path: ${{ steps.getManualLicenseFile.outputs.filePath }}

It's better to create a seperate GitHub Action for these steps;

name: onetime_to_get_unity3d_license
on: [ workflow_dispatch ]
jobs:
  license:
    runs-on: ubuntu-latest
    steps:
    - id: getManualLicenseFile
      uses: game-ci/unity-request-activation-file@v2
    - uses: actions/upload-artifact@v1
      with:
        name: ${{ steps.getManualLicenseFile.outputs.filePath }}
        path: ${{ steps.getManualLicenseFile.outputs.filePath }}

When we run this pipeline one-time, it'll create a file, named like, Unity_v2019.2.6f1.alf.zip, we need to download and extract the file to get the Unity_v2019.2.6f1.alf file.

After getting the Unity_v2019.2.6f1.alf file, we need to visit the Unity3D Manual License Activation Page and upload the file.

When the page process the file, there'll be a button to download the license file, named like Unity_v2019.x.ulf

Now we can open Unity_v2019.x.ulf file with the favorite code editor and copy its content, then we can create a new Secret on GitHub

Go to https://github.com/{ACCOUNT_NAME}/{REPO_NAME}/settings/secrets/actions
Add a new repository secret
Paste the output
Save

Please notice that the name of the secret should be the same with what we're using in the Unity - Builder step;

- name: Unity - Builder
  uses: game-ci/unity-builder@v2.0-alpha-6
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  with:
    targetPlatform: ${{ matrix.targetPlatform }}
    projectPath: "game"

Final step is for uploading compiled artifacts of the Unity3D Game Project;

- uses: actions/upload-artifact@v2
  with:
    path: build/${{ matrix.targetPlatform }}

Now, we're ready to start game project pipeline, go to https://github.com/{ACCOUNT_NAME}/{REPO_NAME}/actions and click Run workflow button;

At the end of the pipeline, we can download artifacts for all of the targetPlatforms, or we can add other steps into the pipeline to sign and publish the artifacts into stores, such as, Microsoft Store, Google Play Store, Apple App Store, etc.

Software Testing Types

Mon, 22 Feb 2021 00:00:00 GMT

Software Testing Types

There are different types of tests that we can use and implement in the CI/CD pipelines to make sure that changes to our code are working as expected.

Some of the most known and used test types;

Unit Tests

Unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use.

Integration Tests

Integration testing is the phase in software testing in which individual software modules are combined and tested as a group. Integration testing is conducted to evaluate the compliance of a system or component with specified functional requirements. It occurs after unit testing and before validation testing

Load Tests

Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program concurrently. As such, this testing is most relevant for multi-user systems; often one built using a client/server model, such as web servers. However, other types of software systems can also be load tested.

UI/UX Tests

UI/UX testing is the process of testing a product's graphical user interface (GUI) to ensure it meets its specifications. This is normally done through the use of a variety of test cases.

Penetration Tests

Penetration Tests (colloquially known as pentest or ethical hacking), is an authorized simulated cyberattack on a computer system, performed to evaluate the security of the system.

A/B Tests

A/B testing is a way to compare two versions of a single variable, typically by testing a subject's response to variant A against variant B, and determining which of the two variants is more effective.

References

Azure DevOps - State Transition Restriction Rules

Mon, 15 Feb 2021 00:00:00 GMT

What is State Transition Restriction Rule

Since the release of Sprint #172 (official documentation: Sprint #172 Updates) on Azure DevOps, it's possible to create rules to restrict state transition of Work Items on Azure DevOps Boards.

With this new feature, it's possible to restrict work items from being moved from one state to another directly. For example, you can restrict Bugs transitioning from New to Resolved directly, instead, they must go from New to Active, then they can go to Resolved.

Guideline

To create a State Transition Restriction Rule you need to create a new process or child process

In this guideline, you'll find how to create an Agile Child Process and how to add a State Transition Restriction Rule into it

First, go to Process list of the Azure DevOps Organization (https://dev.azure.com/{ORG_NAME}/_settings/process) and create a child process, for example, from Agile template...

Then, click one of the work item types;

Go to Rules tab and create a new Rule

You can create a new Rule, select "A work item state moved from ..." from the "Conditions" section, and, "Restriction the transition to state ..." from the "Actions" section, like this;

All of the possible "Conditions";

A work item is created ...
A work item state changes to ...
A work item state is not changed ...
A work item state changes from ...
A work item state is ...
A work item state is not ...
A work item state moved from ...
The value of ... (equals)
The value of ... (not equals)
A value is defined for ...
No value is defined for ...
A change was made to the value of ...
No change was made to the value of ...
Current user is member of group ...
Current user is not member of group ...

Azure CLI DevOps Extension

Mon, 08 Feb 2021 00:00:00 GMT

The Azure command-line interface (Azure CLI) is a set of commands used to create and manage Azure resources.

The Azure CLI is available across Azure services and is designed to get you working quickly with Azure, with an emphasis on automation.

Official documentation of the Azure CLI

With the Azure DevOps extension, we can reach most of the Azure DevOps services from the terminal, such as;

Capabilities are almost endless with the Azure DevOps extension 🎉

But, you need to install the extension first.

Fortunately, installing an extension to Azure CLI is trivial, you just need to add it to the extension list of Azure CLI and configure it;

Usage is simple;

$ az devops --help

Group
    az devops : Manage Azure DevOps organization level operations.
        Related Groups
        az pipelines: Manage Azure Pipelines
        az boards: Manage Azure Boards
        az repos: Manage Azure Repos
        az artifacts: Manage Azure Artifacts.

Subgroups:
    admin            : Manage administration operations.
    extension        : Manage extensions.
    project          : Manage team projects.
    security         : Manage security related operations.
    service-endpoint : Manage service endpoints/service connections.
    team             : Manage teams.
    user             : Manage users.
    wiki             : Manage wikis.

Commands:
    configure        : Configure the Azure DevOps CLI or view your configuration.
    feedback         : Displays information on how to provide feedback to the Azure DevOps CLI team.
    invoke           : This command will invoke request for any DevOps area and resource. Please use
                       only json output as the response of this command is not fixed. Helpful docs -
                       https://docs.microsoft.com/rest/api/azure/devops/.
    login            : Set the credential (PAT) to use for a particular organization.
    logout           : Clear the credential for all or a particular organization.

If you want to take a look at (and hopefully contribute to) Azure CLI DevOps Extension code, it's open-sourced on GitHub with ❤ Azure DevOps Extension for Azure CLI

What is DevOps!

Mon, 01 Feb 2021 00:00:00 GMT

Donovan Brown himself described What is DevOps? as the following;

DevOps is the union of people, process, and products to enable continuous delivery of value to our end users.

I am very deliberate in the terms used in this definition. I choose value over software. DevOps is not just automating a pipeline so we can quickly deliver software. Our goal is to deliver value. The term end users was also very carefully chosen. The value we produce must reach our end users. If the value only reaches the Dev and QA environments but is held up before reaching production where it can be realized by our end users, we are still failing.

It is very important to realize that DevOps is not a product. You cannot buy DevOps and install it. DevOps is not just automation or infrastructure as code. DevOps is people following a process enabled by products to deliver value to our end users.

You can find the official DevOps page of Microsoft in What is DevOps? page.