Custom Ansible Filter Plugin

‼️ Hello readers ‼️

In this article, I and Uditanshu pandey have developed a custom filter plugin using ansible through which we can List all the Top Level Base Folders in the S3 bucket on the top of the AWS cloud.

For example, let’s suppose -

With this picture, it will be easy to understand. We have one bucket named ansiblebucket1 inside this bucket we have multiple folders Folder1, Folder2, Folder3, etc.

Inside these folders, we have uploaded some files like this.

So, when we use the aws_s3 module in ansible and perform the list operation then ansible will list all the folders, sub-folders, and files.

If we talk about the above example the output of list operation will be:-

Folder1/Hello.txt

Folder2/Hello2.txt

Folder3

But what if we want only Top-Level folders. Ansible will not be able to show us only the Top-Level folders.

We want the output of list operation like this but we can’t get this because there is no method present in ansible to perform this.

Folder1

Folder2

Folder3

For this, we will be developing a CUSTOM FILTER PLUGIN.

So, now let’s start with this project

What are Plugins?

Plugins are pieces of code that augment Ansible’s core functionality. Ansible uses a plugin architecture to enable a rich, flexible, and expandable feature set.

There are two types of plugins in ansible:-

• Pre-Created Plugins.
• Customized Plugins.

Ansible has multiple pre-created plugins available like CallBack Plugins, Action plugins, Cache plugins, etc.

We will be developing the Filter plugin.

Filter plugins manipulate data. They are a feature of Jinja2 and are also available in Jinja2 templates used by the template module. As with all plugins, they can be easily extended, but instead of having a file for each one, you can have several per file.

How to create Custom Plugins?

• In the home directory create a directory named as filter_plugins (All the custom plugins should be here only otherwise ansible won’t read those plugins).
• Now, here we will be writing the code for our plugin in the python file.
• At last, we have to define the variable filter_plugins with the path where our filer_plugins folder is located in the ansible configuration file.

In our case it is filter_plugins=/root/filter_plugins.

And it’s done. This is the basic setup we have to do. before starting our project.

Now, we will write code for our plugin.

#!/usr/bin/python3class FilterModule(object):
       def filters(self):
            return {'my_s3_filter': self.top_folders}

This block of code will remain the same for all the plugins. In this block of code, we have defined a python interpreter that is #!/usr/bin/python3 and defined one class named as FilterModule inside this class one function iscreated named as filters.

Now, we will create our Function where we will write the code.

We have named this function as top_folders and passed the s3_keys to it.

Now, we will create the empty set named as a base.

After this, we will be implementing a for loop. All the folders in the S3 bucket will be stored in the variable key one by one. After this, we will put one condition that if “/” is present in the Key then we will split that key by using split () , and the value at the first index will be stored in the variable named as base_folders.

After this, we will add this base_folders variable to the base that is the empty set.

Sets are not handled by ansible therefore at last we have converted them into the list.

Also, we can use the list directly but if we use the list directly we have to write some more code for removing duplicates in the list because List allows duplicate values in it.

SET DO NOT ALLOW DUPLICATE VALUES.

That is why we have used the set and then converted it into a list.

def top_folders(self, s3_keys):
    base= set()
    for key in s3_keys:
        if "/" in key:
             base_folders = key.split("/")[0]
             base.add(base_folders)
    return list(base)

Full code of plugin.

Now, we will create an S3 bucket, Create folders inside the bucket on the top of the AWS cloud.

Before writing this playbook we will create an Ansible Vault to store our AWS credentials.

To create ansible vault.

ansible-vault create --vault-id Pro@prompt <Filename.yml>

This command will create an ansible vault with the given file name.

Inside the vault, we will write our access_key and secret_key.

Now, we will write the main playbook.

For creating the S3 bucket.

- hosts: all
  vars_files:
        - Secretfile.yml
  tasks: 
    - name: "S3_Bucket"
      amazon.aws.s3_bucket:
          aws_access_key: "{{ access_key }}"
          aws_secret_key: "{{ secret_key }}" 
          name: ansibleproject 
          state: present
          versioning: yes

This block of code will create an S3 named ansibleproject and we have enabled versioning.

Now, we will write the code that will create the Folders inside the bucket.

- name: Folder Creation
  aws_s3:
         aws_access_key: "{{ access_key }}"
         aws_secret_key: "{{ secret_key }}"
         bucket: ansibleproject
         object: Folder_1/Folder_2/Folder_3
         mode: create

This block of code will create the Folder_1 inside the bucket ansibleproject.

Also, Folder_2 and Folder_3 will be created inside Folder_1.

Now, we will write the code to perform the List Operation.

- name: Listing
  aws_s3:
         aws_access_key: "{{ access_key }}"
         aws_secret_key: "{{ secret_key }}" 
         bucket: ansibleproject 
         mode: list
  register: bucket_content

Here ansible will perform the List operation and we have stored the output in the variable named as bucket_content.

We will now print all the Folders present in the bucket.

- name: Bucket paths
  debug:
       var: bucket_content.s3_keys

This block of code will print the Folders. In our case output will be:

“Folder_1/Folder_2/Folder_3/”

Now, we will use our Custom filter plugin so that we can list only the Top-level folders.

- name: Only top level folders
  set_fact:
    deployed_packages: "{{ bucket_content.s3_keys | my_s3_filter }}"

Using set_fact, we can store the value after preparing it on the fly using certain tasks like using filters, etc.

In this block of code, all the bucket content we are passing to our custom filter plugin that is named my_s3_filter.

At last, we will print the variable deployed_packages.

- name: Top level folders
  debug:
     var: deployed_packages

Full Playbook

We only have to run the playbook.

To run the playbook:-

ansible-playbook --vault-id Pro@prompt <File_Name.yml>

This command will ask for the password that we have given at the time of vault creation.

Now, let’s check whether this playbook will work or not.

By running this playbook bucket and Folders were created.

Bucket is created named as ansibleproject

Folder_1 is created inside the bucket.

Similarly, Folder_2 is created inside the Folder_1 in the bucke

This is the List operation performed by ansible. In this, all the Folders and Subfolders were listed.

This is the output of our filter plugin. Here we can see that Only Top-Level Folders are listing.

This means our playbook has been successfully worked.

Hope this article will be helpful.

Thanks for reading this article.