Understanding Ansible HttpApi plugins

TL;DR: I wouldn't recommend this method. It's not actually that good.

It's common to need to interact with HTTP APIs in an Ansible playbook. The standard method involves using the uri module or a command with curl. These can be clunky, especially when common connection parameters have to be copypasted between tasks. These also mix the concerns of the connection from those of the actual request. For example, the cert for mTLS has nothing to do with the body being submitted to the API.

Looking through the list of plugin types in Ansible, there's something called an httpapi plugin. The documentation is a bit light, but it's described as:

HttpApi plugins tell Ansible how to interact with a remote device’s HTTP-based API and execute tasks on the device.

This isn't exactly what we're after. It's really set up for devices like network switches where SSH isn't really the intended connection method.

Adapting the HttpApi plugin

But being able to submit HTTP requests is about 100% of what we need, so we might be able to just use it with a little adapting. There are several steps involved in creating an HttpApi plugin:

Create HttpApi plugin
Create Module for invoking the plugin

And for using it, it's fairly simple:

Supply connection information
Delegate tasks to that connection

Understanding the HttpApi connection structure:

The flow for how HttpApi plugins work is a bit involved with a few unexpected outcomes. The flow is as follows:

an Ansible task requests the httpapi connection type, possibly with the ansible_connection parameter
the connection type has parameters which specify the ansible_network_os. This is used to look up which httpapi plugin to use
if the connection has not already been established, ansible uses the ansible-connection command to create it in a separate process. Getting messages back from the connection is weird, as they are only flushed at the start of every task, after the connection has been established. Using queue_message during a module will cause those messages to be displayed at the start of the next module invocation which uses that connection.
A Module will be invoked and can instantiate its connection to the httpapi plugin. This is basically just a wrapper around the socket path to the connection created above.
The connection will allow the Module to send requests through the httpapi plugin
The httpapi plugin send the requests through to its connection, which is the ansible.netcommon.httpapi connection plugin. This connection plugin has a method for enriching an HTTP request
the connection plugin shells out to the util ansible.module_utils.urls.open_url, which is a generic way of making web requests, and eventually shells out to urllib.request.urlopen

Implementing the HttpApi stack

So that's a bit complicated, but it's actually pretty easy to get a barebones version of this:

HttpApi plugin

.../plugins/httpapi/http.py

DOCUMENTATION = """
---
author: lilatomic
httpapi: http
short_description: generic http connection
description: 
  - This HttpApi plugin provides a generic HTTP client. This makes it quicker to interact with HTTP APIs than using the URI module
"""

import json

from ansible.plugins.httpapi import HttpApiBase

EMPTY_DATA = object()


class HttpApi(HttpApiBase):
	def send_request(self, path, data=EMPTY_DATA, method="GET", **message_kwargs):
		# set headers for body
		headers = {'Accept-Encoding': 'application/json'}
		if not data or data == EMPTY_DATA:
			data = {}
		else:
			headers['Content-Type'] = 'application/json'

		# actually send the connection
		r, r_data = self.connection.send(path=path, data=data, method=method, headers=headers)

		# data is returned as bytes, so we have to pull it out
		r_data_serialisable = json.loads(r_data.read().decode("utf-8"))

		return {
			"response": {
				"data": r_data_serialisable,
				"msg": r.msg,
				"code": r.code,
				},
			"request": {
				"url": r.url
				}
			}

Wow there's actually a lot going on here just to submit requests, and we haven't even gotten into authentication.
This barebones version assumes that everything will be JSON in and JSON out, so we can just set the Content-Type header to that.

Sending the data is done with a call to self.connection.send. As mentioned before, this resolves to the ansible.netcommon.httpapi connection plugin and will eventually make their way to urllib, but not before 2 more layers of munging.

We then have to unload the body ourselves.

For the return, note that we're rebuilding most of the response ourselves. This is because the response will need to be serialised over JSON for interprocess communication back to the main Ansible process. Remember when ansible-connection spun this connection off as a separate process? Well there's also a bespoke JSON-rpc remote-method-invocation framework which fails to serialise most of the things returned by the ansible.netcommon.httpapi. One of the things which fails to return is error messages, so I guess just hope you don't get those...

Module

.../plugins/modules/http.py

#!/usr/bin/env python


__metaclass__ = type

from ansible.module_utils.basic import AnsibleModule
from ansible.module_utils.connection import Connection
from ansible.utils.display import Display

display = Display()


def main():
	argument_spec = {
		"path": {"type": "str", "default": None},
		"method": {"type": "str", "default": "GET"},
		"data": {"type": "raw", "default": None},
		}

	result = {}

	module = AnsibleModule(argument_spec=argument_spec)
	try:
		# get reference to httpapi process through socket path for RMI
		connection = Connection(module._socket_path)

		r = connection.send_request(
			path=module.params.get("path"),
			method=module.params.get("method"),
			data=module.params.get("data"),
			)
		result.update(r)
		module.exit_json(**result)
	except Exception as e:
		module.fail_json(msg=f'{e}', **result)


if __name__ == '__main__':
	main()

We start with the argument spec. This is sorta nice for automatically performing parameter validation for us. For example, if we mistyped the one of the parameter names, we'd get a warning that that parameter wasn't allowed. That's checked during the AnsibleModule instantiation.

We then get a reference to the httpapi connection, which we defined just above. We use the connection to send the request, which requires us to forward all the parameters. It's a bit cumbersome.

We then need to use the AnsibleModule.exit_json function. This method is the way to return from an Ansible module. These are executed in separate processes and their return values are gathered from stdout. It's certainly a way to do that. In the event of an error, we use the AnsibleModule.fail_json for the same type of thing, but for failure.

Don't forget the shebang and the thing that launches the function is it's invoked.

Supply connection information

One of the sneaky ways of adding these connections is to include them as a Host in the inventory file:

all:
  hosts:
    httpbin:
      ansible_connection: "httpapi"
      ansible_network_os: "lilatomic.api.http"

      ansible_host: "httpbin.org"
      ansible_port: "443"
      ansible_httpapi_use_ssl: true

ansible_connection this must be set to httpapi, and tells Ansible to select the httpapi connection plugin.

ansible_network_os this must match the name of your httpapi plugin.

ansible_host the host subcomponent of the url

ansible_port the port

ansible_httpapi_use_ssl whether to use http or https

I haven't found a way to combine the last 3 parameters. I also haven't found a way to add a start of the URL path component (for example, tomcat-hosted applications are by default hosted at /).

Delegate tasks to the connection

This is the easiest way of sending the requests to this connection, in my opinion. It's literally just the delegate_to line:

- name: test
  hosts: localhost
  gather_facts: false

  tasks:
    - name: test get
      lilatomic.api.http:
        path: "/get"
        method: "GET"
      delegate_to: "httpbin"
    - name: test post
      lilatomic.api.http:
        path: "/post"
        method: "POST"
        data: "HIHELLO"
      delegate_to: "httpbin"

Other features of HttpApi Plugins

The HttpApiBase base class has several other methods:

login
logout
update_auth
handle_httperror

The only method which isn't a virtual method is the handle_httperror. Its behaviour is to try to authenticate if a 401 is returned.

These methods are also invoked appropriately during the ansible.netcommon.httpapi sessions.

Tradeoffs

So, there's a lot of implementation stuff. And there's a lot of boilerplate. And it's not even correct.
The interface for libraries like requests or urllib3 are so much better than this, and don't require any of the machinery to get correct.

There are many other downsides to this approach. The requests happen either in another process or in a module, which is another process. This makes it almost impossible to get trace logging or unexpected errors.
You might also write modules which wrap more complicated operations, like creating a user and granting them permissions. This either needs to happen in a module, so your tracebacks are terrible; or in an action plugin invoking the module multiple times, so now we need to either copypaste the invocations or make another layer as a helper method to invoke the method.
And there are at least 4 layers of munging, so if you're trying to figure out why a GET request is being given "Content-Type": "application/x-www-form-urlencoded" you might be in for a bad time.

So all-in-all, I wouldn't recommend using this method.