Motivation⌗

Recently I wrote two Matrix (as in the communication protocol) bots using maubot externer Link.

  • The MensaBot can tell you today’s menu in the canteen. It also supports you in finding a time when you can go to the canteen together with you friends or co-workers.
  • The ReminerBot is target towards students and can remind you of several university related deadlines. It can remind you of the deadlines to register for or de-register from your exams, pay your semester fee or to register for the universities recreational sports courses. You’ll never miss these deadlines again!

During the years in which I really learned to program I was exposed a lot to the Django web framework externer Link. I also worked a lot with it during these years. Now, 5 years later, it is still the framework that I have the most experience with.

One thing that Django does very well is ORM (Object-relational mapping; mapping Objects in your programming language to relational databases). In Django this is so good that I never thought much about this aspect - it just worked. maubot on the other hand is lacking in this aspect. The old and deprecated SQLAlchemy (version 1.4) based database supported ORM. The new database is a thin wrapper layer around asyncpg/aiosqlite. But it is very minimal and does not support any ORM. Looking at some other bots written using maubot none of the contained some ORM. All of them were managing their tables manually and had custom boiler plate code for querying and modification - with handwritten SQL queries and all.

I would only willing to put up with this for the most basic bots. ReminderBot would be over the limit for me from the very beginning and MensaBot would probably get there with some more advanced features. This meant that I had to search for an ORM library that could fill this hole. After some research I landed on Tortoise ORM externer Link and decided to give it a try.

Using Tortoise ORM in a maubot plugin⌗

With its first commit in March 2018 Tortoise ORM externer Link is a somewhat new ORM library for Python. There are two main points that convinced me to give it a try.

  • Its design heavily inspired by Django’s ORM and the API is very similar. I expected this to result in a quick learning curve for me.
  • Tortoise ORM is a async library and uses asyncpg and sqlite as the backends for PostgreSQL and SQLite. maubot is also using async and mautrix (the matrix library behind maubot) also uses the same database drivers. This should result in good compatibility. Using Tortoise ORM with maubot will require some tinkering but not excessive amounts.

We will first need to define our models and then initialise our database connection. After that we can use the Models. Finally we should close all the lingering connections.

Define the models with module path mybot.models like so:

from tortoise.models import Model
from tortoise import fields, Tortoise


class Subscription(Model):
    room_id = fields.CharField(max_length=255)
    list: fields.ForeignKeyRelation[EventList] = fields.ForeignKeyField("myapp.EventList", related_name="subscriptions")


class Event(Model):
    id = fields.CharField(max_length=255, primary_key=True)
    name = fields.CharField(max_length=255)
    timestamp = fields.DatetimeField()
    list: fields.ForeignKeyRelation[EventList] = fields.ForeignKeyField("myapp.EventList", related_name="events")
    sent = fields.BooleanField(default=False)


class EventList(Model):
    id = fields.CharField(max_length=255, primary_key=True)
    name = fields.CharField(max_length=255)
    description = fields.TextField()

The myapp in myapp.EventList of the ForeignKeyField comes from the Tortoise module definition in the call to Tortoise.init below.

Initialise the database like so:

class MyBot(Plugin):
	async def start(self):
		await Tortoise.init(db_url=str(self.database.url), modules={'myapp': ['mybot.models']})
		await Tortoise.generate_schemas(safe=True)

Use the ORM like so:

@command.new(name="subscriptions", help="List the lists you're subscribed to.")
async def subscriptions(self, evt: MessageEvent) -> None:
	subscriptions = await Subscription.filter(room_id=evt.room_id).values("list__name")
	self.log.debug(f"Subscriptions: {subscriptions}")

Reference the documentation externer Link for further usage information.

On shutdown close any open connection with:

class MyBot(Plugin):
	async def stop(self):
		await Tortoise.close_connections()

Before we can deploy our maubot plugin we have to one final thing: install the tortoise-orm python package. maubot plugins can specify dependencies but these currently don’t do anything at all. So we have to do that ourselves. If you run maubot natively just install tortoise-orm[asyncpg,aiosqlite] with pip. If you run the maubot docker mage you can build a custom image like this:

FROM dock.mau.dev/maubot/maubot
RUN pip install tortoise-orm[asyncpg,aiosqlite]

Then reference the above Dockerfile in your docker-compose.yaml

services:
	maubot:
		build:
			dockerfile: maubot.Dockerfile
			context: .
		volumes:
			- ./volumes/maubot:/data:z

Closing thoughts on Tortoise ORM⌗

All in all I am quite happy with Tortoise ORM. I felt at home very quickly thanks to its similarity to Django’s ORM. It works reasonably well for my needs but it is sometimes a bit rough around the edges. The documentation could be improved with a couple more code example in my opinion. But what I am missing the most is comprehensive support for Migrations though that is on the projects roadmap. The current solution Aerich externer Link requires a database connection to generate the migrations. The migration is then tied to that database. This eliminates Aerich because I don’t have access to the database during development.