Preheat oven to 325°F (165°C). Combine almond flour, softened butter, brown sugar substitute, egg, and vanilla into a dough. Roll thin and cut into small rounds. Bake 10–12 minutes, until golden. Cool completely.
In a saucepan, warm cream and almond milk over medium heat until just steaming.
Whisk egg yolks with brown sugar substitute in a bowl, then slowly whisk in the warm cream mixture.
Return to the saucepan and cook over low heat, stirring constantly, until thickened, about 5–6 minutes. Do not boil.
Remove from heat and stir in vanilla and banana extract. Cool slightly.
In a trifle dish or individual cups, layer wafers, custard, and banana slices, repeating until all ingredients are used.
Chill at least 4 hours, or overnight, before serving.