What Happens After You Start Your Mojolicious ‘app’

Please note, THIS IS A WORK IN PROGRESS, not a final article.

First let’s look at the skeleton version of the main script that gets created when you execute “mojo generate app Foo” − this is foo/script/foo −

#!/usr/bin/env perl

use strict;
use warnings;

use lib 'lib';

# Start command line interface for application
require Mojolicious::Commands;
Mojolicious::Commands->start_app('Foo');

what  Mojolicious::Commands does:

  • enables all the processing for Mojolicious commands like daemon, get, generate, routes, test, etc.

Then we do Mojolicious::Commands->start_app(‘Foo’) which:

  • handles all the switches and arguments in the invocation command-line
  • Loads the application class (here, lib/Foo.pm)
  • Calls the application

The source for start_app (in Mojolicious/Commands.pm) looks like this:

sub start_app {
  shift;
  Mojo::Server->new
    ->build_app(shift)
      ->start(@_)
}

This calls the new() method of Mojo::Server to create a Mojo server object (and subscribe to the “request” event with default request handling); the server object  is returned and upon that we call build_app().  The argument to build_app will in this example be “Foo” which is the name of the application class for which we are building an instance.

→ Q: What does it mean, subscribe to the request event?

Then the call to start() runs as shown in Mojolicious.pm, namely by invoking the commands() and run() methods of Mojolicious, which have the effect of processing any commands and then running the event loop.

The only other code, aside from a stock Controller, is in lib/Foo.pm:

package Foo;
use Mojo::Base 'Mojolicious';

# This method will run once at server start
sub startup {
  my $self = shift;

  # Documentation browser under "/perldoc"
  $self->plugin('PODRenderer');

  # Router
  my $r = $self->routes;

  # Normal route to controller
  $r->get('/')->to('example#welcome');
}

What’s going on here?

 

the argument is the ‘app’ (instance of ___ class) that ___ creates

From this we can call built-ins like plugin() and routes()

 

 

−−

https://en.wikipedia.org/wiki/Event-driven_architecture

from http://mojolicio.us/perldoc/Mojolicious/Guides/FAQ#What-is-an-event-loop

An event loop is basically a loop that continually tests for external events and executes the appropriate callbacks to handle them, it is often the main loop in a program. Non-blocking tests for readability/writability of file descriptors and timers are commonly used events for highly scalable network servers, because they allow a single process to handle thousands of client connections concurrently.

  1. Set up all the routes, timers, and other reactions to be processed by the event engine.
  2. The script calls app->start (for Lite) or −−− (for Mojolicious)
  3. Event loop runs − Mojo::IOLoop (detail?)

from http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Timers

Timers, another primary feature of the event loop, are created with “timer” in Mojo::IOLoop and can for example be used to delay rendering of a response, and unlike sleep, won’t block any other requests that might be processed concurrently… Recurring timers created with “recurring” in Mojo::IOLoop are slightly more powerful, but need to be stopped manually, or they would just keep getting emitted.

from http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Exceptions-in-events

Since timers and other non-blocking operations are running solely in the event loop, outside of the application, exceptions that get thrown in callbacks can’t get caught and handled automatically. But you can handle them manually by subscribing to the event “error” in Mojo::Reactor or catching them inside the callback.

 

Why I’m Excited about Mojo::Pg

Marcus Ramberg wrote in the December 2014 Perl Advent Calendar about Mojo::Pg, the new DBI wrapper for the Mojolicious framework.

Particularly of interest are

  • Automatic transaction rollback if the ‘guard variable’ goes out of scope.   In practice this means if you just return or throw an exception in the middle of a routine, any transactions get rolled back without having to do anything yourself.
  • Migrations. Written in plain SQL and easily embedded or attached to your program, this was one of the few things I liked about Rails, but it’s here now.
  • Asynchronous triggers.  You could have two processes attached to the same database; when one does an INSERT, the other process gets a notification. This opens up all kinds of things that were nearly impossible before.

On top of the upcoming “INSERT … ON CONFLICT …” grammar in Postgres 9.5, which lets you do things like:

INSERT INTO distributors (did, dname)
  VALUES (5, 'Gizmo transglobal'), (6, 'Associated Computing, inc')
  ON CONFLICT (did) DO UPDATE SET dname = EXCLUDED.dname;

it’s well worth considering whether to keep using MySQL at all.

Building more Simutrans tools: the Perl Imager package

In building some new tools to support Simutrans development, I found this bit of sage advice dated 2005 from Tony Cook, the developer of the Imager module. From that I revised this example to translate the “transparent” cyan that Simutrans wants in its pakfile images, to actual transparency.

Note the use of new function signatures, and hash slices.

use v5.20;
use feature 'signatures';
no warnings 'experimental::signatures';

use Imager;

sub replace_color ($img, $from_color, $to_color) {
    # Inspired by http://www.perlmonks.org/?node_id=497355
    my $rpnexpr = <<'EOS';
x y getp1 !pix
@pix red from_red eq
@pix green from_green eq
@pix blue from_blue eq
and and
@pix alpha from_alpha eq
and
to_red to_green to_blue to_alpha rgba @pix ifp
EOS

    my %constants;
    # Load values via hash slices
    @constants{map {"from_$_"} qw{red green blue alpha}} = 
        $from_color->rgba;
    @constants{map {"to_$_"  } qw{red green blue alpha}} =
        $to_color  ->rgba;
    return Imager::transform2({ rpnexpr => $rpnexpr,
                                constants => \%constants,
                                channels => 4},
                              $img);
}

my %special_colors = (
    'transparent_cyan' => [231,255,255],
    );

my $image = Imager->new(file => $filename);
$image = $image->convert(preset => 'addalpha');
#
# Replace visible color with transparency
#
my $color = Imager::Color->new( @{$special_color{transparent_cyan}} );

# make a work image the same size as our input; add an alpha channel
my $work = Imager->new(xsize => $image->getwidth,
                       ysize => $image->getheight, channels => 4);
# fill it with the color we want transparent
$work->box(filled => 1, color => $color);

# get an image with that color replaced with transparent black
my $out = $work->difference(other => $image);

$out->write(file => 'example-output.png');

Further examples of the little Transform script language could change from more-standard “real” transparent PNGs to what Simutrans wants, or help highlight unwanted special colors that can result from anti-aliasing, or do all kinds of color or shape transforms.

Simutrans: De-mystifying the PAK format

Let’s de-mystify the .pak file format.  Paks are actuallyfairly simple data files, although the details can certainly be complex.

In besch/reader/obj_reader.cc, obj_reader_t::read_file() opens a file and calls read_nodes() in that same .cc to reach each node.  The file begins with the version information, terminated with a Ctrl+Z (0x1A) byte:

53 69 6d 75 74 72 61 6e  73 20 6f 62 6a 65 63 74  |Simutrans object|
20 66 69 6c 65 0a 43 6f  6d 70 69 6c 65 64 20 77  | file.Compiled w|
69 74 68 20 53 69 6d 4f  62 6a 65 63 74 73 20 30  |ith SimObjects 0|
2e 31 2e 33 65 78 70 0a  1a eb 03 00 00 52 4f 4f  |.1.3exp......ROO|
54 01 00 00 00 42 55 49  4c 26 00 25 00 08 80 03  |T....BUIL&.%....|

Following that are four bytes of Pak-File version (eb 03 00 00 above), and then a series of nodes until the end of file.  Each node is processed by its appropriate reader found in the besch/reader/ subdirectory. In the file, each node begins with four characters describing the node type, as defined in besch/objversion.h:

enum obj_type
{
        obj_bridge      = C4ID('B','R','D','G'),
        obj_building    = C4ID('B','U','I','L'),

and then a 2-byte (16-bit) child count and 2-byte (16-bit) data block size.  If the data block is more than 64k bytes, 0xFFFF is used for the data size, followed by a four-byte (32-bit) data block size.  Then the actual data block bytes, followed by any additional nodes in this same format.

The child count indicates how many of the following nodes are considered to belong to (be “inside”) the current node.  The BUIL node in the example has 0x0026 child nodes.  This is how, for example, a single pak file can contain multiple objects, with each object containing several child nodes.

Note that read_nodes() chooses the internal class type from the four-character name, using the following line of code:

        obj_reader_t *reader = obj_reader->get(static_cast<obj_type>(node.type));

How exactly that works, in converting a text representation to a somewhat conceptual C++ class type, is left to the student as an exercise.

Toadfarm on Base Debian: An absolutely minimal multi-tenant webserver

Here we create an absolutely minimal multi-tenant webserver, without even installing Apache or NginX. Although we can add either of those later, with our content running under them as reverse proxies, this tutorial lets you run one or more Mojolicious “apps” mounted under a single toadfarm startup script, saving you the “default perl memory” times the number of worker processes and apps.

We will b using Jan Henning Thorsen’s Toadfarm:

Toadfarm is a module for configuring and starting your Mojolicious applications. You can either combine multiple applications in one script, or just use it as a init script.


First, perform a Debian Netinstall on a virtual machine or virtual host. The only modification to all the default selections is to enable only “Standard System Utilities” and disable desktop, database, mail, print, file, and web server. We will be building our own of all those!

Now boot up into that absolutely stock, base machine. For illustration, we will assume your username is gronk − change this as necessary. First let’s install a few system packages (ssh for access, curl for installing Perlbrew et al, sudo for convenience, and the no-X version of emacs because I’ve been using it since 1980):

$ su
# apt-get install sudo ssh curl emacs23-nox
# usermod -a -G sudo gronk
# exit

Now we install our Perlbrew environment as the user:

$ \curl -L http://install.perlbrew.pl | bash
$ echo source ~/perl5/perlbrew/etc/bashrc >> ~/.bash_profile
$ perlbrew install 5.20.2 -j5

The -j5 means use five processes for the Make. Then:

$ perlbrew switch 5.20.2
$ perlbrew install-cpanm
$ curl -L https://cpanmin.us | \
perl - -M https://cpan.metacpan.org -n Mojolicious
$ cpanm Toadfarm

jhthorsen explains how this goes together:

You can’t run individual apps inside toadfarm as different user, but you can start toadfarm as root and change to a different user. This is true of Mojolicious generally.

In particular, we create an Init script with a #! (hashbang) that points to our user’s “brewed” Perl. From the documentation:

Remember that the hashbang can be anything, so if you have Toadfarm and Mojolicious running under plenv or Perlbrew you need to change the hashbang part…

So we change our Init script to start with:

#!/home/gronk/perl5/perlbrew/perls/perl-5.20.2/bin/perl

Then in the Toadfarm script called by the Init script, which is started as root, we do the switch to our user (called some-www-user in the documentation, gronk here):

#/usr/bin/env perl
use Toadfarm -init;
# …
start [qw( http://*:80 https://*:443 )], user => "gronk", group => "www";

From the documentation:

Changing user has one implication: The workers will not use some-www-user’s secondary groups. So if “some-www-user” is part of the “www” and “db” groups, then the process will only be run as “some-www-user” part of the “www” group.

See also the IRC log for 26 April 2014.

The process of writing a Mojolicious plugin

I have gotten far enough into Mojolicious development to feel rather baffled again. This post is a work in progress, to be edited and filled in as I discover the answers.

According to the Rendering guide,

A skeleton for a full CPAN compatible plugin distribution can be automatically generated.

We are going to create a plugin called StaticText, so let’s do:

$ mojo generate plugin StaticText

What that actually does is the following actions inside your current directory, as we can see from its output log:

  [mkdir] ./Mojolicious-Plugin-StaticText/lib/Mojolicious/Plugin
  [write] ./Mojolicious-Plugin-StaticText/lib/Mojolicious/Plugin/StaticText.pm
  [mkdir] ./Mojolicious-Plugin-StaticText/t
  [write] ./Mojolicious-Plugin-StaticText/t/basic.t
  [exist] ./Mojolicious-Plugin-StaticText
  [write] ./Mojolicious-Plugin-StaticText/Makefile.PL

Q: How exactly can I develop this inside an example application? The documentation’s standard load procedure:

$self->plugin('Mojolicious::Plugin::StaticText');

fails because it doesn’t look in Mojolicious-Plugin-StaticText/lib/Mojolicious/Plugin/ of course.

Q: How do I install my common personal plugins into my individual “apps” ? Presumably I should be able to “git checkout” them directly into… lib/Plugin/ …? Obviously I’m not going to edit them into the MyApp:: namespace, right?

Q: Should use the Mojolicious::Plugin namespace for plugins that I consider to be eventual candidates for CPAN, or what is best practice for naming my personal plugins that I plan on using across multiple projects?

Q: I see how to pass parameters when loadiing a plugin, but does each plugin get loaded, or registered, just once, or can I have multiple instances of a plugin?

NOTE: Refer also to the Plugins document

Q: Where do I store “instance data” that is computed from the configuration parameters I pass when loading a plugin?

A: Plugins are simply modules, not objects. If we want to save configuration data at load-time, we could save that in an attribute of the application like this: $app->attr(‘some_attribute_name’ => value) which we later access as just “$app->some_attribute_name”.

A good example of how configuration parameters are passed, handled, and saved is http://search.cpan.org/~madcat/Mojolicious-Plugin-Database/lib/Mojolicious/Plugin/Database.pm

Behind the curtain: when you call attr, that actually builds and then eval(s) a string which defines a function by the supplied name. That function optionally sets (if called with a value), and always returns the value of the named key in the object upon which you originally called ->attr(). So in the case of doing $app->attr(‘foo’) the function would create and use $attr->{‘foo’}.

Q: The Mojolicious::Plugins guide shows the example code

$plugins->register_plugin('MyApp::Plugin::SomeThing', Mojolicious->new);

below the presumably required code:

use Mojolicious::Plugins;

my $plugins = Mojolicious::Plugins->new;
push @{$plugins->namespaces}, 'MyApp::Plugin';

Yet elsewhere, for example in http://mojolicio.us/perldoc/Mojolicious/Plugin/JSONConfig ,  we see a completely different style that runs with application start:

sub startup {
  my $self = shift;
  $self->plugin('Mojolicious::Plugin::StaticText');

What exactly is the difference between these two?

Q: In the version with

$plugins->register_plugin('MyApp::Plugin::SomeThing', Mojolicious->new);

what does the Mojolicious->new do? According to the documentation for Mojolicious->new(), that should “Construct a [whole other] new Mojolicious application and call ‘startup’… set up logging… the renderer, static file server, default set of plugins…” We can’t seriously be loading a whole new “app” nested inside of our current “app” can we? What’s going on?

Q: After the name of the plugin, what happens to the remaining arguments to $self->plugin()?

A: The optional second argument must be either a reference (to a scalar, array, hash, subroutine, etc.) or a scalar. If it’s just a plain scalar such as 3, like the plugin’s register() subroutine will be called with a reference to a hash like this: {3, undef}. In general, you will probably want to pass a reference to a hash that contains your configuration parameters.

Mojolicious: An interview with author Sebastian Riedel

We are experiencing Perl’s most successful period since the 1990s. I-Programmer spoke with Sebastian Riedel, originator of the relatively new Mojolicious system built on Perl.

…The truth is that installing Mojolicious is simply a very fast and pleasant experience… we’ve done with tight integration of WebSockets and event loops, [and] the ability to grow from single-file prototypes to well structured web applications.

The first example application you encounter on our website… doesn’t look very complicated at all. But once you start digging a little deeper, you’ll quickly realize how crazy (in a good way) it really is, and how hard it would be to replicate with any other web framework, in any language…

Read the rest of the story at i-programmer, 11 December 2014.

Mojo: Content boxes like “Widgets”

When we create a sidebar, generally we would like to suppress emitting the wrapper and title of our “widgets” (yes, I am still coming from the view of a developer too long having done nothing but WordPress) when there is no content “in the box.”

In code, we can examine the currently defined stash data or content areas like this:

if (length($c->content('content')) { … }

Note that, at least as of Mojolicious 6.0, content() will actually create an empty content area even if not given any content, so we must test length(); for stash data, we probably would use defined instead.

We can do this inside a template too. Here’s an example for both content and stash:

#!/usr/bin/env perl

use Mojolicious::Lite;

# setup base route
any '/' => sub {
  my $self = shift;

  # Example stash data and content
  $self->stash( text2 => 'some text' );
  $self->content( area3 => 'some text' );
  $self->render('index');
};

app->start;

__DATA__

@@ index.html.ep

<!DOCTYPE html>
<html>
<head><title>Widget Boxes</title></head>
<body>

  % if (defined stash('text1')) {
      <b><%= stash('text1') %></b>
  % } else {
      <em>No content</em>
  % }

<hr width='50%'>
  % if (defined stash('text2')) {
      <b><%= stash('text2'); %></b>
  % } else {
      <em>No content</em>
  % }

<hr>

  % if (length content('area1')) {
      <b><%= content('area1') %></b>
  % } else {
      <em>No content</em>
  % }
<hr width='50%'>

  % if (length content('area2')) {
      <b><%= content('area2') %></b>
  % } else {
      <em>No content</em>
  % }
<hr width='50%'>

  % if (length content('area3')) {
      <b><%= content('area3') %></b>
  % } else {
      <em>No content</em>
  % }

</body>
</html>

Zero to Mojolicious, Part 3: Using External Content from WP-API

Last time, we created a site_design template for Mojolicious which built on a basic layout template for Zurb Foundation. On top of this we can use our content-specific templates, with the two lower templates nested below.

Here’s an example that uses content from a WordPress™ Post. The simplest way to retrieve the actual content of just a post from WordPress is to use the rather new WP-API. This was to be part of WP core in their version 4.1, and although its appearance in core has been delayed, can be installed as the “WP REST API” plugin from the WP repository.

Once the WP-API plugin is enabled, we can request a URI like one on this site as:

http://blog.wlindley.com/wp-json/wp/posts/1265

That will return a JSON result with the rendered HTML (WordPress stores post and page content in the database with newline breaks instead of paragraph tags, and an assortment of other changes we do not want to deal with here) as well as a variety of metadata about the post. Unfortunately the current WP-API plugin does not let us request by post name, only by internal ID. One hopes the API will be more complete when it is fully released. Also beware the URI and request schema may differ in a final release.

We can use Mojo::UserAgent to retrieve the remote content and parse its JSON.

In a new file lib/MyApp/Controller/ then we add a controller. We use the rendered HTML versions for the content, title, and excerpt, while copying the other values with unmodified keys all to the stash, which we can then reference in our template. The two templates notfound and nopage (not shown here) are used when things go wrong.

package MyApp::Controller::Blogpage;
use Mojo::Base 'Mojolicious::Controller';

use Data::Dumper;
use Mojo::UserAgent;

# This action will render a template
sub retrieve {
  my $self = shift;

  my $ua = Mojo::UserAgent->new;

  my $id = $self->param('id');

  if (defined $id) {
      $id =~ s/\D//g; # remove all non-digits
  }
  if (!$id) {
      $self->redirect_to('notfound');
      return;
  }

  my $tx = $ua->get("http://blog.wlindley.com/wp-json/wp/posts/${id}");

  if ($tx->success) {
      my $value = $tx->res->json;

      # Render template "blogpage/retrieve.html.ep" with
      # values from WP-API
      $self->render((map { $_ => $value->{$_}->{rendered} }
                           qw/content title excerpt/) ,
		    (map { my $vname = $_;
			   $vname =~ s/^format\Z/wp_format/;
			   $vname => $value->{$_} }
                           qw/date type format comment_status id modified
				ping_status featured_image author sticky
                                excerpt slug guid link/) );
  } else {
      $self->redirect_to('nopage');
  }  
}

1;

and create a template in templates/blogpage/retrieve.html.ep. Note how we are using the site_layout template from last time:

% layout 'site_layout';
% title "$title";
<h1><%= $title =%></h1>

<blockquote>Published on <%= $date %></blockquote>
%== $content

Of course you can use any of the metadata tags which we copy in the controller (featured_image might be an interesting one).

Then in our MyApp.pm we add this into the startup subroutine:

$r->get('/blog')->to('blogpage#retrieve');

Now we can visit http://localhost:3000/blog?id=1265 and you should see the content of the WordPress post, dropped into the content section that you made in the Foundation style theme from last time.

Next time, we still plan to access a database.

"Si datur citrea, sucus faciunt" (When life gives you lemons, make lemonade)