Building more Simutrans tools: the Perl Imager package

Tags:

In building some new tools to support Simutrans development, I found this bit of sage advice dated 2005 from Tony Cook, the developer of the Imager module. From that I revised this example to translate the "transparent" cyan that Simutrans wants in its pakfile images, to actual transparency.

Note the use of new function signatures, and hash slices.

use v5.20;
use feature 'signatures';
no warnings 'experimental::signatures';

use Imager;

sub replace_color ($img, $from_color, $to_color) {
    # Inspired by http://www.perlmonks.org/?node_id=497355
    my $rpnexpr = <<'EOS';
x y getp1 !pix
@pix red from_red eq
@pix green from_green eq
@pix blue from_blue eq
and and
@pix alpha from_alpha eq
and
to_red to_green to_blue to_alpha rgba @pix ifp
EOS

    my %constants;
    # Load values via hash slices
    @constants{map {"from_$_"} qw{red green blue alpha}} = 
        $from_color->rgba;
    @constants{map {"to_$_"  } qw{red green blue alpha}} =
        $to_color  ->rgba;
    return Imager::transform2({ rpnexpr => $rpnexpr,
                                constants => \%constants,
                                channels => 4},
                              $img);
}

my %special_colors = (
    'transparent_cyan' => [231,255,255],
    );

my $image = Imager->new(file => $filename);
$image = $image->convert(preset => 'addalpha');
#
# Replace visible color with transparency
#
my $color = Imager::Color->new( @{$special_color{transparent_cyan}} );

# make a work image the same size as our input; add an alpha channel
my $work = Imager->new(xsize => $image->getwidth,
                       ysize => $image->getheight, channels => 4);
# fill it with the color we want transparent
$work->box(filled => 1, color => $color);

# get an image with that color replaced with transparent black
my $out = $work->difference(other => $image);

$out->write(file => 'example-output.png');

Further examples of the little Transform script language could change from more-standard "real" transparent PNGs to what Simutrans wants, or help highlight unwanted special colors that can result from anti-aliasing, or do all kinds of color or shape transforms.

Simutrans: De-mystifying the PAK format

Tags:

Let's de-mystify the .pak file format. Paks are actuallyfairly simple data files, although the details can certainly be complex.

In besch/reader/obj_reader.cc, obj_reader_t::read_file() opens a file and calls read_nodes() in that same .cc to reach each node. The file begins with the version information, terminated with a Ctrl+Z (0x1A) byte:

<code class="bbc_code" style="font-size: 66%;">53 69 6d 75 74 72 61 6e  73 20 6f 62 6a 65 63 74  |Simutrans object|
20 66 69 6c 65 0a 43 6f  6d 70 69 6c 65 64 20 77  | file.Compiled w|
69 74 68 20 53 69 6d 4f  62 6a 65 63 74 73 20 30  |ith SimObjects 0|
2e 31 2e 33 65 78 70 0a  1a eb 03 00 00 52 4f 4f  |.1.3exp......ROO|
54 01 00 00 00 42 55 49  4c 26 00 25 00 08 80 03  |T....BUIL&.%....|</code>

Following that are four bytes of Pak-File version (eb 03 00 00 above), and then a series of nodes until the end of file. Each node is processed by its appropriate reader found in the besch/reader/ subdirectory. In the file, each node begins with four characters describing the node type, as defined in besch/objversion.h:

<code class="bbc_code">enum obj_type
{
        obj_bridge      = C4ID('B','R','D','G'),
        obj_building    = C4ID('B','U','I','L'),</code>

and then a 2-byte (16-bit) child count and 2-byte (16-bit) data block size. If the data block is more than 64k bytes, 0xFFFF is used for the data size, followed by a four-byte (32-bit) data block size. Then the actual data block bytes, followed by any additional nodes in this same format.

The child count indicates how many of the following nodes are considered to belong to (be "inside") the current node. The BUIL node in the example has 0x0026 child nodes. This is how, for example, a single pak file can contain multiple objects, with each object containing several child nodes.

Note that read_nodes() chooses the internal class type from the four-character name, using the following line of code:

<code class="bbc_code">        obj_reader_t *reader = obj_reader->get(static_cast<obj_type>(node.type));
</code>

How exactly that works, in converting a text representation to a somewhat conceptual C++ class type, is left to the student as an exercise.

HTML 5 Semantic markup divisions: When to use which

Tags:

[caption id="" align="alignnone" width="1095"] HTML 5 semantic section flowchart[/caption]

Available from "HTML 5 Doctor" as a PDF

See also "Dive into HTML 5 Semantics"

Toadfarm on Base Debian: An absolutely minimal multi-tenant webserver

Tags:

Here we create an absolutely minimal multi-tenant webserver, without even installing Apache or NginX. Although we can add either of those later, with our content running under them as reverse proxies, this tutorial lets you run one or more Mojolicious "apps" mounted under a single toadfarm startup script, saving you the "default perl memory" times the number of worker processes and apps.

We will b using Jan Henning Thorsen's Toadfarm:

Toadfarm is a module for configuring and starting your Mojolicious applications. You can either combine multiple applications in one script, or just use it as a init script.


First, perform a Debian Netinstall on a virtual machine or virtual host. The only modification to all the default selections is to enable only "Standard System Utilities" and disable desktop, database, mail, print, file, and web server. We will be building our own of all those!

Now boot up into that absolutely stock, base machine. For illustration, we will assume your username is gronk − change this as necessary. First let's install a few system packages (ssh for access; curl for installing Perlbrew et al; sudo for convenience; build-essential for GCC, Make, and the compiler tools; and the no-X version of emacs because I've been using it since 1980):

$ su
# apt-get install sudo ssh curl build-essential emacs-nox
# usermod -a -G sudo <em>gronk</em> # exit

Now we install our Perlbrew environment as the user:

$ \curl -L http://install.perlbrew.pl | bash
$ echo source ~/perl5/perlbrew/etc/bashrc >> ~/.bash_profile
$ perlbrew install 5.22.0 -j5

The -j5 means use five processes for the Make. Then:

$ perlbrew switch 5.22.0
$ perlbrew install-cpanm
$ curl -L https://cpanmin.us |     perl - -M https://cpan.metacpan.org -n Mojolicious
$ cpanm Toadfarm

jhthorsen explains how this goes together:

You can't run individual apps inside toadfarm as different user, but you can start toadfarm as root and change to a different user.

[

This is true of Mojolicious generally ](https://metacpan.org/pod/Mojo::Server#user) .

In particular, we create an Init script with a #! (hashbang) that points to our user's "brewed" Perl. From the documentation:

Remember that the hashbang can be anything, so if you have Toadfarm and Mojolicious running under

[

plenv ](https://github.com/tokuhirom/plenv) or Perlbrew you need to change the hashbang part…

So we change our Init script to start with:

#!/home/<em>gronk</em>/perl5/perlbrew/perls/perl-5.22.0/bin/perl

Then in the Toadfarm script called by the Init script, which is started as root, we do the switch to our user (called some-www-user in the documentation, gronk here):

#/usr/bin/env perl
use Toadfarm -init;
# …
start [qw( http://*:80 https://*:443 )], user => "<em>gronk</em>", group => "<em>www</em>";

From the documentation:

Changing user has one implication: The workers will not use some-www-user's secondary groups. So if "some-www-user" is part of the "www" and "db" groups, then the process will only be run as "some-www-user" part of the "www" group.

See also the IRC log for 26 April 2014.

The process of writing a Mojolicious plugin

Tags:

I have gotten far enough into Mojolicious development to feel rather baffled again. This post is a work in progress, to be edited and filled in as I discover the answers.

According to the Rendering guide,

A skeleton for a full CPAN compatible plugin distribution can be automatically generated.

We are going to create a plugin called StaticText, so let's do:

$ mojo generate plugin StaticText

What that actually does is the following actions inside your current directory, as we can see from its output log:

  [mkdir] ./Mojolicious-Plugin-StaticText/lib/Mojolicious/Plugin
  [write] ./Mojolicious-Plugin-StaticText/lib/Mojolicious/Plugin/StaticText.pm
  [mkdir] ./Mojolicious-Plugin-StaticText/t
  [write] ./Mojolicious-Plugin-StaticText/t/basic.t
  [exist] ./Mojolicious-Plugin-StaticText
  [write] ./Mojolicious-Plugin-StaticText/Makefile.PL

Q: How exactly can I develop this inside an example application? The documentation's standard load procedure:

$self->plugin('Mojolicious::Plugin::StaticText');

fails because it doesn't look in Mojolicious-Plugin-StaticText/lib/Mojolicious/Plugin/ of course.

Q: How do I install my common personal plugins into my individual "apps" ? Presumably I should be able to "git checkout" them directly into... lib/Plugin/ ...? Obviously I'm not going to edit them into the MyApp:: namespace, right?

Q: Should use the Mojolicious::Plugin namespace for plugins that I consider to be eventual candidates for CPAN, or what is best practice for naming my personal plugins that I plan on using across multiple projects?

Q: I see how to pass parameters when loadiing a plugin, but does each plugin get loaded, or registered, just once, or can I have multiple instances of a plugin?

NOTE: Refer also to the Plugins document

Q: Where do I store "instance data" that is computed from the configuration parameters I pass when loading a plugin?

A: Plugins are simply modules, not objects. If we want to save configuration data at load-time, we could save that in an attribute of the application like this: $app->attr('some_attribute_name' => value) which we later access as just "$app->some_attribute_name".

A good example of how configuration parameters are passed, handled, and saved is http://search.cpan.org/~madcat/Mojolicious-Plugin-Database/lib/Mojolicious/Plugin/Database.pm

Behind the curtain: when you call attr, that actually builds and then eval(s) a string which defines a function by the supplied name. That function optionally sets (if called with a value), and always returns the value of the named key in the object upon which you originally called ->attr(). So in the case of doing $app->attr('foo') the function would create and use $attr->{'foo'}.

Q: The Mojolicious::Plugins guide shows the example code

$plugins->register_plugin('MyApp::Plugin::SomeThing', Mojolicious->new);

below the presumably required code:

use Mojolicious::Plugins;

my $plugins = Mojolicious::Plugins->new;
push @{$plugins->namespaces}, 'MyApp::Plugin';

Yet elsewhere, for example in http://mojolicio.us/perldoc/Mojolicious/Plugin/JSONConfig , we see a completely different style that runs with application start:

sub startup {
  my $self = shift;
  $self->plugin('Mojolicious::Plugin::StaticText');

What exactly is the difference between these two?

Q: In the version with

$plugins->register_plugin('MyApp::Plugin::SomeThing', Mojolicious->new);

what does the Mojolicious->new do? According to the documentation for Mojolicious->new(), that should "Construct a [whole other] new Mojolicious application and call 'startup'… set up logging… the renderer, static file server, default set of plugins…" We can't seriously be loading a whole new "app" nested inside of our current "app" can we? What's going on?

Q: After the name of the plugin, what happens to the remaining arguments to $self->plugin()?

A: The optional second argument must be either a reference (to a scalar, array, hash, subroutine, etc.) or a scalar. If it's just a plain scalar such as 3, like the plugin's register() subroutine will be called with a reference to a hash like this: {3, undef}. In general, you will probably want to pass a reference to a hash that contains your configuration parameters.