MariaDB: Get the Load Average

Here we present a simple user defined function in C, that returns the load average.

The function returns a string formatted like top‘s because C functions cannot return individual rows nor an array of reals.

The following code can be viewed as an example code for user defined functions in C — or alternatively as an example of how to use snprintf() correctly.

Continue reading MariaDB: Get the Load Average

MariaDB Replication on the Same Host for Testing

In this short tutorial, we’ll go through the steps required to set up MariaDB for replication on the same host. We are doing this for development purposes, and only run ephemeral MariaDB servers directly in terminal windows. We don’t edit any config files and instead use command line parameters to set up the server processes.

These instructions go through the motion of creating a database, populating it with some data, dumping it, and configuring replication for it—even though this is strictly not necessary. This is in preparation for a database which already contains some data in it; if you want to configure replication for an entirely fresh installation, you can cut out some steps.

The path to the MariaDB installation reflects MacPorts but apart from that these instructions are entirely independent of any operating system.

Notice. This tutorial does not go over the steps required for securing a MariaDB installation and doesn’t include any reference to passwords nor grant statements.

By the end of the process, you should have at least four open terminal windows, two of them running MariaDB servers, and two connection to each of them.

Continue reading MariaDB Replication on the Same Host for Testing

Building OpenSSL with OpenWatcom on ArcaOS: First Porting Effort

I have previously written about my OpenSSL port to OS/2. Here I go over the first C code changes that make it compile with OpenWatcom 1.9.

As previously mentioned, I’m doing this on ArcaOS; so some Unix-ism will be present in the build system. This is apparently unavoidable, and fortunately, ArcaOS already comes with a lot of Unix ports; both out of the box and installed through yum (or the Arca Noae Package Manager).

The following changes leave us with a port that builds for the most part. Some parts will need more changes before they are strictly “correct” and we’ll return to those later.
Continue reading Building OpenSSL with OpenWatcom on ArcaOS: First Porting Effort

Building OpenSSL with OpenWatcom on ArcaOS: The Initial Hurdles

Building OpenSSL on a new system or compiler presents a lot of hurdles. The build system is a perl hack upon a perl hack; and no documentation about how it works, from what I can tell.

I am porting OpenSSL 1.0.2, which is what the software I want to work on expects. Hopefully my work can be applied to other versions as well.

Here I go through the initial steps to get a working build system on ArcaOS 5, also known as OS/2, using the OpenWatcom compiler; even though OpenSSL has already been ported to the EMX runtime, with GCC.

Because we’re using perl, the steps here are taken after installing perl with

yum install perl

and this gives us a working build system with the changes below, so the real work on porting OpenSSL can start.
Continue reading Building OpenSSL with OpenWatcom on ArcaOS: The Initial Hurdles

Using the OpenWatcom Assembler with Jam

I’ve previously covered the OS/2 port of Perforce Jam.

Here we’ll go over what it takes to make Jam know about the OpenWatcom assembler, Wasm.

First we can look at the Jambase file, in the source where we find the

actions As
{
$(AS) $(ASFLAGS) $(ASHDRS) -o $(<) $(>)
}

and we see that Jam has a few variables related to assembler files.

The Jamfile

First, we tell Jam that our assembler is Wasm.

AS = wasm ;

Then we redefine the As actions to be

actions As
{
$(AS) -q $(ASFLAGS) $(ASHDRS) -fo=$(<) $(>)
}

where -fo=$(<) is the output file, and $(>) the input file. We could also have used the newer syntax, $(1) and $(2). The -q tells Wasm to operate quetly; it means we won’t see the copyright information every time we run Jam; only the nice As file.asm output.

Now, we can make use of these changes with

Main hello.exe : hello.c sum.asm ;

The Sources

The source files we mentioned above are hello.c and sum.asm.

Here is hello.c

  #include <stdio.h>

  int sum( int a, int b );

  int main( int argc, char *argv[] ) {

    printf( "Hello, %d.\n", sum( 3, 5 ) );
    return 0;
  }

And here is sum.asm

  .386
  .model flat
  .code
  PUBLIC sum_
  sum_ PROC
      add eax, edx
      ret
  sum_ ENDP
  END

These two files demonstrate how we can declare a function in C, and then define it in an assembly file. We are using the register calling convention of OpenWatcom. The first argument is in eax, the second in edx and the return value is again in eax. That means we don’t have to worry about stack manipulation at all, in a simple function like sum(). OpenWatcom appends _ to function names with the register convention, so we have to append it also in the .asm file.

With the above Jamfile, we just run

jam

and our demonstration program is built and linked properly for us.

Running the Demo

Now, when we run

hello

on the command line, we see

Hello, 8.

Final Words

This has been written as Jamfile demonstration, but it can also be viewed as a demonstration of the register calling in OpenWatcom.

Do you need professional help with OS/2? Write to johann@myrkraverk.com.

Building Jam 2.6 on OS/2 with OpenWatcom

These instructions pertain to ArcaOS 5.0.1 and OpenWatcom 1.9. Any other OS/2 versions and compilers may behave slightly differently; your mileage may wary.

Here we will go through the process of porting Perforce Jam 2.6, released in 2014. At the time of writing, the source can be downloaded from the Perforce website.

There is also FT Jam, from the FreeType project, which is quite possibly more advanced, even though it’s based on Perforce Jam 2.5. and hasn’t been updated since about 2006, apparently.

Haiku Jam is also worth checking out.

JamPlus may also be a thing.

Copyright and disclaimers

Jam has the following copyright:

Copyright 1993-2002 Christopher Seiwald and Perforce Software, Inc.

License is hereby granted to use this software and distribute it freely, as long as this copyright notice is retained and modifications are clearly marked.

ALL WARRANTIES ARE HEREBY DISCLAIMED.

Furthermore, no work has been done to insure the OS/2 port is fit for any particular purpose, except to ensure it successfully builds itself.

The Makefile

The settings at the start should look like this.

  CC = wcl386
  CFLAGS =
  EXENAME = jam0.exe
  TARGET = -fe=$(EXENAME)

That is, the compiler is wcl386, with no special C flags, and the name is jam0.exe without the period and slash. The -fe= tells wcl386 what to name the executable file; possible mnemonic: file executable.

Optionally add wtouch all as the final command in the “all” rule, or build with -c; otherwise WMake will complain the “all” target has not been built.

That takes care of the Makefile.

The make1.c file

There are minor changes in the make1.c file, because it unconditionally calls unlink() and we “fix that” with these macros at the start.

  # ifdef OS_OS2                                         
  # define INCL_DOSFILEMGR 
  # include <os2.h>
  # define unlink DosDelete
  # endif

You will need to remove the READ ONLY flag from the file before it
can be saved.

attrib -r make1.c

The execunix.c file

In this file, there are some major changes; so to speak.

At the top we add

  # define INCL_DOSPROCESS
  # define INCL_DOSFILEMGR
  # include <os2.h>
  # define unlink DosDelete

inside the OS/2 section, because it also calls unlink() unconditionally. And we also need to get access to process information, to get the PID.

The complete section shoudl look like this:

  # ifdef OS_OS2
  # define USE_EXECNT
  # include <process.h>
  # define INCL_DOSPROCESS
  # define INCL_DOSFILEMGR
  # include <os2.h>
  # define unlink DosDelete
  # endif

After this change, on line 188, is a

  # ifdef USE_EXECNT

inside which is an NT call GetCurrentProcessId() which we port to use DosGetInfoBlocks() instead.

So at the start of the scope, we declare two new pointers, to the thread and process structure respectively.

            TIB *thread;
            PIB *process;

And before sprintf() that refers to the PID, we call

  DosGetInfoBlocks( &thread, &process );

while inside the sprint() call itself, where GetCurrentProcessId() used to
be, we have

  process->pib_ulpid

instead. The complete function call now looks like

	    sprintf( cmdtab[ slot ].tempfile, "%s\\jam%lut%d.bat", 
				tempdir, process->pib_ulpid, slot );

where we have carefully updated the format specification to reflect unsigned long %lu, instead of integer %d. Even though integers and longs are the same size on 32bit OS/2, there’s no reason to be sloppy about format specifiers in the printf() family of functions.

You will also need to remove the read only bit before saving the file.

attrib -r execunix.c

The jamgram.c and parse.c files

These source files will have some warning about undeclared function names. This is ok, and is not worth fixing just to get Jam working on OS/2. Maybe later, and particularly if Perforce is willing to accept contributions, it may turn out to be a worthwhile effort to fix.

This takes care of the source files.

Building It

Now do

wmake -u -c

in a command prompt, and Jam will bootstrap itself. The Jam built object files and executable will be in a subdirectory called bin.os2x86 so don’t be surprised if you only see the jam0.exe file at first.

On modern hardware, the complete build takes only a few seconds.

The -u is for unix compatibility mode, and allows WMake to understand the backslash line continuation syntax; lines that end with \ are joined. Without this, there will be an error.

As mentioned previously, the -c is to stop WMake from complaining that “all” has not been built. Leave it out of you added wtouch all to the rule.

Installation

To install Jam, just copy the jam.exe file into c:\usr\local\bin, which you may have to create first, or wherever your %UNIXROOT% is.

mkdir c:\usr\local
mkdir c:\usr\local\bin
copy bin.os2x86\jam.exe c:\usr\local\bin\

It is normal that the mkdir commands fail when the directory exists already.

Alternatively, you can write the commands like this, in case your
UNIXROOT is not C:.

mkdir %UNIXROOT%\usr\local
mkdir %UNIXROOT%\usr\local\bin
copy bin.os2x86\jam.exe %UNIXROOT%\usr\local\bin

Final Words

Congratulations, now you have completed the build and installation of Perforce Jam 2.6.

Happy building.

Do you need professional assistance with OS/2? Write to johann@myrkraverk.com.

Setting up Perforce Helix Core Service on OS X High Sierra

Here is a guide about how to start Perforce Helix Core as a global daemon on OS X High Sierra, running under a separate user called perforce. By the nature of this guide, some of it applies directly to my own system; readers are expected to identify those instances and change appropriately.

First, we download the p4d and p4 executable files. Some people may prefer the visual client, but this guide is focused on the command line.

Then we install the binaries into /usr/local/sbin and /usr/local/bin, respectively.

Install the server binary.

  sudo mkdir -p /usr/local/sbin
  sudo cp p4d /usr/local/sbin
  sudo chown root:wheel /usr/local/sbin/p4d
  sudo chmod 555 /usr/local/sbin/p4d

And install the client binary.

  sudo mkdir -p /usrl/local/bin
  sudo cp p4 /usr/local/bin/
  sudo chown root:wheel /usr/local/bin/p4
  sudo chmod 555 /usr/local/bin/p4

We’ll create a group and user called perforce, and store the version control database in /usr/local/var/perforce with the log in /usr/local/var/log/p4d.log.

Here I’ve carefully chosen the unique id 268 for both the group and user because it’s free on my system. To see all group ids in use on a system, this one liner can be used,

  for f in `dscl . -list /Groups`; do dscl . -read /Groups/$f; \
      done | grep PrimaryGroupID | sort -k2 -n

and to see all the user ids, this one liner can be used.

  for f in `dscl . -list /Users`; do dscl . -read /Users/$f; \
      done | grep UniqueID | sort -k2 -n

We create the group with

  sudo dscl . -create /Groups/perforce
  sudo dscl . -create /Groups/perforce PrimaryGroupID 268
  sudo dscl . -create /Groups/perforce Password '*'

and then we create the user with

  sudo dscl . -create /Users/perforce      
  sudo dscl . -create /Users/perforce UniqueID 268
  sudo dscl . -create /Users/perforce UserShell /usr/bin/false
  sudo dscl . -create /Users/perforce RealName 'Perforce Server'
  sudo dscl . -create /Users/perforce NFSHomeDirectory /usr/local/var/perforce
  sudo dscl . -create /Users/perforce PrimaryGroupID 268
  sudo dscl . -create /Users/perforce Password '*'

Then we create the log file and database directory

  sudo mkdir -p /usr/local/var/log
  sudo touch    /usr/local/var/log/p4d.log
  sudo mkdir -p /usr/local/var/perforce

and set the ownership to the new user and group.

  sudo chown perforce:perforce /usr/local/var/perforce
  sudo chown perforce:perforce /usr/local/var/log/p4d.log 

Then we create the launch daemon description file.

  sudo nano /Library/LaunchDaemons/com.perforce.plist

and add the following.

  <?xml version="1.0" encoding="UTF-8"?>
  
  <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Debug</key>
      <false/>
      <key>Label</key>
      <string>com.perforce</string>
      <key>OnDemand</key>
      <false/>
      <key>Program</key>
      <string>/usr/local/sbin/p4d</string>
      <key>ProgramArguments</key>
      <array>
        <string>/usr/local/sbin/p4d</string>
      </array>
      <key>EnvironmentVariables</key>
      <dict>
        <key>P4LOG</key>
        <string>/usr/local/var/log/p4d.log</string>
        <key>P4PORT</key>
        <string>1666</string>
        <key>P4ROOT</key>
        <string>/usr/local/var/perforce/</string>
        <key>P4NAME</key>
        <string>Shinka</string> 
        <key>P4DESCRIPTION</key>
        <string>Myrkraverk's Perforce</string> 
      </dict>
      <key>RunAtLoad</key>
      <true/>
      <key>ServiceDescription</key>
      <string>Launches Perforce Server</string>
      <key>StartInterval</key>
      <integer>180</integer> 
      <key>KeepAlive</key>
      <true/>
      <key>UserName</key>
      <string>perforce</string>
      <key>GroupName<key>
      <string>perforce</string>
    </dict>
  </plist>

And finally, start the Helix Core service with

  sudo launchctl load /Library/LaunchDaemons/com.perforce.plist

we then issue

   p4 set P4PORT=localhost:1666

and we test it with

  p4 help

which outputs at the end

  Server 2018.1/1637071.

so we know it's all working.

The steps to properly set up Helix Core with p4 protect and so forth are left out of this guide; please see the administration guide from Perforce.

To use SSL for the connection, the P4PORT should start with ssl:.

XEmacs: Sorting Key-Value Lines by Value

XEmacs 21.4.24 [direct ftp download] and the latest stable release (2015) is the version I’m personally using. The directions here may well apply to GNU Emacs as well; I don’t know.

Most Emacs users are familiar with the command M-x sort-lines which alphabetically sorts the lines highlighted in the current buffer.

However, I had the wish to sort key: values as follows, by the value.

foo: 12
baz: 7
bar: 2

As you can see, in this instance, the value is numeric and lexicographical sorting of numbers results in

foo: 12
bar: 2
baz: 7

that is, alphabetical and not numeric.

In order to “fix this” we have to delve into the sorting internals of XEmacs.

First, let’s look at the function sort-lines, which is fairly straight forward.1

(defun sort-lines (reverse beg end)
  ;; [documentation string elided]
  (interactive "P\nr")
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (sort-subr reverse 'forward-line 'end-of-line))))

The most important thing to note here is the use of the function sort-subr. The rest of the code is simply boiler plate to limit the sort to the highlighted region. The (interactive "P\nr") is to read the start and end of the region into the argument values beg and end. For our purposes, we can simply treat this as “magic”; the effect of the newline embedded in the string is to separate the reverse (read with "P") from the region (read with "r") and that means people can sort in reverse alphabetical order with C-u M-x sort-lines. The code we’ll develop in here does not have this feature; adding it can be considered an exercise for the reader.

Alright, now we know how sort-lines works, and we know that we can use sort-subr to sort by values, the question is how do we do it?

First, and obviously, we read the documentation string with C-h f sort-subr. This tells us that there are two variables we can make use of,1

STARTKEYFUN moves from the start of the record to the start of the key. It may return either a non-nil value to be used as the key, or else the key is the substring between the values of point after STARTKEYFUN and ENDKEYFUN are called. If STARTKEYFUN is nil, the key starts at the beginning of the record.

and

COMPAREFUN compares the two keys. It is called with two strings and should return true if the first is “less” than the second, just as for `sort’. If nil or omitted, the default function accepts keys that are numbers (compared numerically) or strings (compared lexicographically).

The first thing we note here is the startkeyfun. It’ll allow us to limit the sort comparison to the value part of the lines. The trick here is to just move the point past the : (colon). We can do that with search-forward. Since in my case, and the example here, all the lines do have a colon, we’ll not consider the case where it might be missing in the line, hence we don’t impose any limit on the search (nil) nor do we care about errors (we allow them with nil); however, we limit the count to exactly one.

That leaves us with a call that looks like

  (search-forward ":" nil nil 1)

and to apply that to sort-subr we define an entirely new function, my-sort-key-value-lines. We wrap search-forward in a lambda for simplicity. Notice that we return nil explicitly from the lambda, because otherwise sort-subr will use its return value (the location of point in the buffer after the colon) and sort from that.2

(defun my-sort-key-value-lines (beg end)
  (interactive "r")
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (sort-subr nil 'forward-line 'end-of-line
                 (lambda ()
                   (search-forward ":" nil nil 1)
                   ;; returns point, so we explicitly return
                   nil)))))

And this is the code that results in lexicographical sorting of the values, but we want numeric sorting. There are at least two ways to fix that.

First, we can use the comparison function, to compare both arguments as numbers. We just have to convert the arguments to integers, and then compare them.2

(defun my-sort-key-value-lines (beg end)
  (interactive "r")
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (sort-subr nil 'forward-line 'end-of-line
		 (lambda ()
		   (search-forward ":" nil nil 1)
		   nil)
		 nil
		 (lambda (a b)
		   (< (string-to-number a)
		      (string-to-number b)))))))

In the code above we do that in the second lambda. The nil between them is the end-of-key function, which we don’t need to define because it’s the same as the end-of-record (represented by 'end-of-line in the above code).

The simpler method, is to do the conversion in the previous lambda, and use the default comparison function. Which results in the third revision.2

(defun my-sort-key-value-lines (beg end)
  (interactive "r")
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (sort-subr nil 'forward-line 'end-of-line
                 (lambda ()
                   (search-forward ":" nil nil 1)
                   (string-to-number (buffer-substring (point) (point-at-eol))))))))

You can now just drop this into your ~/.xemacs/init.el and use the command M-x my-sort-key-value-lines to sort key: value lines, whenever you have the need. And this leaves us with the desired numerical sort order.

bar: 2
baz: 7
foo: 12

1 This code is GPL.

2 This code can be considered WTFPL 2.0; at least the parts inside the lambdas and the rest is just boilerplate.

Compiling Sysbench on OS X Yosemite or Later

These instructions are applicable after cloning the git repository and generating the autoconfigure scripts.

git clone 'https://github.com/akopytov/sysbench.git' sysbench
cd sysbench
./autogen.sh

In order to build Sysbench1 with PostgreSQL and MariaDB support, you need to make sure both mysql_config and pg_config are in your path.

I use Zsh, so this is my way of doing it, when both Postgres and MariaDB have been installed with MacPorts.

path=( /opt/local/lib/mariadb-10.1/bin /opt/local/lib/postgresql96/bin $path )

Then run

./configure --with-pgsql --with-mysql --prefix=/path/of/your/choice

You are likely to get an error like

ld: library not found for -lgcc_s.10.4

if you do not also

export MACOSX_DEPLOYMENT_TARGET=10.10

before running make, while building the bundled LuaJit. This is documented in their installation instructions.

Of course, this isn’t taken care of by the wrapper Autotools, nor is there a configure flag to set this.

An alternative might be --with-system-luajit but that depends on your situation.

Then you finish it off with make install. Happy benchmarking.


1 I hope I’m linking to the right git repository.

Parsing Command Line Parameters with Yacc & Flex

This is a repost from 2012, but my old blog site disappeared.

Every once in a while someone comes along and asks how to parse command line parameters with Yacc & Flex. This is rather straight forward, but requires some knowledge of the generated code to get right.

Here we present a source template that does this. The user only has to edit the grammar and scanning rules. Some knowledge of C, Yacc and Flex is assumed.

The code is WTFPL licensed.

The template is written for Berkeley Yacc and the Reflex variant of Flex. It may be made to work with GNU Bison and (formerly SourceForge) Flex (now on GitHub), possibly with a few changes.

Table of Contents

Using the Template

In the file commandline.l we start to edit the scanner rules. For our example we make do with

%%

 // Here we put regular old scanning rules.

[a-z]+ { commandlinelval = commandlinetext; return WORD; }

%%

The only thing different here is that our customary yylval and yytext variables have changed names. The WORD token is defined in commandline.y.

Then in commandline.y we edit grammar rules as usual. We start with a list of tokens.

// Here we put regular old token declarations.
%token WORD SPACE

and then write our grammar

%%

// Here we put regular old grammar rules.

command: /* empty */
	|	words
	;

words:		word
	|	words word
	;

word:		WORD { printf( "\"%s\"\n", $1 ); }
	;

%%

Here we just print out the words returned by the scanner, one per line. We are using the fact that the lexer starts a new lexeme on calls to yywrap(). This means we do not have to insert any separator characters between the command line arguments we are parsing.

The provided makefile builds the example with the -p prefix parameter to yacc, which changes the symbol prefix from yy and the -P prefix parameter to reflex to do the same. This makes the template usable as-is with projects that use yacc & flex already.

% make
yacc -bcommandline -pcommandline -di commandline.y
reflex -Pcommandline commandline.l
cc -o commandline commandline.tab.c lex.commandline.c

Now we can run the example.

% ./commandline this is a simple example
"this"
"is"
"a"
"simple"
"example"

Understanding the Template

We use the technique presented previously to pass parameters to yacc and flex [link is to an archived copy] to feed argc and argv to our yywrap() function.

In commandlin.h we declare the argument structure.

// The argument structure we pass to yywrap().
struct arguments
{
    int argc, // The total number of arguments passed to main().

        arg;  // The argument we are actually going to parse.

    char **argv; // Pointer to the argument vector itself.
};

In commandline.l we have

int nextargument( struct arguments *args )
{

  // Prevent memory leaks.  This is safe because yy_current_buffer
  // is initialized to zero.
  if ( YY_CURRENT_BUFFER )
    {
      yy_delete_buffer( YY_CURRENT_BUFFER );
    }

  // If there are no more arguments, return 1 to signal we are done.
  if ( args->argc == args->arg )
    return 1;

  // Notice we increase args->arg here with ++.
  commandline_scan_string( args->argv[ args->arg++ ] );

  return 0;
}

as the yywrap() function (renamed) which calls yy_scan_string() for each argument passed to main(). yy_scan_string() has been renamed too.

The main() function itself is purely a template which builds a structure holding argc and argv which it then uses to pass on to yywrap() and yyparse().

int main( int argc, char *argv[] )
{
    // Initialize the argument structure we pass to yywrap().
    struct arguments args;
    args.argc = argc;
    args.arg = 1; // start at argument 1, not the command name.
    args.argv = argv;


    if ( argc > 1 )
	// This is actually our yywrap() function.  We could also have
	// used its return value to determine if there is an argument
	// to parse.
	nextargument( &args );
    else
	return 1;

    // We pass the argument structure to our yyparse().  Notice it's
    // been renamed to "commandlineparse."
    commandlineparse( (void *) &args );

    return 0;
}

Here we are careful to call yywrap() before our first call to yyparse() to initialize the input buffer.

Depending on the application, there may be no reason to change the main() function itself, merely rename it and called from the actual main().

References

Downloads

Downloads of individual files.

Downloads of the complete source archive.