MariaDB: Get the Load Average
Here we present a simple user defined function in C, that returns the load average.
The function returns a string formatted like top
‘s because C functions cannot return individual rows nor an array of real
s.
The following code can be viewed as an example code for user defined functions in C — or alternatively as an example of how to use snprintf()
correctly.
MariaDB Replication on the Same Host for Testing
In this short tutorial, we’ll go through the steps required to set up MariaDB for replication on the same host. We are doing this for development purposes, and only run ephemeral MariaDB servers directly in terminal windows. We don’t edit any config files and instead use command line parameters to set up the server processes.
These instructions go through the motion of creating a database, populating it with some data, dumping it, and configuring replication for it—even though this is strictly not necessary. This is in preparation for a database which already contains some data in it; if you want to configure replication for an entirely fresh installation, you can cut out some steps.
The path to the MariaDB installation reflects MacPorts but apart from that these instructions are entirely independent of any operating system.
Notice. This tutorial does not go over the steps required for securing a MariaDB installation and doesn’t include any reference to passwords nor grant statements.
By the end of the process, you should have at least four open terminal windows, two of them running MariaDB servers, and two connection to each of them.
Continue reading MariaDB Replication on the Same Host for Testing
Building OpenSSL with OpenWatcom on ArcaOS: First Porting Effort
I have previously written about my OpenSSL port to OS/2. Here I go over the first C code changes that make it compile with OpenWatcom 1.9.
As previously mentioned, I’m doing this on ArcaOS; so some Unix-ism will be present in the build system. This is apparently unavoidable, and fortunately, ArcaOS already comes with a lot of Unix ports; both out of the box and installed through yum (or the Arca Noae Package Manager).
The following changes leave us with a port that builds for the most part. Some parts will need more changes before they are strictly “correct” and we’ll return to those later.
Continue reading Building OpenSSL with OpenWatcom on ArcaOS: First Porting Effort
Building OpenSSL with OpenWatcom on ArcaOS: The Initial Hurdles
Building OpenSSL on a new system or compiler presents a lot of hurdles. The build system is a perl hack upon a perl hack; and no documentation about how it works, from what I can tell.
I am porting OpenSSL 1.0.2, which is what the software I want to work on expects. Hopefully my work can be applied to other versions as well.
Here I go through the initial steps to get a working build system on ArcaOS 5, also known as OS/2, using the OpenWatcom compiler; even though OpenSSL has already been ported to the EMX runtime, with GCC.
Because we’re using perl, the steps here are taken after installing perl with
yum install perl
and this gives us a working build system with the changes below, so the real work on porting OpenSSL can start.
Continue reading Building OpenSSL with OpenWatcom on ArcaOS: The Initial Hurdles
Using the OpenWatcom Assembler with Jam
I’ve previously covered the OS/2 port of Perforce Jam.
Here we’ll go over what it takes to make Jam know about the OpenWatcom assembler, Wasm.
First we can look at the Jambase file, in the source where we find the
actions As
{
$(AS) $(ASFLAGS) $(ASHDRS) -o $(<) $(>)
}
and we see that Jam has a few variables related to assembler files.
The Jamfile
First, we tell Jam that our assembler is Wasm.
AS = wasm ;
Then we redefine the As actions to be
actions As
{
$(AS) -q $(ASFLAGS) $(ASHDRS) -fo=$(<) $(>)
}
where -fo=$(<)
is the output file, and $(>)
the input file. We could also have used the newer syntax, $(1)
and $(2)
. The -q
tells Wasm to operate quetly; it means we won’t see the copyright information every time we run Jam; only the nice As file.asm
output.
Now, we can make use of these changes with
Main hello.exe : hello.c sum.asm ;
The Sources
The source files we mentioned above are hello.c
and sum.asm
.
Here is hello.c
#include <stdio.h> int sum( int a, int b ); int main( int argc, char *argv[] ) { printf( "Hello, %d.\n", sum( 3, 5 ) ); return 0; }
And here is sum.asm
.386 .model flat .code PUBLIC sum_ sum_ PROC add eax, edx ret sum_ ENDP END
These two files demonstrate how we can declare a function in C, and then define it in an assembly file. We are using the register calling convention of OpenWatcom. The first argument is in eax
, the second in edx
and the return value is again in eax
. That means we don’t have to worry about stack manipulation at all, in a simple function like sum()
. OpenWatcom appends _
to function names with the register convention, so we have to append it also in the .asm
file.
With the above Jamfile, we just run
jam
and our demonstration program is built and linked properly for us.
Running the Demo
Now, when we run
hello
on the command line, we see
Hello, 8.
Final Words
This has been written as Jamfile demonstration, but it can also be viewed as a demonstration of the register calling in OpenWatcom.
Do you need professional help with OS/2? Write to johann@myrkraverk.com.
Building Jam 2.6 on OS/2 with OpenWatcom
These instructions pertain to ArcaOS 5.0.1 and OpenWatcom 1.9. Any other OS/2 versions and compilers may behave slightly differently; your mileage may wary.
Here we will go through the process of porting Perforce Jam 2.6, released in 2014. At the time of writing, the source can be downloaded from the Perforce website.
There is also FT Jam, from the FreeType project, which is quite possibly more advanced, even though it’s based on Perforce Jam 2.5. and hasn’t been updated since about 2006, apparently.
Haiku Jam is also worth checking out.
Copyright and disclaimers
Jam has the following copyright:
Copyright 1993-2002 Christopher Seiwald and Perforce Software, Inc.
License is hereby granted to use this software and distribute it freely, as long as this copyright notice is retained and modifications are clearly marked.
ALL WARRANTIES ARE HEREBY DISCLAIMED.
Furthermore, no work has been done to insure the OS/2 port is fit for any particular purpose, except to ensure it successfully builds itself.
The Makefile
The settings at the start should look like this.
CC = wcl386 CFLAGS = EXENAME = jam0.exe TARGET = -fe=$(EXENAME)
That is, the compiler is wcl386
, with no special C flags, and the name is jam0.exe
without the period and slash. The -fe=
tells wcl386
what to name the executable file; possible mnemonic: file executable.
Optionally add wtouch all
as the final command in the “all” rule, or build with -c
; otherwise WMake will complain the “all” target has not been built.
That takes care of the Makefile.
The make1.c file
There are minor changes in the make1.c file, because it unconditionally calls unlink() and we “fix that” with these macros at the start.
# ifdef OS_OS2 # define INCL_DOSFILEMGR # include <os2.h> # define unlink DosDelete # endif
You will need to remove the READ ONLY flag from the file before it
can be saved.
attrib -r make1.c
The execunix.c file
In this file, there are some major changes; so to speak.
At the top we add
# define INCL_DOSPROCESS # define INCL_DOSFILEMGR # include <os2.h> # define unlink DosDelete
inside the OS/2 section, because it also calls unlink()
unconditionally. And we also need to get access to process information, to get the PID.
The complete section shoudl look like this:
# ifdef OS_OS2 # define USE_EXECNT # include <process.h> # define INCL_DOSPROCESS # define INCL_DOSFILEMGR # include <os2.h> # define unlink DosDelete # endif
After this change, on line 188, is a
# ifdef USE_EXECNT
inside which is an NT call GetCurrentProcessId()
which we port to use DosGetInfoBlocks()
instead.
So at the start of the scope, we declare two new pointers, to the thread and process structure respectively.
TIB *thread; PIB *process;
And before sprintf() that refers to the PID, we call
DosGetInfoBlocks( &thread, &process );
while inside the sprint()
call itself, where GetCurrentProcessId() used to
be, we have
process->pib_ulpid
instead. The complete function call now looks like
sprintf( cmdtab[ slot ].tempfile, "%s\\jam%lut%d.bat", tempdir, process->pib_ulpid, slot );
where we have carefully updated the format specification to reflect unsigned long %lu
, instead of integer %d
. Even though integers and longs are the same size on 32bit OS/2, there’s no reason to be sloppy about format specifiers in the printf()
family of functions.
You will also need to remove the read only bit before saving the file.
attrib -r execunix.c
The jamgram.c and parse.c files
These source files will have some warning about undeclared function names. This is ok, and is not worth fixing just to get Jam working on OS/2. Maybe later, and particularly if Perforce is willing to accept contributions, it may turn out to be a worthwhile effort to fix.
This takes care of the source files.
Building It
Now do
wmake -u -c
in a command prompt, and Jam will bootstrap itself. The Jam built object files and executable will be in a subdirectory called bin.os2x86
so don’t be surprised if you only see the jam0.exe
file at first.
On modern hardware, the complete build takes only a few seconds.
The -u
is for unix compatibility mode, and allows WMake to understand the backslash line continuation syntax; lines that end with \
are joined. Without this, there will be an error.
As mentioned previously, the -c
is to stop WMake from complaining that “all” has not been built. Leave it out of you added wtouch all
to the rule.
Installation
To install Jam, just copy the jam.exe
file into c:\usr\local\bin
, which you may have to create first, or wherever your %UNIXROOT%
is.
mkdir c:\usr\local
mkdir c:\usr\local\bin
copy bin.os2x86\jam.exe c:\usr\local\bin\
It is normal that the mkdir
commands fail when the directory exists already.
Alternatively, you can write the commands like this, in case your
UNIXROOT
is not C:
.
mkdir %UNIXROOT%\usr\local
mkdir %UNIXROOT%\usr\local\bin
copy bin.os2x86\jam.exe %UNIXROOT%\usr\local\bin
Final Words
Congratulations, now you have completed the build and installation of Perforce Jam 2.6.
Happy building.
Do you need professional assistance with OS/2? Write to johann@myrkraverk.com.
Setting up Perforce Helix Core Service on OS X High Sierra
Here is a guide about how to start Perforce Helix Core as a global daemon on OS X High Sierra, running under a separate user called perforce. By the nature of this guide, some of it applies directly to my own system; readers are expected to identify those instances and change appropriately.
First, we download the p4d and p4 executable files. Some people may prefer the visual client, but this guide is focused on the command line.
Then we install the binaries into /usr/local/sbin
and /usr/local/bin
, respectively.
Install the server binary.
sudo mkdir -p /usr/local/sbin sudo cp p4d /usr/local/sbin sudo chown root:wheel /usr/local/sbin/p4d sudo chmod 555 /usr/local/sbin/p4d
And install the client binary.
sudo mkdir -p /usrl/local/bin sudo cp p4 /usr/local/bin/ sudo chown root:wheel /usr/local/bin/p4 sudo chmod 555 /usr/local/bin/p4
We’ll create a group and user called perforce, and store the version control database in /usr/local/var/perforce
with the log in /usr/local/var/log/p4d.log
.
Here I’ve carefully chosen the unique id 268
for both the group and user because it’s free on my system. To see all group ids in use on a system, this one liner can be used,
for f in `dscl . -list /Groups`; do dscl . -read /Groups/$f; \ done | grep PrimaryGroupID | sort -k2 -n
and to see all the user ids, this one liner can be used.
for f in `dscl . -list /Users`; do dscl . -read /Users/$f; \ done | grep UniqueID | sort -k2 -n
We create the group with
sudo dscl . -create /Groups/perforce sudo dscl . -create /Groups/perforce PrimaryGroupID 268 sudo dscl . -create /Groups/perforce Password '*'
and then we create the user with
sudo dscl . -create /Users/perforce sudo dscl . -create /Users/perforce UniqueID 268 sudo dscl . -create /Users/perforce UserShell /usr/bin/false sudo dscl . -create /Users/perforce RealName 'Perforce Server' sudo dscl . -create /Users/perforce NFSHomeDirectory /usr/local/var/perforce sudo dscl . -create /Users/perforce PrimaryGroupID 268 sudo dscl . -create /Users/perforce Password '*'
Then we create the log file and database directory
sudo mkdir -p /usr/local/var/log sudo touch /usr/local/var/log/p4d.log sudo mkdir -p /usr/local/var/perforce
and set the ownership to the new user and group.
sudo chown perforce:perforce /usr/local/var/perforce sudo chown perforce:perforce /usr/local/var/log/p4d.log
Then we create the launch daemon description file.
sudo nano /Library/LaunchDaemons/com.perforce.plist
and add the following.
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Debug</key> <false/> <key>Label</key> <string>com.perforce</string> <key>OnDemand</key> <false/> <key>Program</key> <string>/usr/local/sbin/p4d</string> <key>ProgramArguments</key> <array> <string>/usr/local/sbin/p4d</string> </array> <key>EnvironmentVariables</key> <dict> <key>P4LOG</key> <string>/usr/local/var/log/p4d.log</string> <key>P4PORT</key> <string>1666</string> <key>P4ROOT</key> <string>/usr/local/var/perforce/</string> <key>P4NAME</key> <string>Shinka</string> <key>P4DESCRIPTION</key> <string>Myrkraverk's Perforce</string> </dict> <key>RunAtLoad</key> <true/> <key>ServiceDescription</key> <string>Launches Perforce Server</string> <key>StartInterval</key> <integer>180</integer> <key>KeepAlive</key> <true/> <key>UserName</key> <string>perforce</string> <key>GroupName<key> <string>perforce</string> </dict> </plist> And finally, start the Helix Core service with sudo launchctl load /Library/LaunchDaemons/com.perforce.plist
we then issue
p4 set P4PORT=localhost:1666
and we test it with
p4 help
which outputs at the end
Server 2018.1/1637071.
so we know it's all working.
The steps to properly set up Helix Core with p4 protect
and so forth are left out of this guide; please see the administration guide from Perforce.
To use SSL for the connection, the P4PORT should start with ssl:
.
XEmacs: Sorting Key-Value Lines by Value
XEmacs 21.4.24 [direct ftp download] and the latest stable release (2015) is the version I’m personally using. The directions here may well apply to GNU Emacs as well; I don’t know.
Most Emacs users are familiar with the command M-x sort-lines
which alphabetically sorts the lines highlighted in the current buffer.
However, I had the wish to sort key: values
as follows, by the value.
foo: 12 baz: 7 bar: 2
As you can see, in this instance, the value is numeric and lexicographical sorting of numbers results in
foo: 12 bar: 2 baz: 7
that is, alphabetical and not numeric.
In order to “fix this” we have to delve into the sorting internals of XEmacs.
First, let’s look at the function sort-lines, which is fairly straight forward.1
(defun sort-lines (reverse beg end) ;; [documentation string elided] (interactive "P\nr") (save-excursion (save-restriction (narrow-to-region beg end) (goto-char (point-min)) (sort-subr reverse 'forward-line 'end-of-line))))
The most important thing to note here is the use of the function sort-subr
. The rest of the code is simply boiler plate to limit the sort to the highlighted region. The (interactive "P\nr")
is to read the start and end of the region into the argument values beg
and end
. For our purposes, we can simply treat this as “magic”; the effect of the newline embedded in the string is to separate the reverse
(read with "P"
) from the region (read with "r"
) and that means people can sort in reverse alphabetical order with C-u M-x sort-lines
. The code we’ll develop in here does not have this feature; adding it can be considered an exercise for the reader.
Alright, now we know how sort-lines
works, and we know that we can use sort-subr
to sort by values, the question is how do we do it?
First, and obviously, we read the documentation string with C-h f sort-subr
. This tells us that there are two variables we can make use of,1
STARTKEYFUN moves from the start of the record to the start of the key. It may return either a non-nil value to be used as the key, or else the key is the substring between the values of point after STARTKEYFUN and ENDKEYFUN are called. If STARTKEYFUN is nil, the key starts at the beginning of the record.
and
COMPAREFUN compares the two keys. It is called with two strings and should return true if the first is “less” than the second, just as for `sort’. If nil or omitted, the default function accepts keys that are numbers (compared numerically) or strings (compared lexicographically).
The first thing we note here is the startkeyfun
. It’ll allow us to limit the sort comparison to the value part of the lines. The trick here is to just move the point past the :
(colon). We can do that with search-forward
. Since in my case, and the example here, all the lines do have a colon, we’ll not consider the case where it might be missing in the line, hence we don’t impose any limit on the search (nil
) nor do we care about errors (we allow them with nil
); however, we limit the count to exactly one.
That leaves us with a call that looks like
(search-forward ":" nil nil 1)
and to apply that to sort-subr
we define an entirely new function, my-sort-key-value-lines
. We wrap search-forward
in a lambda for simplicity. Notice that we return nil
explicitly from the lambda, because otherwise sort-subr
will use its return value (the location of point in the buffer after the colon) and sort from that.2
(defun my-sort-key-value-lines (beg end) (interactive "r") (save-excursion (save-restriction (narrow-to-region beg end) (goto-char (point-min)) (sort-subr nil 'forward-line 'end-of-line (lambda () (search-forward ":" nil nil 1) ;; returns point, so we explicitly return nil)))))
And this is the code that results in lexicographical sorting of the values, but we want numeric sorting. There are at least two ways to fix that.
First, we can use the comparison function, to compare both arguments as numbers. We just have to convert the arguments to integers, and then compare them.2
(defun my-sort-key-value-lines (beg end) (interactive "r") (save-excursion (save-restriction (narrow-to-region beg end) (goto-char (point-min)) (sort-subr nil 'forward-line 'end-of-line (lambda () (search-forward ":" nil nil 1) nil) nil (lambda (a b) (< (string-to-number a) (string-to-number b)))))))
In the code above we do that in the second lambda. The nil
between them is the end-of-key
function, which we don’t need to define because it’s the same as the end-of-record
(represented by 'end-of-line
in the above code).
The simpler method, is to do the conversion in the previous lambda, and use the default comparison function. Which results in the third revision.2
(defun my-sort-key-value-lines (beg end) (interactive "r") (save-excursion (save-restriction (narrow-to-region beg end) (goto-char (point-min)) (sort-subr nil 'forward-line 'end-of-line (lambda () (search-forward ":" nil nil 1) (string-to-number (buffer-substring (point) (point-at-eol))))))))
You can now just drop this into your ~/.xemacs/init.el
and use the command M-x my-sort-key-value-lines
to sort key: value
lines, whenever you have the need. And this leaves us with the desired numerical sort order.
bar: 2 baz: 7 foo: 12
1 This code is GPL.
2 This code can be considered WTFPL 2.0; at least the parts inside the lambdas and the rest is just boilerplate.
Compiling Sysbench on OS X Yosemite or Later
These instructions are applicable after cloning the git repository and generating the autoconfigure scripts.
git clone 'https://github.com/akopytov/sysbench.git' sysbench cd sysbench ./autogen.sh
In order to build Sysbench1 with PostgreSQL and MariaDB support, you need to make sure both mysql_config
and pg_config
are in your path.
I use Zsh, so this is my way of doing it, when both Postgres and MariaDB have been installed with MacPorts.
path=( /opt/local/lib/mariadb-10.1/bin /opt/local/lib/postgresql96/bin $path )
Then run
./configure --with-pgsql --with-mysql --prefix=/path/of/your/choice
You are likely to get an error like
ld: library not found for -lgcc_s.10.4
if you do not also
export MACOSX_DEPLOYMENT_TARGET=10.10
before running make
, while building the bundled LuaJit. This is documented in their installation instructions.
Of course, this isn’t taken care of by the wrapper Autotools, nor is there a configure flag to set this.
An alternative might be --with-system-luajit
but that depends on your situation.
Then you finish it off with make install
. Happy benchmarking.
1 I hope I’m linking to the right git repository.
Parsing Command Line Parameters with Yacc & Flex
This is a repost from 2012, but my old blog site disappeared.
Every once in a while someone comes along and asks how to parse command line parameters with Yacc & Flex. This is rather straight forward, but requires some knowledge of the generated code to get right.
Here we present a source template that does this. The user only has to edit the grammar and scanning rules. Some knowledge of C, Yacc and Flex is assumed.
The code is WTFPL licensed.
The template is written for Berkeley Yacc and the Reflex variant of Flex. It may be made to work with GNU Bison and (formerly SourceForge) Flex (now on GitHub), possibly with a few changes.
Table of Contents
Using the Template
In the file commandline.l we start to edit the scanner rules. For our example we make do with
%% // Here we put regular old scanning rules. [a-z]+ { commandlinelval = commandlinetext; return WORD; } %%
The only thing different here is that our customary yylval
and yytext
variables have changed names. The WORD
token is defined in commandline.y.
Then in commandline.y we edit grammar rules as usual. We start with a list of tokens.
// Here we put regular old token declarations. %token WORD SPACE
and then write our grammar
%% // Here we put regular old grammar rules. command: /* empty */ | words ; words: word | words word ; word: WORD { printf( "\"%s\"\n", $1 ); } ; %%
Here we just print out the words returned by the scanner, one per line. We are using the fact that the lexer starts a new lexeme on calls to yywrap()
. This means we do not have to insert any separator characters between the command line arguments we are parsing.
The provided makefile builds the example with the -p prefix parameter to yacc, which changes the symbol prefix from yy and the -P prefix parameter to reflex to do the same. This makes the template usable as-is with projects that use yacc & flex already.
% make yacc -bcommandline -pcommandline -di commandline.y reflex -Pcommandline commandline.l cc -o commandline commandline.tab.c lex.commandline.c
Now we can run the example.
% ./commandline this is a simple example "this" "is" "a" "simple" "example"
Understanding the Template
We use the technique presented previously to pass parameters to yacc and flex [link is to an archived copy] to feed argc
and argv
to our yywrap()
function.
In commandlin.h we declare the argument structure.
// The argument structure we pass to yywrap(). struct arguments { int argc, // The total number of arguments passed to main(). arg; // The argument we are actually going to parse. char **argv; // Pointer to the argument vector itself. };
In commandline.l we have
int nextargument( struct arguments *args ) { // Prevent memory leaks. This is safe because yy_current_buffer // is initialized to zero. if ( YY_CURRENT_BUFFER ) { yy_delete_buffer( YY_CURRENT_BUFFER ); } // If there are no more arguments, return 1 to signal we are done. if ( args->argc == args->arg ) return 1; // Notice we increase args->arg here with ++. commandline_scan_string( args->argv[ args->arg++ ] ); return 0; }
as the yywrap()
function (renamed) which calls yy_scan_string()
for each argument passed to main()
. yy_scan_string()
has been renamed too.
The main()
function itself is purely a template which builds a structure holding argc
and argv
which it then uses to pass on to yywrap()
and yyparse()
.
int main( int argc, char *argv[] ) { // Initialize the argument structure we pass to yywrap(). struct arguments args; args.argc = argc; args.arg = 1; // start at argument 1, not the command name. args.argv = argv; if ( argc > 1 ) // This is actually our yywrap() function. We could also have // used its return value to determine if there is an argument // to parse. nextargument( &args ); else return 1; // We pass the argument structure to our yyparse(). Notice it's // been renamed to "commandlineparse." commandlineparse( (void *) &args ); return 0; }
Here we are careful to call yywrap()
before our first call to yyparse()
to initialize the input buffer.
Depending on the application, there may be no reason to change the main()
function itself, merely rename it and called from the actual main()
.
References
- Passing Parameters to Yacc and Flex [archive.org] (Previous blog entry).
- Berkeley Yacc Website
- Reflex Website
Downloads
Downloads of individual files.
Downloads of the complete source archive.