SCore-S System Development Manual (3)


Developer Defined Functions
#include <score.h>
void _score_mpcrt0( void )
int _score_mpcrt1( void )

#include <score_wsc.h>
void _score_vmfault_handler(int, int, struct sigcontext*, char* );
MessageBoard Color Specification
#include <score_wsc.h>
int _score_msgb_color( void )
Usage Interface
#include <score.h>
void _score_lang_usage( void )
void _score_separate_usage( char *separater )
void _score_print_usage( char *keyword, char *argtype, char *format, ... )
Option DB Interface
#include <score_opt.h>
char *_score_get_opt( char* keyword )
int _score_argv( char*** argvp )
Communication Interface
#include <score.h>
void *_score_get_send_buf( int length )
void _score_send_message( int node, int length )
int _score_peek_network( int flag )
void *_score_recv_message( void )
void _score_arecv_done( int length )
int _score_flush_network( void )

SCore-D Interface
#include <scrd_serv.h>
void _scored_become_idle( void )
void _scored_become_busy( void )
Error Handling
#include <score.h>
void _score_warning( char* format, ... )
void _score_error( char* format, ... )
void _score_panic( char* format, ... )
#include <score.h>
int _score_mtu;
int _score_self_pe;
int _score_pe_count;


This manual page describes how to develop your language or standard communication environment on top of SCore-S. Those who want to develop such systems should define four functions, _score_mpcrt0( void ), _score_mpcrt1( void ), _score_vmfault_handler(int, int, struct sigcontext*, char* ) _score_msgb_color( void ), and _score_lang_usage( void ). The purpose of those functions and library functions provided by SCOre-S for the development are described below.

Control Sequence
When a user program linked with SCore-S library is invoked, the invoked process assigned node number 0 and the main() function in SCore-S library does the followings;

  1. It tries to lock cluster hosts. If it fails then the program execution is aborted.
  2. It setups communication, and spawns processes over specified cluster hosts.
  3. The function _score_mpcrt0() that should be defined by a developer is called. SCore-S assumes that user programs are written in C++, and the function is called before C++ constructors are called.
  4. When returning from _score_mpcrt0(), SCore-S invokes C++ construtors.
  5. Finally, SCore-S calls _score_mpcrt1() defined by a developer.

The spawned processes do the followings;

  1. It setups communication.
  2. The function _score_mpcrt0() is called.
  3. SCore-S invokes C++ construtors.
  4. SCore-S calls _score_mpcrt1().

SCore-S assumes SPMD programming model. The termination of a parallel execution under SCore-S means terminations of all processes. Returning from the _score_mpcrt1() function or invoking exit() systemcall in a process is not enough to terminate the parallel execution. It is the developers' responsibility to terminate all other processes doing the parallel computation. Furhter, when SCore-S/D first detects termination of a process, it tries to kill all others. Like C/C++, the return value of _score_mpcrt1() becomes the exit number. The process that is assigned node number 0 returns the exit code of the first terminated process.

MessageBoard Color
The function _score_msgb_color() should also be define and return one of the color number defined as;

_SCORE_MSGB_PINK (Reserved by OCore)

and the color PURPLE, not defined in score_wsc.h though, but reserved by SCore-D also. The specified color is displayed on a MessageBoard browser, msgb, to let cluster users know what kind of applications are running on which cluster hosts.

Usage and Options
The next and final function which developer should define is _score_lang_usage() function. This function is called when user specifies usage option in -score options to get possible option settings.

void _score_lang_usage( void ) {
  _score_separate_usage( "MPC++/ULT" );
  _score_print_usage( "stacksize", "NN", "maximum stack size" );

This sample code for displaying a language runtime usage produces the following message.

-- MPC++/ULT --
  stacksize=NN  maximum stack size

Since SCore-S applications consists of many software layers, and each layer may have different option settings, _score_separate_usage() is to separate option categories. The function _score_print_usage() displays an option usage in an unified format. The keyword argument is option keyword string, the argtype argument is the type of value, and the format argument and the followings are an explanation format string and its arguments for the option. The possible candidates for the argtype is, AA for an alphanumeric string, NN for a natural number (an integer larger than 0), and II for an integer string. Those argument types are displayed as a legend when usage option is specified.

== Legend ==
  AA:String, NN:Natural Number, II:Integer

The SCore options specified by a user can be obtained using the function _score_get_opt(). It returns a string (char*) associated with the keyword, if any. Like getenv() function to get the value of an environment variable, it returns NULL if the option is not specified by a user, and it returns to a pointer to '\0' if the option is specified without its associating value.

Command argument can be obtained by the function _score_argv(). It returns an integer corresponding to the value of argc. The argument argvp is a pointer to argv that is an array of character string. After returning from the function, argvp now points to the array of the command argument.

Virtual Memory Exception Handling
The function void _score_vmfault_handler() is called when a virtual memory protection exception signal (SIGSEGV or SIGBUS) is caught by SCore-S. For its arguments, refer to sigvec man page. If a library does not care about the exceptional events, simply call _score_panic() (described below).

Communication is ready when the developer-defined function _score_mpcrt0() is called. The number of hosts assigned is in the variable _score_pe_count, and the relative node number where the code is running is in the variable _score_self_pe. The maximum transmission unit, the maxmum length of message for sending and receiving in bytes, is in the variable _score_mtu. No message can be sent or received if length of the message exceeds the limit. All these variables are set in SCore initializing routine.

To send a message, first message buffer region should be alllocated by calling _score_get_send_buf(). The length argument is the length of the message in bytes. It is guaranteed the allocated buffer region is double word aligned, so that it can be casted to an arbitrary type. When the buffer region is ready to send, _score_send_message() function sends the buffer region as a communication message to the destination node specified in node argument. The length argument is the length of the message in bytes. A developer should take care the situation in which function _score_get_send_buf() returns NULL. This means there is no space in send buffer. This case should happen when the destination node is so busy that it cannot consume arrived messages. When this happens, the sender should retrieve received messages. Otherwise a deadlock may occur.

The function _score_peek_network() is to check if one or more messages arrives. It returns the length of the first message arrived in bytes, otherwise it returns 0. The argument flag should be 0 in any case. The function _score_recv_message() retrieves the elderly received message and returns the pointer to the message. The returned pointer is double-word aligned. The function _score_arecv_done() declares the end of message retrieval. Retrieving (accessing) the same message beyond this point is not allowed.

The sequence of calling sending functions, _score_get_send_buf() and _score_send_message(), and the sequence of calling receiving functions, _score_peek_network(), _score_recv_message(), and _score_arecv_done() functions should be kept strictly. Any other sequences result in catastrophe.

The message sending described in this section is asynchronous sending. This means that returning from the function _score_send_message() does not indicate that the sent messages are actually received at each destination node. The function _score_flush_network() returns non-zero, if previously sent messages have been received and pu them in receive buffer at each receiver node. Note that this does not mean that sent messages are actually retrived by the function _score_recv_message() function. This function is very useful when implementing a barrier synchronization.

If user requires just one host, then the communication is NOT initialized, and _score_get_send_buf(), , _score_send_message(), _score_peek_network(), _score_recv_message(), _score_arecv_done(), and _score_flush_network() functions are not ready for use. Further, _score_mtu variable is not initialized.

SCore-D Interface
_scored_become_idle() and _scored_become_busy() functions let SCore-D know running state of a user process. If a user process becomes idle, even if the program is actually waiting in a busy-wait loop, it is preferable to call _scored_become_idle() function, and vice versa. By reporting the status of a user process correctly, SCore-D can detect a global state of a user parallel processes, and SCore-D tries to deschedule if the process is global idle or tries to kill if the process is assumed to be globally terminated, possibly because of deadlock. The timing of message consumption by the _score_recv_message() and the calling of those functions is very important.

void idle_loop( void ) {
  int len;

  while( len = _score_peek_network( 0 ) ) {
    process_the_message( _score_recv_message(), len );

    _score_arecv_done( len );



As in the above example, _scored_become_busy() function must be called before the program consumes the received message by _score_recv_message(). If violated, a user parallel process can be descheduled or killed by SCore-D errorneously.

Error handling
_score_warning(), _score_error() and _score_panic() functions are to output warning or error messages when a runtime library detects an error. _score_warning() function just outputs a formatted message to stderr. _score_error() function prints a formatted error message and terminates program execution with exit(9). _score_panic() function is similar to _score_error(), but it tries to attach a debugger if possible.


The maximum total length of SCore options and command arguments is affected by the value of _score_mtu. This restriction should be relaxed.

For some historical reason, apparently some message receiving functions have unnecessary argument. This interface shoulld be reconsidered.

PM has multicast message sending facility. However, corresponding interface is not documented here.

Systemcall interface to SCore-D is not documented.


Real World Computing Partnership
Parallel Distributed System Software Tsukuba Laboratory

$Id: scores-develop.html,v 1.14 1998/06/02 11:50:41 kameyama Exp $