Reducing complexity and size must be the goal in every step. Niklaus Wirth

Why Smaller is Better

Languages that count several thousands of native library functions have humungus footprints, trashing CPU caches – and inflicting overwhelming learning curves to developers – for the sake of "productivity", we are told.

G-WAN scripts do not lack features (/usr/lib lists thousands of libraries). Libraries let you decide what your application needs. This is why G-WAN aims to offer only core services. Exceptions (in-memory GIF I/O, 2D frame buffer, charts, 'wait-free' Key-Value store, etc.) exist only because no library was found doing the job fast enough (for our needs).

API Categories

The information below is also available in the /gwan/include/gwan.h header file.

The server reply buffer


Get a pointer on the server reply dynamic buffer (used in almost all G-WAN C scripts), or use a pre-allocated buffer for the reply (instead of building a new reply):

// prototypes:
xbuf_t *get_reply(char *argv[]);
void    set_reply(char *argv[], char *buf, u32 len, u32 status);

// examples:
xbuf_t *reply = get_reply(argv); // get the server reply buffer
xbuf_cat(reply, "Hello World"); // build a reply

// use a user-defined allocated (not static) buffer for the server reply
// (usually used to serve entries cached in a G-WAN KV store, see the
// main_404.cxx handler example; see build_headers() also)
void set_reply(argv, my_reply, my_reply_length, 200); // 200:OK (HTTP status)

// tell G-WAN when to run a script again (for the same request)
// type: WK_MS | WK_FD
#define WK_MS 1  // milliseconds
#define WK_FD 2  // file descriptor
void wake_up(char *argv[], int delay_or_fd, int type);

// Setup/update the send() throttling policy with a KiB/second transfer rate.
// This can be done both by servlets and handlers.
// 'kbps1' is the first packets of a reply, 'kbps2' is the speed to use after
// the first KiB of the defined 'kbps1' have been sent.
// Throttling can be defined on a per-connection basis ('global' == FALSE)
// or globally ('global' == TRUE).
// If you are enabling the 'global' flag then the init() function of a
// listener's connection handler is a good place to modify the default
// Throttling policy (none) one time for all (but this global setting
// can also be modified dynamically by handlers and servlets).
// Use 0 to disable throttling (if throttling was previously enabled).
// Examples:
// throttle_reply(argv, 100, 150, TRUE);
// throttle_reply(argv,   0,   0, TRUE);
// ----------------------------------------------------------------------------
// because of the delay between the timeout loop and the actual processing of
// the event triggered by a write() call, the actual rate is only half of what
// is requested - that' s why we double the values below
void throttle_reply(char *argv[], u16 kbps1, u16 kbps2, int global);

// return the context of the specified file descriptor 'fd2'.
// 'fd2' may be used for relaying or to query a backend server, in which case
// your script may want to peer it to the client connection accepted by G-WAN.
// So, if you specify a non-zero 'fd1' value:
//  - 'fd1' is the remote client connection accepted by G-WAN (if any).
//  - 'fd2' is the connection created by G-WAN to reach a backend server.
// if the 'fd1' is irrelevant (no backend server was used) then set it to zero,
// this call will return the (unpeered) 'fd2' connection context.
// (this function is aimed at being used by G-WAN handlers)
void *get_fd_ctx(int fd2, int fd1);

Dynamic xbuffers


Dynamic buffers grow dynamically to let you build ASCII or binary contents by using the standard printf() syntax – without having to care about memory allocation.

xbuffers are used in almost all G-WAN C scripts, notably to build the server reply:

typedef struct
   char *ptr;       // data buffer
   u32   allocated; // memory allocated
   u32   len;       // memory used
   u32   growby;    // memory allocation increment
} xbuf_t;

// load a file into the buffer
void  xbuf_frfile  (xbuf_t *ctx, char *szFile);

// save the buffer in a file
int  xbuf_tofile  (xbuf_t *ctx, char *szFile);

// allocate more memory to match the requested size
u32   xbuf_growto  (xbuf_t *ctx, u32 len);

// reset the buffer contents and length (but keep memory allocated)
// ctx->len = 0; // that's what it does
// if(ctx->ptr)
//   *ctx->ptr = 0; // in case that's a string, keep it working
void  xbuf_empty   (xbuf_t *ctx);

// return the end of the buffer
// return(ctx->ptr + ctx->len); // that's what it does
char *xbuf_getend  (xbuf_t *ctx);

// attach ctx to the provided buffer
void  xbuf_attach  (xbuf_t *ctx, char *ptr, s32 size, s32 len);

// detach ctx from any previously attached buffer
char *xbuf_detach  (xbuf_t *ctx);

// release the memory allocated for the buffer
void  xbuf_free    (xbuf_t *ctx);

// clear the contents of the buffer
void  xbuf_clear   (xbuf_t *ctx);

// must be called after xbuf_t buf; has been declared to initialize struct:
// ctx->ptr       = NULL; // that's what it does
// ctx->len       = 0;
// ctx->allocated = 0;
// ctx->growby    = PAGE_SIZE;
void  xbuf_init   (xbuf_t *ctx);
void  xbuf_reset  (xbuf_t *ctx); // alias, for compatibility with old code

// copy 'srclen' bytes from 'src' into the buffer
char *xbuf_ncat    (xbuf_t *ctx, char *src, s32 srclen);

// copy the string 'str' into the buffer
char *xbuf_cat     (xbuf_t *ctx, char *str);

// format 'a la sprintf()' into the buffer
char *xbuf_xcat    (xbuf_t *ctx, char *src, ...);

typedef struct
   char *ptr;       // data (can be binary)
   u32   len;       // data length
} strtab_t;

// concatenate 'n' buffers 'a la writev()' into the buffer
char *xbuf_vcat    (xbuf_t *ctx, const strtab_t *array, int nbr);

// sort text entries separated by 'separator' in the buffer
void  xbuf_sort    (xbuf_t *ctx, char separator, s32 remove_duplicates);

// find the string 'str' in the buffer
char *xbuf_findstr (xbuf_t *ctx, char *str);

// replace the first occurence of the 'old' string by the 'new' string in the buffer
char *xbuf_repl    (xbuf_t *ctx, char *old, char *new);

// same as above but using a range in the buffer
char *xbuf_replfrto(xbuf_t *ctx, char *beg, char *end, char *old, char *new);

// truncate buffer at byte position using pointer 'ptr'
void  xbuf_truncptr(xbuf_t *ctx, char *ptr);

// truncate buffer at byte position using length 'len'
void  xbuf_trunclen(xbuf_t *ctx, s32 len);

// copy up to 'dstlen'-1 bytes of the first available line into 'dst' buffer
// (a line is any number of bytes followed by \n or \r\n)
// return number of bytes read
// return -1 if no complete lines are available
long  xbuf_getln   (xbuf_t *ctx, char *dst, s32 dstlen);

// move up to 'dstlen' bytes into 'dst' from the top of the buffer
// return number of bytes moved
// return 0 if no data is available
long  xbuf_pull    (xbuf_t *ctx, char *dst, s32 dstlen);

// move 'len' bytes to 'bytes' from the position 'pos' in the buffer
void  xbuf_delete  (xbuf_t *ctx, char *pos, s32 len, char *bytes);

// insert 'len' bytes from 'bytes' into the buffer at position 'pos'
long  xbuf_insert  (xbuf_t *ctx, char *pos, s32 len, char *bytes);

// send an HTTP request to a server, save the reply and return the HTTP status
// ('headers' is adding headers, not overwriting any existing header)
// 'headers' can be used for POST URI parameters (here with SSL):
// xbuf_frurl(xbuf, "", 443, HTTPS | HTTP_POST, "/?form", 1000, "&arg=123");
long  xbuf_frurl   (xbuf_t *ctx, char *host, u32 port, u32 method, char *uri,
                    u32 mstimeout, char *headers);

The error.log file


Output text in the current virtual host 'error.log' file:

// prototype:
void log_err(char *argv[], const char *msg);
// example:
char str[256];
u64 nbr_records = 1234567890123;
s_snprintf(str, sizeof(str)-1, "records: %llu", nbr_records);
log_err(argv, str);

Environment Variables


These enums allow you to get and modify G-WAN internal values like the traditional CGI environment variables, but also performance counters and even internal structures:

// get an environment variable (a performance counter, or a persistence ptr)
// (see the 'served_from.c' example and 'enum HTTP_Env' below for all values)
// 'inval' is only used to get a *pointer* on a value (so we can change the

// prototype:
u64 get_env(char *argv[], int name);

// example #1: get a value
int session = (int)get_env(argv, SESSION_ID);
char *www   = (char*)get_env(argv, WWW_ROOT);

// example #2: get a pointer on a value
int *pHttp_code = (int*)get_env(argv, HTTP_CODE);
  *pHttp_code = 200;

// example #3: get a pointer on a pointer (to READ the pointer value)
// (see the /csp/persistence.c example)
void *pVhost_persistent_ptr = (void*)get_env(argv, US_VHOST_DATA);
   printf("%.4s\n", pVhost_persistent_ptr);

// example #4: get a pointer on a pointer (to CHANGE the pointer value)
// (see the /csp/persistence.c example)
void **pVhost_persistent_ptr = (void*)get_env(argv, US_VHOST_DATA);
   *pVhost_persistent_ptr = strdup("persistent data");
enum HTTP_codes
  RC_CLOSE = 0,        // close connection
  RC_NOHEADERS,        // prevent G-WAN from injecting HTTP headers
                       // [100-600] return codes are for HTTP status codes
  RC_STREAMING = 1000  // call script again after partial reply was sent
  RC_NOCACHE   = 2000, // prevent G-WAN from caching servlet output

// see the 'PONG.c' protocol handler example
enum PROTOCOL_state
   PRT_CONNECTED = 1, // a new connection was established
   PRT_ACCEPTED,      // a new connection was accepted
   PRT_CAN_READ,      // data can be fetched from the input socket buffer
   PRT_CAN_WRITE,     // data can be added to the output socket buffer
   PRT_CLOSED        // the connection was closed

enum HTTP_Method
    HTTP_ANY=0, HTTP_GET, // RFC 2616
    // G-WAN currently supports this list until 'PATCH'
    HTTP_PROPFIND,  // RFC 2518: WebDAV
    HTTP_INVALID    // RFC 3253: WebDAV versioning

// letting you translate HTTP method codes into character strings (tracing)
static char *szHTTP_Method[] =
   [HTTP_ANY]        = "?",
   [HTTP_GET]        = "GET",
   [HTTP_HEAD]       = "HEAD",
   [HTTP_POST]       = "POST",
   [HTTP_PUT]        = "PUT",
   [HTTP_DELETE]     = "DELETE",
   [HTTP_PATCH]      = "PATCH",
   // G-WAN currently supports this list until 'PATCH'
   [HTTP_TRACE]      = "TRACE",
   [HTTP_PROPFIND]   = "PROPFIND",  // RFC 2518: WebDAV
   [HTTP_MKCOL]      = "MKCOL",
   [HTTP_COPY]       = "COPY",
   [HTTP_MOVE]       = "MOVE",
   [HTTP_LOCK]       = "LOCK",
   [HTTP_UNLOCK]     = "UNLOCK",
   [HTTP_UPDATE]     = "UPDATE",
   [HTTP_LABEL]      = "LABEL",
   [HTTP_REPORT]     = "REPORT",
   [HTTP_MERGE]      = "MERGE",
   [HTTP_INVALID]    = "INVALID"    // RFC 3253: WebDAV versioning

enum HTTP_Type
   TYPE_IDENTITY,   // plain text (default)
   TYPE_URLENCODED, // "&name=toto"
   TYPE_MULTIPART,  // headers + base64

enum ENC_Type
   ENC_IDENTITY=0, // default

enum AUTH_Type
   AUTH_SRP=4, // DH-like authentication and key exchange (shared secret)
   AUTH_x509=5 // can be used on the top of AUTH_BASIC/AUTH_DIGEST/AUTH_SRP

enum LANGUAGES // used by get_env(argv, DEFAULT_LANG);

// the HTTP state of a connection
// (see the served_from.c example for how to use it)
typedef struct
   off_t    h_range_from,
   u32      session_id;              // csp: only used by get_env() at the moment
   int      file_fd;                 // 32-bit as it is tested for -1 equality
   u32      h_if_modified,           // EPOCH time (in seconds)
   u32      h_entity            :12, // 0-4096 (room for many HTTP headers,
            h_host              :12, // G-WAN rejects requests with too many)
            h_useragent         :12,
            h_referer           :12, // "en-us,en;q=0.5"
            h_accept_language   :12,
            h_cookies           :12,
            h_if_nonematch      :12, // If-None-Match: "68689-7696a7c-876b7e" (ETag)
            h_auth_type         : 3, // 0-7
            h_keepalive         : 1, // 0-1
            h_accept_encoding   : 4, // 0-15
            h_content_encoding  : 4, // 0-15
            h_transfer_encoding : 4, // 0-15
            h_port              :16, // 0-65,535 (server port)
            h_content_type      : 2, // 0-3
            h_expect            : 1, // 0-1
            h_maj_ver           : 1, // 0-1
            h_min_ver           : 4, // 0-15
            h_ver               :10, // 0-1023
            h_method            : 5, // 0-32 (0-27 to cover the HTTP standard)
            pipelined           : 1, // 0-1  (another request follows)
            h_do_not_track      : 1; // 0-1  (optional W3C "DNT:" header)
   char    *h_auth_user, *h_auth_pwd;
#ifdef _USE_SSL
   ssl_ctx *ssl;
} http_t;

enum HTTP_Env
   // -------------------------------------------------------------------------
   // Server 'environment' variables used by the get_env() API call
   // -------------------------------------------------------------------------
   // NOTE: QUERY_CHAR and DEFAULT_LANG are global settings (server-wide)
   REQUEST=0,       // char  *REQUEST;        // "GET / HTTP/1.1\r\n..."
   REQUEST_LEN,     // int    REQUEST_LEN     // strlen(REQUEST); with headers
   QUERY_STRING,    // char  *QUERY_STRING    // request URL after first '?'
   FRAGMENT_ID,     // char  *FRAGMENT_ID     // request URL after last '#'
   REQ_ENTITY,      // char  *ENTITY          // "arg=x&arg=y..."
   CONTENT_TYPE,    // int    CONTENT_TYPE;   // 1="x-www-form-urlencoded"
   CONTENT_LENGTH,  // int    CONTENT_LENGTH  // body length provided by client
   CONTENT_ENCODING,// int    CONTENT_ENCODING// entity, gzip, deflate
   SESSION_ID,      // int    SESSION_ID;     // 12345678 (range: 0-4294967295)
   HTTP_CODE,       // int   *HTTP_CODE;      // 100-600 range (200:'OK')
   HTTP_HEADERS,    // struct *http_t;        // see struct http_t above
   AUTH_TYPE,       // int    AUTH_TYPE;      // see enum AUTH_Type {}
   REMOTE_ADDR,     // char  *REMOTE_ADDR;    // ""
   REMOTE_BIN_ADDR, // u64    REMOTE_BIN_ADDR;// u64 ip = numeric_ip_address;
   REMOTE_PORT,     // int    REMOTE_PORT;    // 1460 (range: 1024-65535)   
   REMOTE_PROTOCOL, // int    REMOTE_PROTOCOL // ((HTTP_major*1000)+HTTP_minor)
   REMOTE_USER,     // char  *REMOTE_USER     // "Pierre"
   REMOTE_PWD,      // char  *REMOTE_PWD      // "secret"
   CLIENT_SOCKET,   // int    CLIENT_SOCKET   // 1032 (-1 if invalid/closed)
   USER_AGENT,      // char  *USER_AGENT;     // "Mozilla ... Firefox"
   SERVER_NAME,     // char  *SERVER_NAME;    // ""
   SERVER_ADDR,     // char  *SERVER_ADDR;    // ""
   SERVER_PORT,     // int    SERVER_PORT;    // 80 (443, 8080, etc.)
   SERVER_DATE,     // char  *SERVER_DATE;    // "Tue, 06 Jan 2009 06:12:20 GMT"
   SERVER_PROTOCOL, // int    SERVER_PROTOCOL // ((HTTP_major*1000)+HTTP_minor)
   VHOST_ROOT,      // char  *VHOST_ROOT;     // the (virtual) host root folder
   WWW_ROOT,        // char  *WWW_ROOT;       // the HTML pages root folder
   CSP_ROOT,        // char  *CSP_ROOT;       // the CSP .C files folder
   LOG_ROOT,        // char  *LOG_ROOT;       // the log files folder
   HLD_ROOT,        // char  *HLD_ROOT;       // the handlers folder
   FNT_ROOT,        // char  *FNT_ROOT;       // the fonts folder
   DOWNLOAD_SPEED,  // int   *DOWNLOAD_SPEED; // min CLIENT READ rate (default:1)
   READ_XBUF,       // xbuf_t*READ_XBUF;      // HTTP request(s) are stored there
   HEAD_XBUF,       // xbuf_t*HEAD_XBUF;      // HTTP response headers, if any
   SCRIPT_TMO,      // u32   *SCRIPT_TMO;     // time-out in milliseconds
   KALIVE_TMO,      // u32   *KALIVE_TMO;     // HTTP Keep-Alive time-out (ms)
   REQUEST_TMO,     // u32   *REQUEST_TMO;    // time-out in milliseconds
   MIN_SEND_SPEED,  // u32   *MIN_SEND_SPEED; // min client SEND speed, bytes/sec
   NBR_CPUS,        // int    NBR_CPUS;       // total of available CPUs
   NBR_CORES,       // int    NBR_CORES;      // total of available CPU Cores
   NBR_WORKERS,     // int    NBR_WORKERS;    // total of server workers
   CUR_WORKER,      // int    CUR_WORKER;     // worker thread number: 1,2,3...
   REPLY_MIME_TYPE, // char  *REPLY_MIME_TYPE;// set script's reply MIME type
   DEFAULT_LANG,    // u8     DEFAULT_LANG;   // LG_D: /?hello.d => /?hello
   QUERY_CHAR,      // u8     QUERY_CHAR;     // replace '?' by [ -_.!~*'() ]
   REQUEST_TIME,    // u64    REQUEST_TIME;   // time (parse+build) in ms
   MAX_ENTITY_SIZE, // u32   *MAX_ENTITY_SIZE;// maximum POST entity size
   USE_WWW_CACHE,   // u8    *USE_WWW_CACHE;  // 0:disabled (default) 1:enabled
   USE_CSP_CACHE,   // u8    *USE_CSP_CACHE;  // 0:disabled (default) 1:enabled
   CACHE_ALL_WWW,   // u8    *CACHE_ALL_WWW;  // 1:cache all /www at startup
   USE_MINIFYING,   // u8    *USE_MINIFYING;  // '1' by default (JS/CSS/HTML)
   // -------------------------------------------------------------------------
   // Server performance counters
   // -------------------------------------------------------------------------
   // -------------------------------------------------------------------------
   // Handler and VirtualHost Persistence pointers
   // -------------------------------------------------------------------------
   US_REQUEST_DATA = 200, // Request-wide pointer
   US_HANDLER_DATA,       // Listener-wide pointer
   US_VHOST_DATA,         // Virtual-Host-wide pointer
   US_SERVER_DATA,        // global pointer (for maintenance script)
   US_HANDLER_STATES      // states registered to get server-state notifications
   US_HANDLER_CTX_SIZE    // size of "Protocol Handler" per-connection context

HTTP Response Headers


To modify G-WAN's generated HTTP Headers, or create ones from scratch:

// *only* for STATIC requests used by "content-type" handlers
// prototype:
void http_header(u32 flags, char *buf, u32 buflen, char *argv[]);

// example:
char header[] = "Powered-by: ANSI C scripts\r\n";
http_header(HEAD_ADD, header, sizeof(header) - 1, argv);

enum HEADERS_flags
   HEAD_ADD   = 1, // add this HTTP header to response headers
   HEAD_MOD   = 2, // not implemented yet
   HEAD_DEL   = 4, // not implemented yet
   HEAD_AFTER = 8  // add this data chunck just after [HTTP headers CRLFCRLF]

// connection context structure (do not modify it)
typedef struct { u8  str[12]; u32 num; } ip_t;
typedef struct
   ip_t   client_ip;    // "a.b.c.d" + IP address as u32
   int    client_fd;    // client socket we accepted from client
   u32    id       :20, // 0- 1m+, client_fd initial value, even after close()
          state    :12, // 0- 4095, connection state
          reserved :32; // do not modify!
   void  *ctx;          // user-defined protocol-handler context
} conn_t;

// prototype:
void build_headers(char *argv[], char *format, ...);

// example:
// build HTTP response headers (usually used with set_reply(), see the
// main_404.cxx handler example)
char *date = get_env(argv, SERVER_DATE);
char szmodified[32];
static const char buf[] =
   "HTTP/1.1 %s\r\n"
   "Date: %s\r\n"
   "Last-Modified: %s\r\n"
   "Content-type: text/html\r\n"
   "Content-Length: %u\r\n" // HTML body length
   "Connection: keep-alive\r\n\r\n";

build_headers(argv, buf,
          http_status(*pHTTP_status), // "200 OK" here
          date,                       // current HTTP time
          time2rfc(mod, szmodified),  // file HTTP time
          len);                       // file length
set_reply(argv, c, len, *pHTTP_status); // re-use cached buffer

HTTP Code Messages


Given an HTTP status code, return the status message ("200 OK", "404 Not found", etc.) or the long litteral description aimed at humans (like "The requested URL was not found on this server"):

// prototypes:
char *http_status(int code);
char *http_error (int code);
// examples:
char *hdr = http_status(404); // "404 Not Found"
char *msg = http_error(404); // "The requested URL was not found on this server"

URL parameters


Let C scripts fetch the value of a specified URL parameter:

// get an URL parameter: ""

// prototype:
void get_arg(char*name, char **value, int argc, char *argv[]);

// example:
char *name = 0; get_arg("name=", &name, argc, argv);
// (now, 'name' points to "Eva" - you MUST check if 'name' is NULL)
// you can also walk main()'s argv[] values directly:
int i = 0;
while(i < argc)
   xbuf_xcat(reply, "argv[%u] '%s'<br>", i, argv[i]);

Server Report


Dumps an ASCII or HTML server report with information like system and server uptimes, levels of disk and RAM, traffic statistics, etc.

// append a server report (in html or text format) to an xbuffer)
// most of this data can be extracted from the performance counters above but
// this is a convenient way to get a 'snapshot' of the server state
void server_report(xbuf_t *reply, int html); // see the report.c example

Run Command


Let a G-WAN script run an external command like ping and get a file descriptor to read() the command's output. GLIBC's popen() does not fit the task here because it buffers the command output.

// see the stream3.c example
int run_cmd(const char *cmd, int argc, char *argv[]);

Key-Value store


The G-WAN Key-Value store has been used under the highest concurrency issues since it is used by G-WAN for many internal lists.

This KV is much faster than Tokyo Cabinet FIXED (an array that can only process fixed-size keys and values) – and, unlike Tokyo Cabinet, G-WAN's KV store scales linearly under multi-threaded read and write tests:

// example: (for more details, see the kv.c example)
// ----------------------------------------------------------------------------
kv_item item;
item.key = strdup("Paula");
item.val = strdup("Accounting");
item.klen = sizeof("Paula");

kv_t store;
kv_init(&store, 4 * 1024, argv); // using 4 KB "maximum"
kv_add(&store, &item, argv);

char *ptr = kv_get(&store, "Paula", 0);
   printf("value: %s\n", ptr);

kv_del(&store, "Paula");
kv_free(&store); // makes the above kv_del() redundant in this example
// ----------------------------------------------------------------------------
// the kv_init() flag options
   KV_REFCOUNT    =    1, // reference count, incremented by kv_get()
   KV_PERSISTANCE =    2, // periodic file I/O (using kv_recfn() callback)
   KV_INCR_KEY    =    4, // 1st field:primary key (automatically incremented)
   KV_CUR_TIME    =    8, // 2nd field:time stamp  (automatically setup/updated)
   KV_NO_UPDATE   =   16, // kv_add() will fail to update an existing entry
                          // (can't be used in kv_init()'s flags)
   KV_SHARED      =   32, // available outside of G-WAN
   KV_DISTRIBUTED =   64, // relying on a distributed topology
   KV_PREFIX      =  128, // kv_get() will return best entry (not implemented)
   KV_SIMILAR     =  256, // kv_get() will return best entry (not implemented)
   KV_NEXT        =  512, // kv_get() will return best entry (not implemented)
   KV_PREV        = 1024  // kv_get() will return best entry (not implemented)

// a key-value store
typedef struct
   char  name[12];  // kv name (optional)
   u32   flags;     // kv flags (optional)
   long  root;      // kv storage
   long  nbr_items; // nbr of items in kv (atomically maintained)
   long  lock;      // global kv lock (not used by G-WAN, only for users)
   void *delfn;     // kv_del() callback (to free memory of custom records)
   void *recfn;     // persistence callback (to format records saved to disk)
   void *pool;      // if specified, used for memory allocations
} kv_t;

// a tuple (key-value, and key-value lengths)
// if(!klen) then kv_add()/kv_get()/kv_del()/kv_do() do klen = strlen(klen);
typedef struct
   char *key, *val; // value length limit: available memory
   union {
   u32   flags;   // kv_add() flags: set to zero for compatibility
   u32   in_use;  // if(KV_REFCOUNT) ref_count atomically incr. by kv_get()
   u32   klen;    // key length limit: 4 GB
} kv_item;

// delfn is an user-defined function to free memory allocated for KV records
typedef void(*kv_delfn_t)(void *value);

// recfn is an user-defined function to format KV records when saved to disk
typedef void(*kv_recfn_t)(void *value);

// create a KV store (all arguments can be NULL but 'store')
void kv_init(kv_t *store, char *name, long max_nbr_items, u32 flags,
             kv_delfn_t delfn, kv_recfn_t recfn);

// add/update a value associated to a key
// return: 0:out of memory, else:pointer on existing/inserted kv_item struct
kv_item *kv_add(kv_t *store, kv_item *item);

// search a 'value' using a 'key'
// return: 0:not found, else:pointer on the value we found
char *kv_get(kv_t *store, const char *key, int klen);

// delete a 'value' using a 'key'
// return: 0:failure (value not found), 1:success
int kv_del(kv_t *store, const char *key, int klen);

// free all the keys and values
void kv_free(kv_t *store);

// run 'proc(kval_node, ctx)' on the specified subset of entries
// if 'key' is NULL then we visit all the entries,
// else we only visit the entries that start with the 'key' string
// (entries are visited in ASCII/binary order)
// if(!klen) then kv_do() does klen = strlen(klen);
// return:
//  1: success: visited all matching entries
//  2: no entry starts with the 'key' string
// It must return 1 to continue searching (any other value stops the search)
// You can use the 'user_defined_ctx' context to store a structure for
// counters, data collection, etc.
typedef int(*kv_proc_t)(const kv_item *item, const void *user_defined_ctx);
int kv_do(kv_t *store, const char *key, int klen, kv_proc_t kv_proc,
          void *user_defined_ctx);

Garbage collector


These are malloc()/free() replacements aimed at making it easier to work with ephemeral allocations. Note that this is non-persistent memory used for the HTTP request cycle (all allocations are lost after a request is processed so it is not suited to share data between connections).

For each new HTTP request, gc_init() must be called prior other gc_xxx() calls (subsequent gc_init() calls are ignored) to define how much memory you will use. gc_malloc() will return NULL if you did not call gc_init() first.

gc_free() is functional (really freeing memory) so you should call it in long loops that allocate temporary memory (for this use, also consider alloca() which works on the stack).

As a bonus, gc_free() does not crash with double-free calls nor with bad pointers (proof that C can be a friendly language).

// If you need to make sure that all the memory allocated by your C script
// is automatically freed when the script (either a handler or a servlet)
// has finished with a request, then these calls are for you:
int   gc_init  (char *argv[], size_t max_size); // called once to reserve memory
void *gc_malloc(char *argv[], size_t size);     // works like malloc()
void  gc_free  (char *argv[], void *ptr);       // useful in a loop

Memory Pools


These malloc()/free() replacements work with persistent allocations that can be shared by connections and threads. But as opposed to libc's allocator, you can decide when to free a memory pool without having to free all the small allocations made in the past for that pool (say good-bye to memory leaks).

Before using a memory pool, call mp_init() to define how much memory you will use. mp_malloc() will return NULL if you did not call mp_init() first.

mp_del() will free the specified memory pool, freeing all attached allocations made with mp_malloc().

As a bonus, mp_free() does not crash with double-free calls nor with bad pointers (proof that C can be a friendly language).

// If you want to be able to free all the memory allocated by your C script
// at once without having to free all the allocated blocks one by one then 
// these calls are for you:
int  *mp_init  (size_t max_size);         // reserve memory
void *mp_malloc(void *pool, size_t size); // works like malloc()
void  mp_free  (void *pool, void *ptr);   // works like free()
void  mp_del   (void *pool);              // free all memory

Handler States


Define which handler states we want to be notified in the handler's main():

   HDL_INIT = 0,
   HDL_AFTER_ACCEPT, // just after accept (only client IP address setup)
   HDL_AFTER_READ,   // each time a read was done until HTTP request OK
   HDL_BEFORE_PARSE, // HTTP verb/URI validated but HTTP headers are not
   HDL_AFTER_PARSE,  // HTTP headers validated, ready to build reply
   HDL_BEFORE_WRITE, // after a reply was built, but before it is sent
   HDL_AFTER_WRITE,  // after a reply was sent
   HDL_HTTP_ERRORS,  // when G-WAN is going to reply with an HTTP error
   HDL_BEFORE_CLOSE, // when G-WAN is going to close a connection

// example:
int init(int argc, char *argv[])
   // get the Handler states that will be notified
   // a pointer on the states integer
   u32 *states = get_env(argv, US_HANDLER_STATES);

   // setup the Handler states that we want to be notified
   *states = (1 << HDL_AFTER_ACCEPT)
           | (1 << HDL_BEFORE_PARSE)
           | (1 << HDL_HTTP_ERRORS); // only those states
   return 0; // >= 0:success



Lets you add, get or delete entries in the G-WAN cache. Note that since G-WAN v2.8+ the KV store can store and serve directly any buffer (without copy, see the set_reply() call), making it even more efficient to build your own caches:

// create/update a cache entry ('file' MUST be imaginary if 'buf' is not NULL)

// examples:
cacheadd(argv, "counter.html", buf, 1024, ".html", 200, 60); // expire:60sec
cacheadd(argv, "archives/doc_1.pdf", 0, 0, ".pdf", 200, 0); // never expire
cachedel(argv, "tool/counter");

// prototypes:
// ('file' MUST exist if 'buf' is NULL)
// if(expire == 0) never expires
// if(expire >  0) expires in 'expires' seconds
// 'code' is the HTTP status code that the server will send to clients
// return 0:failure, !=0:success
// see the cache.c example.

long cacheadd(char*argv[], char *file, char *buf, u32 buflen, char *mime,
              u32 code, u32 expire);

// delete a cached entry
void cachedel(char*argv[], char *file);

// search a cached entry (and, if found, return the requested details)
// examples:
//    u32 len = 0, code = 0, date = 0, exp = 0;
//    char *entry = cacheget(argv, "tool/counter", &len, 0, &code, &date, &exp);
//    char *entry = cacheget(argv, "tool/counter", &len, 0, 0, 0, 0);
//    char *entry = cacheget(argv, "tool/counter", 0, 0, 0, 0, 0);
// return 0:not found, else pointer on cached entry
char *cacheget(char*argv[], char *uri, u32 *buflen, char **mime, u32 *code,
               u32 *modified, u32 *expire);

Comet Streaming


Pushing data to clients without polling (and reducing the number of connections)

// see the comet.c example for how to setup a feed and to add subscribers
typedef int (*make_data_t)(char *argv[]);
typedef int (*push_data_t)(char *argv[], xbuf_t *reply);
typedef void(*free_data_t)(char *argv[]);
int push_list_add(char *argv[], char *feed_name,
                  make_data_t make_fn, u32 make_freq,
                  push_data_t push_fn, u32 push_freq,
                  free_data_t free_fn);

JSON (de-)serialization


Making JSON easier (and much faster) to use:

// NOTE: numbers are stored as 'double', don't forget to *cast* in xbuf_xcat()
// see json.c for an extensive example (it uses all the functions below)
   jsn_FALSE = 0, jsn_TRUE, jsn_NULL, jsn_NUMBER, jsn_STRING,
   jsn_NODE, jsn_ARRAY

typedef struct jsn_s
   struct jsn_s *prev;    // node's prev item (parent if node is 1st child)
   struct jsn_s *next;    // node's next item (list ends with NULL)
   struct jsn_s *child;   // node's child node (NULL if none)
   char         *name;    // node's name
   int           type;    // node's value type (see JSN_TYPE above)
   union {
   char         *string;  // value 'type' == jsn_STRING
   s64           integer; // value 'type' == jsn_INTEGER
   double        real;    // value 'type' == jsn_REAL
   u64           x;       // context
   long          y;       // context
} jsn_t;

// take JSON text as input and return a jsn_t tree
// (call jsn_free() when you are done with the jsn_t tree)
jsn_t *jsn_frtext(char*text, char *name);

// append and format a jsn_t object into text in the specified dynamic buffer
// if(!formated) then a compact formating is used (no separator, CRLF)
// (call xbuf_free(text) when you are done with the text)
char *jsn_totext(xbuf_t *text, jsn_t *node, int formated);

// return a node's item[i] or NULL if item[i] does not exist
jsn_t *jsn_byindex(jsn_t *node, int i);

// search for 'name' in all same-level items; if(deep) in children
// (case insensitive search)
jsn_t *jsn_byname(jsn_t *node, char *name, int deep);

// search for 'value' of 'type' in all same-level items; if(deep) in children
// (case insensitive search)
jsn_t *jsn_byvalue(jsn_t *node, int type, double value, int deep);

// add an 'item' or a 'node' to the specified node
jsn_t *jsn_add(jsn_t *node, char *name, int type, double value);

// modern compilers no longer allow double to void* casts...
jsn_t *jsn_add_real(jsn_t *node, char *name, double value);

// update an 'item' or a 'node'
jsn_t *jsn_updt(jsn_t *node, double value);

// remove an 'item' or a 'node' (and all its nodes/items)
void jsn_del(jsn_t *node);

// free all memory and delete a jsn_t node and all its nodes and items
void jsn_free(jsn_t *node);

// helpers for code clarity
#define jsn_add_null(node, name)       jsn_add(node, name, jsn_NULL,    0)
#define jsn_add_false(node, name)      jsn_add(node, name, jsn_FALSE,   0)
#define jsn_add_true(node, name)       jsn_add(node, name, jsn_TRUE,    0)
#define jsn_add_bool(node, name, n)    jsn_add(node, name, (n != 0),    0)
#define jsn_add_string(node, name, s)  jsn_add(node, name, jsn_STRING, (long)s)
#define jsn_add_node(node, name)       jsn_add(node, name, jsn_NODE,    0)
#define jsn_add_array(node, name, n)   jsn_add(node, name, jsn_ARRAY,   n)
#define jsn_add_integer(node, name, n) jsn_add(node, name, jsn_INTEGER, n)

HTML escaping


Basic HTML and URL processing routines:

u32  url_encode   (u8 *dst, u8 *src, u32 maxdstlen);  // return len
u32  escape_html  (u8 *dst, u8 *src, u32 maxdstlen);  // return len
u32  unescape_html(u8 *str);                          // inplace, return len
int  html2txt     (u8 *html, u8 *text, int maxtxlen); // return len



The standard snprintf() and vsnprintf() calls, just with more features:

// extended sprintf(): (these extensions are also used by xbuf_xcat())
// "%F","%D","%I","%U" - pretty thousands (the ' formatter is also supported)
// "%b"                - binary (8 => "1000")
// "%B","%-B"          - base 64 encode/decode ("%12B" encode a binary buffer)
// "%3C"               - generate a string of length n ("%3C", 'A' => "AAA")
// "%k"                - KB, MB, GB, etc. (1024 => "1 KB")
// "%m"                - strerror_r() system error messages (errno)
int s_snprintf (char*str, size_t len, const char *fmt, ...);
int s_vsnprintf(char*str, size_t len, const char *fmt, va_list a);

In-memory GIF I/O


An ultra-fast GIF generator and parser; to save a GIF image on disk, just save the buffer made by gif_build():

// build an in-memory GIF image from a raw 'bitmap'
// params: buffer      - destination buffer (must be pre-allocated)
//         bitmap      - input pixels (8-bit pixels)
//         width       - image width
//         height      - image height
//         palette     - color palette (3 * 256 = 768 bytes maximum)
//         nbrcolors   - number of entries in palette (256 maximum)
//         transparent - index of transparent colour (-1: no transparency)
//         comment     - a pointer on a text comment (0:none)
// return: length of GIF image, -1 if failure (see the fractal.c example)

int gif_build(u8 *gif, u8 *bitmap, u32 width, u32 height, u8 *palette,
              u32 nbcolors, int transparency, u8 *comment);

// split an in-memory GIF (loaded with xbuf_frfile()?) into its components
// params: buf         - buffer to parse
//         buflen      - the size in bytes of the input buffer
//         width       - returned image width
//         height      - returned image height
//         palette     - pre-allocated palette (3 * 256 = 768 bytes maximum)
//         nbcolors    - returned nbr of colors
//         transparent - returned color transparent index (-1 if none)
//         comment     - returned allocated GIF comment   ( 0 if none)
// return: pointer on newly allocated bitmap (0 on failure)
// notes:  if 'comment' different from null, you MUST free(comment); but you
//         can pass a null to gif_parse() to say that you don't want comments

u8 *gif_parse(u8 *buf, u32 buflen, u32 *width, u32 *height, u8 *palette,
              u32 *nbcolors, int *transparent, u8 **comment);
// see the fractal.c, chart.c or data_uri.c samples

Frame buffer


2D routines to draw in 8-bit memory frame buffers (that can then be encoded as GIF images):

typedef struct { u8  r,g,b; } rgb_t;
typedef struct { u32 x,y,X,Y; } rect_t;

typedef struct
  u8    *bmp,       // bitmap pixels
        *p;         // current cursor position, as a pointer
  int    bbp,       // bits per pixel
         pen, bgd;  // current drawing color index / background color index
  rect_t rect;      // used for clipping, windows, etc.
  u32    flags,     // alignment, type of chart, whatever you need
         w, h,      // bitmap width and height
         x, y;      // current cursor position, as row/column coordinates
} bmp_t;            // (used when the 'p' pointer above is null)

   V_TOP_ALIGN = 1 << 0, // vertical
   V_CEN_ALIGN = 1 << 1,
   V_BOT_ALIGN = 1 << 2,
   H_LEF_ALIGN = 1 << 3, // horizontal
   H_CEN_ALIGN = 1 << 4,
   H_RIG_ALIGN = 1 << 5,
   T_OPAQUE    = 1 << 6  // more flags can be added until 1 << 12:C_CHART_STYLE

// prototypes:
// create a multi-gradient 'nbcolors'-palette using 'nbsteps' RGB values
// palette   - position in the palette where to store gradient
// nbcolors  - gradient size (in colors)
// steps     - array of rgb_t values used to build the gradient
// nbsteps   - number of entries in the rgb_t values array (must be >= 2)

void dr_gradient(u8 *palette, int nbcolors, rgb_t *steps, int nbsteps);

// img.x/y     - starting point
// img.pen     - color
// img.flags   - alignment
// if alignment is provided then x/y is the left/center/right point used to
// align the text
// (if the full-path 'font' is null then "./fonts/9pts.gif" is used)
// return the lenght of the printed text in pixels
// (clipping is done only on the right side of the frame-buffer)

u32  dr_text  (bmp_t *img, u8 *font, const char *fmt, ...);

// img.pen  - line color
// if (img.p != 0) the area below the line is painted with the 
      // img.bgd color until the img.p bitmap's pointer value

void dr_line  (bmp_t *img, int x1, int y1, int x2, int y2);

// img.pen  - circle color
// if(img.bgd > 0), the circle is filled with the img.bgd color palette index
void dr_circle(bmp_t *img, int x, int y, int radius);

Charts & sparklines


Build real-time Area, Bar, Dot, Line, Pie, Ring charts with various styles:

// example:
// make a sparkline, use C_LINE without (C_TITLES | C_LABELS | C_AVERAGE)
float tab[] = {10042, 10098, 10182, 10154, 10160, 10132, 10160, 10146};
bmp_t img;
img.w     = 30; // width
img.h     = 10; // height
img.bbp   =  3; // 1 << 3 = 8 colors
img.flags = C_LINE | C_GRID; // or img.flags = C_BAR;
dr_chart(&img, 0, 0, 0, 0, tab, sizeof(tab) / sizeof(float));

   C_LINE    = 1 << 12, // line chart
   C_AREA    = 1 << 13, // fill the chart with 'img.bgd' color
   C_BAR     = 1 << 14, // bar  chart
   C_PIE     = 1 << 15, // pie  chart (prettier with dedicated palette entries)
   C_RING    = 1 << 16, // ring chart (a pie chart with a central hole)
   C_DOT     = 1 << 17, // dot  chart
   C_AVERAGE = 1 << 18, // horizontal dotted line showing the average value
   C_LABELS  = 1 << 19, // make room for (and print) x/y axis ticks and labels
   C_TITLES  = 1 << 20, // make room for (and print) title & sub-title (if any)
   C_GRID    = 1 << 21, // draw vertical and horizontal lines in the background
   C_FGRID   = 1 << 22  // the background grid is filled by alternance

// prototype:
// img      - pointer on a bmp_t bitmap structure
// title    - text printed in black at the upper-left corner of the bitmap
// subtitle - text printed in black at the upper-right corner of the bitmap
// tags     - x axis labels (if null, 'ntag' numbers are printed instead)
// ntag     - number of x axis labels
// val      - values to render as a chart
// nval     - number of values to render as a chart
void dr_chart(bmp_t *img, u8 *title, u8 *subtitle,
              u8  **tags, u32 ntag,
              float *val, u32 nval);



Sending email from your C scripts:

// 'mail_server' can be "" to pass a port number different
// from the default SMTP port 25.
// Return value 0 means success. See the email.c example.

int sendemail(char *mail_server,
              char *src_mailaddr, char *dst_mailaddr,
              char *subject, char *text,
              char *auth_user, char *auth_pass, // 'login' auth. only
              char *error); // pre-allocated (and later freed) by caller
int isvalidemailaddr(char*szEmail); // true or false (only checks the syntax)

// the 'error' parameter must point to an allocated buffer sized to the total
// size of headers + email body + attachments (encoded in base64: worst case).
// Using 'total size' * 2 is more than safe as base64 inflates by ~33%.
// See the contact.c example for more details:

typedef struct attach_s
   char *name; // file name (the extension is used to find the MIME type)
   char *file; // file contents
   u32  size;  // file size
   u32  errlen;// size of the pre-allocated error buffer (mandatory)
   u32  nbr;   // number of attachments (only the 1st array item is necessary)
} attach_t;

// If you wonder why 'nbr' is reduncdant then this is because the attachment
// feature was added to sendemail() but I did not want prior code using it
// to break because of a new 'attachment' function argument.

typedef struct email_s
   char       *text;   // must be first
   const char *tag;    // *tag = "@"; to detect an attachment structure
   attach_t   *attach; // array of attachments
} email_t;

// When you want to send an email with attachment(s) (or a body with contents
// using 8-bit data, in which case the 'text' field of the attach_t structure
// can be set to "."), the email_t structure must be passed as the 'text'
// sendemail() argument.



Initially made to replace the Windows WinInet's atrociously slow routines and then declined in various flavors for portable high-resolution timing and HTTP strings processing:

typedef struct _tm_s // just to make our life easier under MS-Windows
  u32 tm_sec;   // seconds after the minute - [0,59]
  u32 tm_min;   // minutes after the hour   - [0,59]
  u32 tm_hour;  // hours since midnight     - [0,23]
  u32 tm_mday;  // day of the month         - [1,31]
  u32 tm_mon;   // months since January     - [0,11]
  u32 tm_year;  // years since 1900
  u32 tm_wday;  // days since Sunday        - [0,6]
  u32 tm_yday;  // days since January 1     - [0,365]
  u32 tm_isdst; // daylight savings time flag
} tm_t;

u32      cycles     (void); // return CPU clock cycles (in-minutes overflow)
u64      cycles64   (void); // return CPU clock cycles (will never overflow)
u64      getns      (void); // EPOCH time in nanoseconds  (1 second/1bn)
u64      getus      (void); // EPOCH time in microseconds (1 second/1m)
u64      getms      (void); // EPOCH time in milliseconds (1 second/1000)

time_t   s_time     (void); // on Windows, much much faster than time(0);
tm_t    *s_gmtime   (time_t t, tm_t *ts); // those are thread-safe
tm_t    *s_localtime(time_t t, tm_t *ts);
char    *s_asctime  (time_t t, char *buf);
size_t   rfc2time   (char*s); // "Tue, 06 Jan 2009 06:12:20 GMT" => u32
char    *time2rfc   (time_t t, char *buf); // inverse of above operation



Useful to blacklist offenders in real-time:

int  fw_block(char *addr, int is_ip); // at the moment, 'is_ip' MUST be '1'
int  fw_allow(char *addr, int is_ip); // at the moment, 'is_ip' MUST be '1'
void fw_state(xbuf_t *rules);         // return the current rules

Random numbers


Whether you need raw speed or true (hardware) random numbers, this is probably your best options:

typedef struct { u32 x[5]; } prnd_t;
void sw_init(prnd_t *rnd, u32 seed); // pseudo-random numbers generator
u32  sw_rand(prnd_t *rnd);           // (period: 1 << 158)
typedef struct { u32 x[270340]; } rnd_t;
void hw_init(rnd_t *rnd); // hardware random numbers generator
u32  hw_rand(rnd_t *rnd); // (save the context: hw_init() takes time)



Since they are already implemented in G-WAN, there is no real need to write your own:

// prototypes:
u32 crc_32  (char*data, u32 len, u32 crc);
u32 adler_32(char*data, u32 len, u32 crc); // adler_32 is slower than crc_32
// example:
u32 crc = 0; // starting value
crc = crc_32(data, length, crc);



Like for Checksums, these calls aren't likely to be implemented better in your code, so G-WAN just makes them available for your applications:

// u8 dst[16]; // the resulting 128-bit hash
// md5_t ctx;
// md5_init(&ctx);
// int i = 10;
// while(i--)
//    md5_add(&ctx, data[i].ptr, data[i].len);
// md5_end(&ctx, dst);

typedef struct { u8 x[220]; } md5_t;
void md5_init(md5_t *ctx);
void md5_add (md5_t *ctx, u8 *src, int srclen);
void md5_end (md5_t *ctx, u8 *dst);
// a wrapper on all the above MD5 calls
void md5(u8 *input, int ilen, u8 *dst);

// u8 dst[20]; // the resulting 160-bit hash
// sha1_t ctx;
// sha1_init(&ctx);
// int i = 10;
// while(i--)
//    sha1_add(&ctx, data[i].ptr, data[i].len);
// sha1_end(&ctx, dst);

typedef struct { u8 x[220]; } sha1_t;
void sha1_init(sha1_t *ctx);
void sha1_add (sha1_t *ctx, u8 *src, int srclen);
void sha1_end (sha1_t *ctx, u8 *dst);
// a wrapper on all the above SHA-160 calls
void sha1(u8 *input, int ilen, u8 *dst);

// u8 dst[32]; // the resulting 256-bit hash
// sha2_t ctx;
// sha2_init(&ctx);
// int i = 10;
// while(i--)
//    sha2_add(&ctx, data[i].ptr, data[i].len);
// sha2_end(&ctx, dst);
typedef struct { u8 x[232]; } sha2_t;
void sha2_init(sha2_t *ctx);
void sha2_add (sha2_t *ctx, u8 *src, int srclen);
void sha2_end (sha2_t *ctx, u8 *dst);
// a wrapper on all the above SHA-256 calls
void sha2(u8 *input, int ilen, u8 *dst);



AES is the U.S. NIST FIPS PUB 197 standard (2001) developed by Belgians Joan Daemen & Vincent Rijmen and approved by the NSA. Useful for compliance:

typedef struct aes_s
{  u32 rounds;
   u32 *keys;
   u32 buf[68];
} aes_t;

// mode     - values 1:ENCRYPT, 0:DECRYPT
// keylen   - values 128, 192 or 256*
//            (*) AES-128 is faster and safer than AES-256

void aes_init(aes_t *ctx, u32 mode, u8 *key, u32 keylen);

// mode     - values 1:ENCRYPT, 0:DECRYPT
// len      - length in bytes to process
// iv       - initialization vector (modified), declare: u8 iv[16];
// src      - source to encrypt
// dst      - destination (encrypted)
// Cipher-Block Chaining (CBC) has been invented by IBM in 1976:
// Each plaintext block is XORed with the previously encrypted block before
// being encrypted. It makes all blocks dependent on all the previous blocks.
// To make the ciphertext unique, an IV must be used for the first block.
// It makes encryption sequential (no parallelization) and the message
// requires padding to match the block size.
// A bit change in a plaintext affects all the ciphertext. A plaintext can
// be recovered from 2 contiguous ciphertext blocks, which makes it possible
// to parallelize decryption.
void aes_enc(aes_t *ctx, u32 mode, u32 len, u8 *iv, u8 *src, u8 *dst);



Gzip and Deflate compression at hand (decompression is left as an exercise for the reader because those two standards are dangerous by-design with untrusted input):

// if(gzip != 0) then we use the 'gzip' format, else we use the 'zlib' format
// (the new 'zlib' format is both slower and larger than the old 'gzip' format)
// if you already have the crc32, pass it into 'crc', else 'crc' MUST be NULL
// return the dstlen, 0 on error

u32 zlib_cmp(char *src, u32 *crc, u32 srclen, char *dst, u32 dstlen, int gzip);