Qore Programming Language Reference Manual  0.8.10
 All Classes Namespaces Functions Variables Groups Pages
Release Notes

Qore 0.8.9

Release Summary
Major release with many new features and also many bugfixes.

New Features in Qore

Bug Fixes in Qore

  • fixed a crashing bug when HTTP messages with duplicate Connection, Content-Encoding, Transfer-Encoding, Location, or Content-Type headers is received
  • fixed a bug parsing octal character constants in the lexer when octal digits followed the octal constant (ex: "\0441" where the "1" would cause an error)
  • allow escaping "$" character in regular expression substitution target strings, previously it was impossible to output a literal "$" + a digit, since this would be interpreted as a numbered input pattern expression
  • fixed a bug in the HTTPClient::getURL() and HTTPClient::getProxyURL() methods where the URL's path was not given with a leading "/" character
  • CsvUtil module fixes:
    • fixed "date" field handling with empty input (now maps to 1970-01-01)
    • fixed CsvDataIterator::next() when header_lines > 0 and working with empty input data
  • added support for compiling on OSX Mavericks
  • fixed an infinitely recursive call in Table::del() in SqlUtil
  • fixed a bug in v*printf() where '%' was not handled correctly in all cases
  • fixed bugs in microseconds and milliseconds() with large arguments
  • fixed a bug where a call to a call reference to a static method across a program boundary with local variables as arguments would cause a crash due to improper setting of the program context before the arguments are evaluated
  • fixed a bug in Datasource::copy() method where implicitly-set options were not carried over into the new object
  • fixed a bug in the DatasourcePool class where implicitly-opened connections would not be guaranteed to have the same server time zone setting as the initial connections (for example, could cause problems with server timezone settings if running in a program context with a different local time zone attribute)
  • fixed bugs in SqlUtil generating "create table" and "align table" SQL with DBs where unique indexes automatically create unique constraints (ex: MySQL)
  • fixed a bug in lchown() where chown() was used interally instead of lchown()
  • fixed a bug in PgsqlSqlUtil retrieving sequence values with Database::getNextSequenceValue()
  • fixed an off-by-one memory bug in Qore::date(string, string) parsing a 4-digit date mask
  • fixed memory leaks in class member and class static variable management
  • fixed memory leaks when an entire class has to be rolled back due to parse errors and the class has pending static variables
  • fixed memory leaks in constant handling with values containing call references
  • fixed a memory leak in constant destruction with parse rollbacks when the constant value was NULL
  • fixed an error in the rounding heuristic for arbitrary-precision numeric values that could produce invalid results (ex: 34.9n * 100 = 34902n)
  • enforce PO_NO_FILESYSTEM with the %include directive
  • fixed a bug managing object private data in complex inheritance cases where the same class may be inherited with virtual private data and also real private data
  • fixed a bug in socket timeout handling with select() errors
  • fixed a memory leak in handling abstract methods when multiple abstract methods with the same name but different signatures were declared in a class

Qore 0.8.8

Release Summary
Major new features and bug fixes with a particular focus on enhanced HTTP capabilities and enhanced database processing

Changes That Can Affect Backwards-Compatibility

New Features in Qore

Bug Fixes in Qore

  • fixed a bug where the ?: operator could thrown spurious exceptions when parsing because it would return the type of the initial boolean expression as the return type of the operator
  • fixed a bug where classes with unimplemented inherited abstract variants would sometimes cause runtime exceptions to be thrown when instantiated but should have instead been caught at parse time
  • fixed a parser bug where out-of-line class method definitions could not be defined in a namespace block
  • fixed a bug parsing arguments in parse_uri_query() in the HttpServer module
  • fixed several bugs where parse exceptions could show the wrong source location:
    • with type errors in function calls
    • when resolving global variables
    • in base class constructor arguments
    • for empty blocks with a missing return statement
    • when validating types used with the return statement (also associated warnings)
    • in methods calls
    • in hash value expressions
    • with redeclaring local variable return types
    • in local variable object instantiations
  • really fixed the bug thought to be fixed in 0.8.7 "where SSL errors would cause the affected thread to go into a infinite loop using 100% CPU" - this turned out to be easily reproducible on all platforms; when the SSL connection was shut down cleanly by the remote end before a response message was returned, an infinite loop would result
  • fixed a bug where it was impossible to output a single '\' character in regex substitution expressions; '\' was taken as an escape character, and '\\' was output literally, now '\\' is output as '\'
  • fixed a bug where a parse-time crash would occur when calling the copy() method for a class that does not implement an explicit copy() method
  • fixed a bug where arguments passed to a copy method were ignored; now an exception is thrown
  • fixed a bug where public members and static variables of privately-inherited classes were incorrectly treated as public attributes of the child class
  • fixed a bug where slices could be made of objects from outside the class including private members
  • fixed a bug where memberGate() methods were not being respected when taking a slice of an object
  • fixed bugs in the integer Socket::recv*() methods where a SOCKET-CLOSED exception was not thrown when the remote end closed the connection
  • fixed a bug related to out-of-order parse initialization for functions and methods which resulted in the wrong return type being returned for a method with more than 1 variant where the variant could not be matched at parse time
  • fixed a bug where a non-variable-reference member of an "our" variable declaration list would cause a crash due to passing the incorrect argument in sprintf()
  • fixed sandboxing / protection errors with inherited code; subclasses inheriting code from a parent class with different parse options would cause the child parse options to be used when running the parent class code which caused errors; now parse options are enforced properly on the block level
  • fixed the RangeIterator class; it was still abstract due to a missing Qore::RangeIterator::valid() method
  • fixed a bug where the wrong error was being returned after a connection reset (remote connection close) in sockets with integer recv*() methods which could in some cases lead to an infinite loop
  • fixed a bug where private members of a common base class were not accessible by objects of subclasses sharing the common base class
  • fixed many bugs in CsvUtil and updated the module version to 1.1
  • initialize static openssl crypto locks for multi-threaded openssl library access; without this crashes can result (for example in error queue management)
  • fixed a bug where Qore::HTTPClient::getURL() returned an invalid URL when no URL was set; now it returns NOTHING in this case
  • fixed a bug managing feature/module lists in inherited Program objects; user modules were listed in the child Program object even though user module code is not imported in child Program objects
  • fixed a bug where an invalid guard condition in critical lvalue storage code can cause unreferenced data to be returned while in a lock which can cause a crash in a multithreaded program
  • fixed a bug where references were not being written to the output variable if an exception was active when the code block exited
  • fixed a bug setting the precision for arbitrary-precision numbers with large exponents (like "1e100n")
  • implemented more strict adherence to RFC 2616 (HTTP 1.1) regarding message-body handling in requests and response message generation and parsing
  • fixed a bug with Condition::wait() on Darwin with negative timeout values where a short timeout was used instead of an indefinite wait
  • fixed bugs in the SmtpClient and MailMessage modules where mail messages were being serialized incorrectly if there were no attachments (there was no possibility to set the content transfer encoding) and also where it was not possible to set the content-type for the message body when it was sent as a part of a multipart message
  • fixed bugs handling arguments declared as type *reference (reference or nothing)
  • fixed bugs in executing code accross Program object barriers with reference arguments
  • fixed a bug with the switch statement where character encoding differences would cause strings to mismatch even if they were otherwise identical; now hard comparisons with strings allow for implicit automatic temporary character encoding conversions for the comparison
  • fixed a bug where qore failed to set the time zone region correctly when set from /etc/localtime and this file is a relative symlink rather than absolute
  • fixed a bug where substr() and <string>::substr() were returning NOTHING if the arguments could not be satisifed contrary to the documentation and the declared return type, now an empty string is returned in those cases
  • fixed bugs rounding number values between 10 and -10 (non-inclusive) for display, fixed bugs rounding number value regarding digits after the decimal point for display with Qore::NF_Scientific
  • fixed a bug in the Qore::Dir class where it was not possible to chdir to the root directory "/"
  • fixed a bug where recursive references were allowed and memory leaks would occur due to recursive references; these are now caught at runtime and a REFERENCE-ERROR exception is thrown
  • fixed a configure bug with bison >= 3
  • fixed a bug in the HttpServer module when automatically uncompressing supported content-encodings to set the resulting string's character encoding correctly
  • fixed a bug in the instanceof operator when working with objects and classes created from different source Program objects
  • fixed a bug in *printf() formatting with floating-point and number values where no digits were displayed right of the decimal point unless a specific number of digits was specified in the format string
  • fixed the return type of <bool>::typeCode(); was returning a boolean instead of Qore::NT_BOOLEAN
  • fixed a bug there NULL was evaluated as True in a boolean context rather than False
  • fixed a bug where Qore::Socket::recvBinary() would ignore the first data read
  • fixed starting listeners on UNIX domain sockets on Soalris in the HttpServer module
  • fixed a bug where number("") was being converted to @NaN@n
  • fixed return type of HTTPClient::getConnectionPath()
  • fixed several bugs with logical comparison operators and arbitrary-precision numeric values where arbitrary-precision numeric values were not being prioritized as numeric values and also in some cases were being first converted to doubles and then operated on
  • fixed a bug in the socket code where the socket close condition was not flagged with SSL connections when writes failed due to the remote end closing the connection; an error would only be raised on the following socket operation
  • fixed a mismatched delete/malloc error with time zone initialization and the localtime file

Qore 0.8.7

Release Summary
Code embedding improvements

Changes That Can Affect Backwards-Compatibility

Fixes for Code Inheritance in Program Objects
The following changes are meant to sanitize code inheritance in child Program objects to fix long-standing design bugs in code encapsulation by addressing the lack of fine-grained control over symbol visibility in inherited code.
  • public: The public keyword's usage in modules has now been expanded to provide the same functionality generically in Program objects; if classes, constants, namespaces, functions, or global variables are defined with the public keyword, then these symbols will be inherited into child Program objects as long as no parse options prohibit it.

    This change was made to give programmers complete control over which symbols are inherited in child Program objects, whereas because prior to this change, the control was very course.

  • the default behavior of Qore regarding inherting global variables and functions with user variants was changed to be consistent with namespaces, classes, and constants; that is; public symbols are inherited by default.

    The following constants were renamed:
  • builtin symbols are no longer inherited from user modules; only independent user symbols; the main change is that if a user module adds new user methods to a builtin class or new user variants to a builtin function, these changes are no longer imported into target Program objects.
File Method Changes
The following methods were updated to throw exceptions on all errors rather than a return code for I/O errors in order to avoid hard to debug conditions due to ignoring I/O errors by forgetting to check the return value on the following methods: Note that the above changes will hopefully only minimally impact backwards-compatibilty since the change is in error handling, and additionally each of the above methods could also throw an exception if called when the object was not open.

New Features in Qore

Bug Fixes in Qore

  • fixed a runtime class matching bug when identical user classes were created in different Program objects, the match could fail at runtime because the wrong APIs were being used
  • fixed a crashing bug in the map operator with a select expression when used with an AbstractIterator object for the list operand
  • fixed a bug where the generation of internal strings for abstract method signatures tries to resolve class names that are declared out of order, which incorrectly resulted in a parse exception; the fix is to use the class name in the signature before class resolution; the class is resolved in the second stage of parsing (symbol resolution) anyway, if it can't be resolved then the changes to the Program are rolled back anyway
  • a potential deadlock was fixed when calling exit() while background threads were running; it was possible for a thread to be canceled while holding a Mutex (for example) and then for another thread to deadlock trying to acquire the Mutex and therefore for the process to deadlock because pthread_mutex_lock() is not a cancellation point. The solution was to cancel all threads first, then wait half a second, then call exit()
  • fixed a bug where global variables were being evaluated with strict mathematical boolean evaluation even when %perl-bool-eval was enabled (which is the default)
  • fixed bug in Qore::parseBase64String() and Qore::parseBase64StringToString() when called with an empty string argument; in this case uninitialized memory was returned
  • fixed runtime dynamic memory leaks in the Select From List Operator (select) and Map Operator (map) operators when used with iterators
  • do thread-specific cleanup in the main thread when cleaning up/shutting down the qore library
  • added additional openssl cleanup code for thread-local data and when cleaning up the qore library
  • fixed a bug matching function/method variants at runtime
  • fixed a race condition deleting global dynamic handlers in the HttpServer module
  • fixed a bug where declaring an abstract method with parameters and then declaring a concrete reimplementation of the method in a child class with no parameters caused a parse-time crash
  • fixed a bug where trying to dynamically call a function that does not exist results in a deadlock due to an error where a mutex is not unlocked
  • fixed a bug in the Qore::Socket::sendHTTPMessage() and Qore::Socket::sendHTTPResponse() methods regarding the timeout parameter
  • fixed a bug in an socket SSL error message where the method name was printed from non-string memory (used wrong ptr for the %s format argument)
  • fixed some major crashing bugs related to reference handling; a global variable assigned a reference to a reference to a local variable would cause a crash
  • reference and *reference type fixes: an error in reference type handling allowed non-reference values to be passed to builtin code expecing references which caused a crash; the reference and *reference types would accept any value type
  • attempted to fix a non-reproducible bug seen on rhel5 in the Socket class where SSL errors would cause the affected thread to go into a infinite loop using 100% CPU

Qore 0.8.6.2

Release Summary
Iterator improvements and design fixes

Changes That Can Affect Backwards-Compatibility

Fixes for Iterator Class Design Bugs
Iterators, particular regarding the map and select operators, were implemented in a confusing and inconsistent way; even the qore documentation was incorrect, and examples were given incorrectly. The following changes will break functionality using the badly-implemented behavior of iterators before, but since the fix comes fairly soon after the introduction, hopefully this change will not cause too many problems with existing code. All users polled about the iterator changes in this release saw them as positive and desired changes to the language.

Changes in Qore

Qore 0.8.6.1

Release Summary
Major bug fixes and minor new features

Changes in Qore

  • updated the %try-module parse directive to support a variant without an exception variable for usage in Program objects where Qore::PO_NO_TOP_LEVEL_STATEMENTS is set
  • added code to raise an invalid-operation warning with the elements operator when called with a type that can never return a value with this operator
  • updated the File class's internal buffer size from 4KB to 16KB which greatly improves read performance
  • added new public APIs for the QoreNumberNode class to allow for proper de/serialization in external modules
  • Pop3Client module:
    • added the Pop3Client::logPassword() methods and masked password by default in the debug log
    • updated module to v1.1
  • Mime.qm module:
    • declared the MultiPartMessage::getMsgAndHeaders() method abstract as originally intended
    • added MultiPartMessage::parseBody() static method
    • updated module to v1.3

Bug Fixes in Qore

  • fixed crashing bugs due to the lack of proper lvalue checks with the expression for the background operator with operators using lvalues with local variables
  • fixed rounding of arbitrary-precision numeric values for display purposes when the last significant digit is just to the right of the decimal point (ex: was displaying 10.2 as "11." for example)
  • fixed a race condition in static destruction of the library when a background thread calls exit() that could cause a segfault on exit
  • fixed a static memory leak in Program objects when constants contain code references to functions or static methods
  • fixed a bug parsing user modules; the Program context was not set properly which could lead to a crash when parsing user modules loaded from the command-line or to incorrect parse options when loaded from user Program code
  • fixed a bug where the invalid-operation warning with the keys operator was not being triggered in common cases that should have triggered the warning
  • MailMessage.qm module:
    • fixed recognizing mime messages with additional text after the version number (ex: "Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))")
    • fixed a bug setting the content-type of message parts (this fix is now in the Mime.qm in the MultiPartMessage::getMsgAndHeaders() method
    • fixed multipart message parsing by using MultiPartMessage::parseBody() in the Mime.qm module; now also parts with subparts are parsed correctly as well
    • fixed a bug where the sender and from values were not being set properly when parsing email messages
    • updated module to v1.0.3

Qore 0.8.6

Release Summary
Major new features and a few bug fixes

Changes That Can Affect Backwards-Compatibility

Perl-Style Boolean Evaluation
Qore's default boolean evaluation mode was changed from strict mathematical to a more intuitive perl- (and Python-) like style. This change was implemented to address one of the oldest design bugs in Qore: strict mathematical boolean evaluation. See %perl-bool-eval for a description of the new default boolean evaluation mode.

To get the old strict mathematical boolean evaluation, use the %strict-bool-eval parse option.

An example of the change; now the following if statement block will be executed as the if expression is now evaluated as True:
my string $str = "hello";
if ($str)
printf("Qore says hello\n");
Previously (i.e. with %strict-bool-eval) the if expression above would be evaluated as False because the string value was converted to an integer 0, however as of Qore 0.8.6 (with the default perl-bool-eval) it is True since the string is not empty; empty strings and string value "0" are evaluated as False.

Perhaps counterintuitively (and the reason this was changed to be the default in qore), the chance for regression errors in qore code is very small, because for all cases where the old logic could be applied (meaning excluding cases where the result was always False due to the data types or values being evaluated), the results are the same with the new logic, except for one case; the case where a string has more than one character and begins with a zero (ex: "00"). In this case, the old logic would always return False, because the value was first converted to an integer 0, whereas the new logic will return True. Note that in the case of a string with a single "0", both the old and new boolean logic returns False.

Basically with this option set, qore's boolean evaluation becomes like perl's and Python's, whereas any expression that has the following values is False: NOTHING, string "0" and empty strings, integer, float, and number 0 (zero), absolute date 1970-01-01Z (ie the start of the epoch with an offset of 0), relative date 0s (or any relative date with a 0 duration), NULL, empty binary objects, empty hashes, and empty lists. All other values are True.
Note
also affects the boolean(any) function
Changes in the Socket Class
The Socket class was enhanced to support timeouts with non-blocking I/O on all send operations; many Socket methods that send data were originally implemented to return an error code on error, however they would also throw exceptions if the socket were not open, so the error handling was inconsistent (exceptions versus return codes).

Additionally it was not possible to get error information at all for SSL errors if the socket was connected with SSL, which, according to Qore's socket design, should be transparent for the programmer.

For these reasons the implementation was deemed inconsistent and unintuitive; the change was to add optional timeout parameters to all send methods and to allow the methods to throw exceptions (instead of simply returning -1 and not being able to determine the cause of the error in many cases).

The following methods were updated to accept optional timeout parameters and throw exceptions on all errors rather than a return code for I/O errors:
New Abstract Method in AbstractIterator
The following abstract method was added:
  • Qore::AbstractIterator::valid() was added (with concrete implementations in all iterator classes derived from this base class delivered with Qore); this method tells if the object is currently pointing to a valid iterator.

    For any user classes inherting AbstractIterator directly (as opposed to another concrete iterator class in Qore, where the method has already been added), a concrete implementation of this method will have to be added as well or that class will become abstract with this release of Qore.

New Features in Qore

Arbitrary-Precision Numeric Support
Qore now uses the MPFR and GMP libraries to provide arbitrary-precision numeric support. This type can be used for high-precision mathematics or for storing NUMERIC (ie DECIMAL or NUMBER) column values when retrieved from databases by Qore DBI drivers that support the new capability DBI_CAP_HAS_NUMBER_SUPPORT (previously these values would be retrieved as Qore strings in order to avoid information loss).

For more information, see the new number type, number, and <number>
New CsvUtil Module
The CsvUtil module implements the CsvFileIterator class that allows for easy parsing of csv-like text files
%try-module Parse Directive to Handle Module Load Errors at Parse Time
The new %try-module parse directive allows for module load errors to be handled at parse time; ex:
%try-module($ex) some-module > 1.0
printf("error loading module %y: %s: %s\n", $ex.arg, $ex.err, $ex.desc);
exit(1);
%endtry
Abstract Class Hierarchy Improvement
As of this version of qore, concrete implementations of abstract methods no longer have to have exactly the same return type as the abstract method; it is now sufficient that the return type in the concrete method meets a compatibility test with the return type of the abstract method in the parent class.

For example the following is now valid (and MyConcreteClass is not abstract, whereas previously because the return types in the child class were not exact, MyConcreteClass would be considered abstract by qore):
class MyAbstractClass {
abstract any doSomething();
abstract *string getString();
}
class MyConcreteClass inherits MyAbstractClass {
int doSomething() {
return 1;
}
string getString() {
return "hello";
}
}
DBI Improvements
Three new DBI capabilities were implemented, including a new option API as follows:
  • DBI_CAP_HAS_NUMBER_SUPPORT: DBI drivers declaring this capability can accept number values and can also return number values, if a DBI driver does not declare this capability, then number values sent for binding by value are automatically converted to float values before being sent to the driver
  • DBI_CAP_HAS_OPTION_SUPPORT: this indicates that the driver supports the new option API, allowing options to be set on each connection. See the following for more information:
    • Datasource::constructor(hash): now passes options to the DBI driver if the driver supports the option API
    • Datasource::constructor(string): (new in 0.8.6) passes options to the DBI driver if the driver supports the option API
    • Datasource::getOption(string): (new in 0.8.6) returns the value of the given option if the driver supports the option API
    • Datasource::getOptionHash(): (new in 0.8.6) returns a hash of the current option values for the current connection if the driver supports the option API
    • Datasource::setOption(): (new in 0.8.6) allows options to be changed after the object is created
    • DatasourcePool::constructor(hash): now passes options to the DBI driver if the driver supports the option API
    • DatasourcePool::constructor(string): (new in 0.8.6) passes options to the DBI driver if the driver supports the option API
    • DatasourcePool::getOption(string): (new in 0.8.6) returns the value of the given option if the driver supports the option API
    • DatasourcePool::getOptionHash(): (new in 0.8.6) returns a hash of the current option values for the current connection if the driver supports the option API
    • dbi_get_driver_options(string): (new in 0.8.6) returns a hash of driver option information without values
  • DBI_CAP_SERVER_TIME_ZONE: indicates that the DBI driver will convert any bound date/time values to the server's time zone before binding and also will tag date/time values retrieved from the server with the server's time zone. This capability also implies that the driver supports the new "timezone" option.
Socket Improvements
The Socket class was updated to support non-blocking I/O on all send methods; the following methods were updated to accept optional timeout parameters: The following methods were enhanced to provide better error information when throwing exceptions:
Iterator Improvements
The following improvements were made in qore to support more flexible and ubiquitous iterators:
Text File Parsing Enhancements
The following improvements were made in qore to support more flexible file parsing:
Other Improvements and Changes

Bug Fixes in Qore

  • fixed a bug in the map operator with a select expression when the list operand is NOTHING; it was returning a list with one NOTHING element instead of NOTHING
  • applied a patch by Reini Urban to allow for multi-arch builds on Debian
  • fixed bugs calculating the byte offset for string searches in the c++ QoreString::index() and QoreString::rindex() functions when the offset is negative and the strings have a multi-byte character encoding (such as UTF-8)
  • fixed a bug where calling an abstract method from a class where the abstract method is implemented was causing a parse error to be thrown
  • fixed a bug where the wrong source code location was displayed when raising a parse exception in operator expression parse initialization for some operators
  • fixed bugs in regexes in the HttpServer::addListeners() and HttpServer::addListenersWithHandler() methods (HttpServer module version updated to 0.3.5)
  • fixed bugs handling non-blocking reads in the Socket class; the timeout setting was only enforced for the first read; subsequent reads were made as blocking reads
  • fixed a bug in the Socket class when the SSL session requires renegotiation during non-blocking I/O
  • File::constructor() now throws an exception if called with a tty target and %no-terminal-io is set
  • fixed a bug in split with quote (<string>::split(string, string, bool) and Qore::split(string, string, string, bool)) if the separator pattern was not found and the single field was not quoted either
  • fixed a bug handling nested %ifdef and %ifndef blocks with %else in the inside block
  • fixed a crashing due to the failure to clear the "PF_TOP_LEVEL" flag when initializing statements, this could cause temporary variables in a statement to be marked as the start of the global thread-local variable list, and then after such variables are deleted, then a crash happens when trying to access the global thread-local variable list
  • fixed a crashing bug at parse time merging function lists in namespaces declared multiple times
  • fixed a bug in executing user module init() closures
  • fixed a bug where the qore library could crash when destroying a Program object due to a race condition in removing signal handlers managed by the Program object; the Program calls the signal handler manager to remove the signals, but the signals can be removed concurrently to the request while the Program object is iterating the signal set (ie it is modified while being iterated), which causes a crash
  • added code to detect when the same namespace is declared both with and without the public keyword when defining user modules which can result in entire namespaces being silently not exported (and can be difficult to debug); now a parse exception is thrown if this happens while parsing a user module
  • added code tags to File methods without side effects
  • made many minor documentation fixes

Qore 0.8.5.1

Release Summary
Bugfix release

Bug Fixes in Qore

  • fixed a race condition accessing global and closure-bound thread-local variables in multithreaded contexts
  • fixed a bug in transaction management with the DatasourcePool class when used with the SQLStatement class
  • fixed an error in the MailMessage.qm user module where mail headers requiring encoding were not encoded and those not requiring encoding were encoded with Q encoding
  • fixed an error in the Mime.qm user module where "_" characters in q-encoded headers were not encoded correctly

Qore 0.8.5

Release Summary
Major new features and a few bug fixes

New Features in Qore

Abstract Methods and Interfaces
Qore now supports the abstract keyword when declaring methods; an abstract method has no implementation and must be implemented in child classes with the same signature for the child class to be instantiated.

Classes with abstract methods define interfaces; a concrete implementation of the interface is a class that inherits the class with abstract methods and implements all the abstract methods.

Abstract methods are defined with the following syntax:
class MyAbstractInterface {
abstract string doSomething(int $param);
abstract bool checkSomething(string $arg);
}
The following abstract classes now exist in Qore: The following new iterator classes have been added to Qore: Classes inheriting AbstractIterator have special support so that objects can be easily iterated in the following list operators:
Universal References
All restrictions on references have been removed from Qore; references to local variables may now be passed to the background operator and passed as arguments to closures.

Basically when a reference is taken of a local variable that could result in the local variable being accessed in a multi-threaded context, the variable is treated as a closure-bound local variable in the sense that it's lifetime is reference-counted, and all accesses are wrapped in a dedicated mutual-exclusion lock to ensure thread safety.
Pop3Client Module
A Pop3Client module has been added providing an API for communicating with POP3 servers and retrieving email messages.

The module uses functionality provided by the new MailMessage module to represent email messages (and attachment data) downloaded from the server.
MailMessage Module
The MailMessage module provides common functionality to the Pop3Client and SmtpClient modules to represent email messages for receiving and sending, respectively. This module was created mostly from functionality removed from the SmtpClient and enhanced to provide support for reading email messages in the new Pop3Client module.
SmtpClient Module Changes
The Message and Attachment classes were removed from the SmtpClient module to the MailMessage module. Backwards-compatible definitions for the Message and Attachment classes are provided in the SmtpClient module to rexport the removed functionality for backwards compatibility.
Other Minor Improvements and Changes

Bug Fixes in Qore

  • fixed format_date() output for "MON" and "DAY", etc
  • fixed a memory leak in the parser related to parse exception handling with namespace members
  • fixed an invalid assert() in module handling when an error occurs loading the module (only affected debug builds)
  • tagged digest and crypto functions internally as RET_VALUE_ONLY
  • do not kill TID 1 (the initial / main thread) when calling exit() in background threads as a crash can result with some 3rd party libraries that spawn their own threads on some platforms (observed on Darwin & Solaris 10 at least)
  • fixed a memory bug in the new builtin function API used by modules built with qpp
  • fixed memory bugs in the type system where uninitialized type pointers could be used causing a crash
  • fixed a memory bug in handling "or nothing" types where a non-null pointer would be assumed to be a pointer to the type, however it could actually be a pointer to the NOTHING object, the fix was to ensure that any NOTHING objects in argument lists would be substituted with a null pointer
  • fixed a bug in parse-time variant matching where an argument with parse-time type "object" would be matched as a perfect match to any parameter with any class restriction; this would cause run-time type errors if another valid class was passed that matched another variant of the method or function
  • fixed a build bug that caused qore to be built twice

Qore 0.8.4

Release Summary
Major new features and changes that can affect backwards-compatibility, plus 40 bug fixes

Changes That Can Affect Backwards-Compatibility

Namespace Changes
Qore's internal namespace handling was nearly completely rewritten for Qore 0.8.4. This is because the old code was inefficient and applied namespaces inconsistently to Program objects.

The main change that can cause backwards-compatibility issues is that now functions are full namespace members. If no namespace is explicitly given in a function definition, the function is a member of the unnamed root namespace.

Also the distinction between builtin and user functions was removed. Internally, there is only one kind of function object, which can contain both builtin and user function variants (overloaded variants of the same function with the same name but different arguments).

All Qore builtin functions were moved to the Qore namespace.

Other namespace changes:
  • loading namespaces provided by builtin modules into a Program object is now an atomic operation that may fail, if, for example, objects have already been defined in the target Program with the same name as objects provided by the builtin module. Previously this could cause undefined behavior.
  • namespace lookups are now truly breadth-first as documented; previously the algorithm was depth-first (contrary to the documentation)
  • namespace lookups are now done (both at parse time and runtime) with the help of symbol lookup tables for fast lookups; tables are maintained for both committed and temporary uncomitted parse symbols; this leads to up to 3x faster parsing for Qore code
  • global variables are also now full namespace members, however this does not cause problems with backwards-compatibility

New Features in Qore

User Modules
It is now possible to develop user modules in Qore; several user modules are now included in the Qore distribution, forming Qore-language components of Qore's runtime library.

User modules delivered with Qore 0.8.4:
  • HttpServer: a multi-threaded HTTP server implementation
  • SmtpClient: an SMTP client library
  • TelnetClient: a TELNET client implementation
  • Mime: a set of MIME definitions and functions for manipulating MIME data
There are also new example programs for the above modules in the examples/ directory.

User modules are subject to Qore's functional restriction framework.
Namespace Changes
As listed above:
  • global variables and functions are now full namespace members
  • all builtin functions are now in the Qore namespace
  • real depth-first searches are used for namespace symbols
  • symbols are resolved first in the current namespace when parsing declarations/code in a namespace
The final Keyword
Classes and methods can now be declared "final" to prevent subclassing or overriding in a subclass
Pseudo Methods
Pseudo-methods are class methods that can be implemented on any value; they are also part of class hierarchy. The methods that can be executed on the value depend on the value's type, and all "pseudo-classes" inherit methods from a common base class.

For example:
"string".strlen()
<abf05da3>.size()
500.typeCode()
Are examples of pseudo-methods on literal values.

Some expensive operations such as getting the first or last key (or value) of a hash are now cheap using pseudo-methods, for example:
$hash.firstKey()
$hash.lastValue()
New Doxygen-Based Documentation
The Qore reference documentation is now generated by Doxygen, and is generated directly from the Qore sources. In fact, a new preprocessor known as "qpp" was developed for Qore 0.8.4 to facilitate and enforce doxygen documentation on Qore's runtime library (as well as abstract the relatively complex APIs used to bind C++ code to the Qore runtime library from the C++ programmer).

The documentation is more comprehensive, and corresponds much closer to the actual internal implementation since the documentation is now also contained in and directly generated from the internal C++ implementation of Qore.

For example, there is the <value>::val() method. This method is implemented in the base pseudo class and is reimplemented in other pseudo-classes for other runtime data types as necessary. This method returns True if the value has a value in the same sense as Perl's boolean context evaluation. For example, if the value is a hash with no keys, it returns False; if it is a hash with keys, it returns True; if it is an empty string, it returns False; if it is a non-empty string, it returns True, etc.
LValue Handling Changes
lvalue handling was rewritten as the old implementation was ugly and subject to deadlocks (in rare corner cases).

Furthermore, medium-term, an architectural goal of Qore is to store all ints, floats, and bools internally as the basic C++ type instead of using a class wrapper for each value, which needs dynamic allocation and destruction, which takes up more memory and negatively affects execution speed.

With Qore 0.8.4, all local and global variables are stored using optimized C++ types when declared with the appropriate type restrictions; for example:
my int $i0;
our int $i1;
These declares local and global variables that can only be assigned integer values; in Qore 0.8.4 the value internally will be stored as an "int64" value (and not a dynamically-allocated QoreBigIntNode object).

The same holds for: Note that the optimized lvalue handling has not yet been applied to all lvalues, in particular non-static object members with declared types are not yet implemented with optimized storage; to do this requires a rewrite of Qore's API and ABI (will happen in the next major release of Qore).

This change leads to improved integer and floating-point performance and a smaller runtime memory footprint.
Runtime Optimizations
In addition to the up to 3x faster parsing (as decribed in the namespace changes above), Qore 0.8.4 contains many runtime optimizations designed to reduce the number of dynamic memory allocations performed at runtime.

The optimizations included in this version of Qore are only a half-measure compared to future changes that will necessitate a new binary Qore API.
Per-Thread Initialization
the new set_thread_init() function allows a call reference or closure to be set which will be automatically executed when new threads are started (or a new thread accesses a Program object) which can be used to transparently initialize thread-local data.
More Control Over Thread Resource Exceptions
new functions: Allow for only thread resouces created after a certain point to be processed (for example only thread resources left after some embedded code was called)
New Socket Methods
new methods: Allow upgrading an already-existing socket connection to SSL
Better Socket Error Messages
More information has been added to socket exceptions to provide better feedback when errors occur.
New Socket Event Fields
Support For Blocking Writes in the Queue Class
Queue objects can now be used as a blocking message channel (similar to a Go channel); if a maximum size is given to the Queue constructor, then trying to write data to the Queue when it is full will block until the Queue's size goes below the maximum size; optional timeout parameters have been added to Queue methods that write to the Queue.
New Queue::clear() Method
Does just what you think it does :)
date(string, string) Improvement
added the possibility to specify microseconds when parsing dates against a mask with the date() function
New Support For ++ And – Operators With Floating-Point Lvalues
previously this would either convert the lvalue to an int or throw an exception if the lvalue could not be converted to an int due to type restrictions
Class Recognition/Compatibility Between Program Objects
The problem is that a user class created from the same source code in two different Program objects would be recognized as a different class with parameter and variable type restrictions - ie you could not declare a variable or parameter with a class type restrictions and assign it an object created from the same class source code but created in another Program object.

This problem is analogous to a similar problem with java in that classes built from the same source but from different classloaders are also recognized as different classes.

In Qore 0.8.4 a class signature is created of all public and private objects, and an SHA1 hash is maintained of the class signature, and if the class names and signatures match, then the classes are assumed to be identical, even if they have different internal class IDs (because they were created in different Program objects, for example).
New TimeZone::date(string) Method
to support creating arbitrary dates in a given TimeZone
New GetOpt::parse3() method
This method will display any errors on stderr and exit the program (which is the most typical way of handling command line errors anyway)
+= Operator Optimization For object += hash
this operation is faster in this release
New Parse Option PO_NO_MODULES
Using this option disables module loading
New Parse Option PO_NO_EMBEDDED_LOGIC
Using this option disables all dynamic parsing
New Parse Directives
New Context Functions
SOCKET-HTTP-ERROR Exception Enhancement
The invalid header info received is reported in the exception's "arg" key
Improved Parse Error Messages
Improved some parse error messages dealing with namespace and class declaration errors
Added NT_CLOSURE Constant
type code for runtime closure values

Bug Fixes in Qore

  • fixed a race condition with Program objects when a signal handler is left active and the Program terminates
  • fixed a bug in the File class where the encoding given in the constructor was ignored; if no encoding was given in the File::open*() method then the File's encoding would always be set to the default encoding, now it's set to the encoding given in the constructor (as documented)
  • runtime checks have been implemented so that references to local variables cannot be passed to a closure; this would cause a runtime crash
  • a fix has been made to the delete and remove operators; lists will not be extended when trying to remove/delete list elements that do not exist
  • fixed some bugs showing the error location with bugs in the second stage of parsing (symbol resolution)
  • apply type filters to blocks with a designated return type but no return statement
  • fixed crashing bugs on some 32bit platforms where size_t was assumed to be 64 bits
  • fixed a crashing bug parsing invalid %requires directives in the scanner
  • fixed a bug in usleep() with relative date/time values (added a new usleep() variant to support this)
  • fixed a typo in the command-line help for the qore binary with unknown parse options
  • fixed HAVE_SIGNAL_HANDLING to be False if signal handling is disabled on platforms where signal handling is otherwise available
  • fixed a scanner bug parsing out of line class definitions with a root-justified namespace path (ex: "class ::X::ClassName ...")
  • merging code from binary modules at parse time and at runtime is now transaction-safe (before it would cause memory errors and/or a crash), now if errors are detected then an exception is raised and changes are not applied.
  • fixed a crashing bug in the C++ API function QoreHashNode::setKeyValue() when the value is 0 and an exception occurs or is already active before the call is made
  • fixed a bug in date parsing with a format string - off by one with integer months - added a regression test for this case
  • fixed a memory error with the rethrow statement in enclosing but nested try-catch blocks
  • fixed a crashing bug where qore would try to instantiate a class for a type that did not represent a class (ex: "my int $i();")
  • fixed a memory leak in the softlist and *softlist type implementation
  • make sure and raise a SOCKET-CLOSED error when reading a HTTP header if no data is received
  • make sure and convert encodings with index() and rindex() functions if the encodings don't match
  • build fix: only use a lib64 directory if the directory exists already
  • raise a parse exception in the scanner if a numeric overflow occurs in literal integer values
  • fixed a bug in AbstractSmartLock::lockTID()
  • fixed a major crashing error in the C++ API function QoreStringNode::createAndConvertEncoding(); this function is used by the xml module when parsing XML-RPC sent in a non-UTF-8 character encoding
  • fixed Qore::File::getchar() to always retrieve 1 character (even for multi-byte character encodings)
  • fixed string evaluation in a boolean context to return True with floating-point numbers between -1.0 and 1.0 exclusive
  • printf formatting fix: output YAML-style "null" for NOTHING with %y
  • scanner fix: accept "\r" as whitespace to allow better parsing of sources with Windows EOL markers
  • fixed parse-time type processing/checks for the keys, + and * operators
  • foreach statement fix: unconditionally evaluate the hash when iterating as otherwise it could change during iteration which could cause a crash
  • fixed another parse-time variant matching bug where the variant-matching algorithm was too aggressive and excluded possible matches at parse time which could result in a false parse-time definitive match even though a better match could be available at runtime
  • fixed a static memory leak when signal handlers are left registered when the qore library terminates
  • fixed static memory leaks and 1 dynamic memory leak in strmul()
  • fixed a crashing bug in handling recursive constant references
  • fixed a bug in the C++ API function HashIterator::deleteKey() when the node's value is NULL
  • fixed time zone/DST calculations for time zone regions with DST with dates before the epoch but after the last DST transition before the epoch
  • fixed a memory error where invalid source expressions referenced in a regular expression substitution expression would cause a crash (ex:
    str =~ s/public (name)/$2/g 
  • fixed a memory error in regular expression substitution where the unconverted string (if not given in UTF-8 encoding) was used when copying source expressions to the target string
  • fixed a bug where a recursive class inheritance tree would cause a crash
  • fixed a bug where a static class method could not access private members of the class