Thoughts about metaprogramming - Discord "Conversion"

Ginger Bill Dec. 21, 2016, 11:21 p.m.
Below is a copy of a discord "conversion" between (mostly) me and ezysigh about problems with compile time execution and maybe a partial solution to the problem. This is recorded here for future reference.
n.b. It includes all my amazing typos, grammatical errors, and swearing.


[10:30 PM] ezysigh: Have you added meta/compiletime stuff yet?
[10:30 PM] gingerBill: Not yet as I'm still designing it.
[10:30 PM] gingerBill: I usually design the feature first then implement it.
[10:31 PM] ezysigh: (sorry, i didn't looka the videos... I kind of burned out of trying to parse everyones 1hr video presentations)
[10:32 PM] ezysigh: Any thoughts on the facilities? Rust macro-ish? Jai #run ish?
[10:37 PM] gingerBill: I thinking Nim-style "templates" or rust macros but I want them to be sane and act like a superset of procedure.
[10:38 PM] gingerBill: Jai #run was original thought of and I think it's a clever idea but it has huge fundamental problems which I discuss here: https://gist.github.com/gingerBill/98fe5117a7ba94437a0e7132eb25007f
[10:38 PM] gingerBill: Metaprogramming is a difficult tool to get just right.
[10:39 PM] gingerBill: I don't want something too complicated as it would defeat the point of the language. But I don't want none as that's too limiting.
[10:40 PM] gingerBill: I have introspection already and that's pretty useful already but that's a runtime facility (but I could extended some of the features to be compile time features).
[10:45 PM] ezysigh: If you're using some kind of VM to run the code, why can't you do ptr fixup when populating the .DATA section?
[10:46 PM] ezysigh: e.g. isn't this just the standard memdump serialization problem?
[10:46 PM] gingerBill: What about something like malloc()?
[10:46 PM] gingerBill: Or an external function pointer?
[10:46 PM] gingerBill: or memory mapped file?
[10:47 PM] gingerBill: How the fuck do you handle those edge cases?
[10:50 PM] ezysigh: I guess I'm assuming that you can't do everything compile time... that is, if you malloc() you're going to have to let the vm know where it is and how large it is if you want it to live to runtime. If you mmap(), forget it, unless you copy into some area you know is going to be preserved for runtime, similar for ext fnptr, if has to reify into a linker ref somehow.
[10:50 PM] gingerBill: Exactly my point.
[10:50 PM] gingerBill: It's a very powerful feature but extremely delicate.
[10:51 PM] ezysigh: I think you can make that line super explicit and get away with it. Like you can run an opengl application compiletime... but... unless your app tells the vm explicitly.. nothing gets saved for tunime
[10:51 PM] ezysigh: I believe that's basically what jai does
[10:52 PM] ezysigh: it's not like the opengl context refs live to runtime or something.
[10:52 PM] gingerBill: Thanks for a fucking briliant idea.
[10:53 PM] ezysigh: hey, no problem! :smiley:
[10:54 PM] gingerBill: A solution would be to be explicit about which global memory can saved from the CTE stage.
[10:54 PM] gingerBill: `#saved_cte var some_global int;`
[10:55 PM] gingerBill: However, this does mean some types would not be allowed e.g anything backed by a pointer internally.
[10:55 PM] ezysigh: well, if you know the layout, you can allow those too... you just have to know how to copy the memory associated with a type... if you know enough to move it and it's referents from one place to another, you know how to preserve it.
[10:56 PM] gingerBill: Not necessarily.
[10:56 PM] ezysigh: It's basically a serialization problem, is it not? I'm running this thing in one context (compiletime).. then I want to take the data in that context and transport it to a new context (runtime)
[10:57 PM] gingerBill: Partially. Somethings cannot be serialized.
[10:57 PM] gingerBill: And that's the problem.
[10:57 PM] ezysigh: and since the contexts have some intimate knowledge, you can take speedy shortcuts...
[10:59 PM] ezysigh: The fnptr problem, for example is just tracing the ptr to a root fn that gets turned into a linker ref (at worst), you know it's a fnptr..since you're the compiler :smiley:
[11:00 PM] gingerBill: The problem is that the calling convention in my compiler will have to act exactly like the C calling convention as those pointers could be passed to external programs.
[11:00 PM] gingerBill: e.g. C's qsort
[11:03 PM] gingerBill: I've thought about this problem extensively and it's a very difficult problem, indeed.
0 comments

The Metaprogramming Dilemma

Ginger Bill Nov. 28, 2016, 7:38 p.m.
Designing this language has been difficult but fun. Two of the original goals of this language were simplicity and metaprogramming however, these together could be an oxymoron. But before I explain why, I first need to explain what I mean by "metaprogramming".

Metaprogramming is an "art" of writing programs to treats other programs as their data. This means that a program could generate, read, analyse, and transform code or even itself to achieve a certain solution. The approaches of metaprogramming can be split into a few distinct categories:

  • Introspection (and Reflection for OOP languages)
  • Compile Time Execution (CTE)
  • Template Programming
  • Macros (Textual and Syntactic)
  • Parametric Polymorphism ("Generics")

Many languages have metaprogramming functionalities: C has textual macros; C++ has textual macros, a functional templating language, and rudimentary introspection; Nim has all of the above; Go has external textual "templates/macros". Each approach has its advantages and disadvantages but all can be used together to achieve different solutions and results.

Introspection is already part of the language and is a functionality I think is necessary in a "modern" language. It is needed to have something like a "type-safe printf" (at runtime) and the ability to serialize data with ease. However, introspection does require extra memory to be stored for the type information. (n.b. Reflection is only appropriate for object oriented programming language which this language is not.)

Compile Time Execution (CTE) is an idea that I've been pondering for a while and it's already part of Jon Blow's language, Jai. It would be a stage of the compiler which runs any Odin code the user requests before the creation of the executable. The data modified and generated by this stage will be used as the initialization data for the compiled code. I have come to the conclusion that this CTE stage would be required to be ran through an interpreter to achieve the results needed. However there are few problems with this CTE stage. The main problem being: pointers will point to invalid memory addresses. This is because the memory space of the interpreter in completely different to the memory space of the executable (compiled code). Numerous types are stored with pointers internally and these values would be invalid at runtime. Due to this problem (and few others), this powerful feature becomes extremely delicate.

Templates can be thought of a subset of macros. Templates are usually a simple substitution mechanism that operate on the Abstract Syntax Tree (AST). Macros on the other hand, could be a simple text substitution system (akin to C's preprocessor) or even a compile AST modification and generation tool (similar to Lisp or Nim). Templates could even be an entire language built into the language (like C++'s templates). This makes them both extremely powerful but also "magic".

Parametric polymorphism, or commonly referred to as "Generics" (which is a very "generic" name too), is the ability to duplicate certain "snippets" of code that have a similar structure but different types/names/etc. A basic example is a generic sorting function which accepts an array of a certain type and a sorting function for that specific type. In a language such as C, this would either have to achieved through code duplication (copy&paste or macros), or through void pointers to remove the type information. The former method can become cumbersome and prone to mistakes while the latter, removes a lot of type safety and prevents compiler optimizations. In a language that does have "generics", this "problem" can be solved. However, the problem hand is very really a generic problem and "genericizing" the problem doesn't actually solve the original problem. "Generics" can be emulated through the use of templates and macros which means that it may not need to be a built in feature of a language.

This now brings me to the dilemma. How far do I want to go with metaprogramming in this language? How far can I go whilst keeping Odin a simple language? Or is the very definition of metaprogramming not simple? Or should metaprogramming be left to an external (standardized) tool (like `go generate`)?


- Bill




3 comments

Odin v0.0.3c

Ginger Bill Nov. 23, 2016, 3:07 p.m.
[Odin v0.0.3c](https://github.com/gingerBill/Odin/releases/tag/v0.0.3c)

What's New

  • Rewritten in C99 (from C++)
  • `nil` - Zero value for all types, except:
    • integers
    • floats
    • strings
    • booleans
  • Maybe types (nillable types)
    • `m: ?int; ...; i, ok := m?;`
  • `match type` for `any`
  • `union_cast`
    • `x, ok := var union_cast Foo.Bar`
  • Backend and stability improvements

Bug fixes
  • `#import "..." as .` will not act like `#load` and keep imported entities local to that file
  • Initialize global memory at compile time if possible

  • Fix enums with duplicate entries crash
  • Remove some duplicate error messages

Linux and OSX

Coming Soon™

Enjoy!
0 comments

The Odin License

Ginger Bill Nov. 23, 2016, 1:45 p.m.
Copyright (c) 2016 Ginger Bill. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
0 comments