joelschumacher.de Blog

joelschumacher.de Blog 2025-02-07T20:03:00+0000 https://joelschumacher.de/blog Joel Schumacher Stop Overcomplicating Parsing https://joelschumacher.de/posts/stop-overcomplicating-parsing 2025-02-07T20:03:00+0000 <h2 id="introduction">Introduction</h2> I have been working on a little programming language for the last couple of days and the software and theory around parsing is just eternally frustrating. People have built mind-boggling theory temples around this and I think nothing attracts architecture astronauts like parsing. I not convinced that most of it is not necessary and it is not actually this difficult. You read about <code>LL(k)</code> parsers, <code>LL(*)</code> parsers, <code>LL(1)</code> parsers, <code>LR(k)</code> parsers, <code>LALR</code> parsers, <code>SLR</code> parsers, <code>GLR</code> parsers, Earley parsers (all of these are real) and more. It makes your head spin and barf all the <code>LOL(k)</code> parsers right back onto the screen. I keep thinking &ldquo;no one cares dude, just parse the thing&rdquo;. We haven&rsquo;t just lost bright minds to parsing theory, but (the real tragedy) countless of people have been discouraged to parse something by the overwhelming overcomplication of the subject. It&rsquo;s so silly to me that people that know nothing about parsing just parse stuff with recursive descent parsers and when you go to the really complicated languages and the really large parsers (JavaScript V8, TypeScript, Rust, CPython, GCC), you end up having to use recursive descent again. It&rsquo;s kind of weird there is a whole science inbetween. Maybe the stuff in the middle is not really that useful? <h2 id="recursive-descent">Recursive Descent</h2> For everyone who doesn&rsquo;t know much about parsing: A recursive descent parser is just the thing you would write down if you didn&rsquo;t know anything about parsing. Consider this simple grammar (<a href="https://de.wikipedia.org/wiki/Erweiterte_Backus-Naur-Form">EBNF</a>-like): <code>none action: ("run" | "stop") (flag | arg)* flag: "--verbose" | "--force" arg: WORD</code> which defines a language that has <code>action</code>s, which start either with <code>"run"</code> or <code>"stop"</code> and are followed by any number of <code>flag</code>s or <code>arg</code>s. <code>flag</code> can either be <code>--verbose</code> or <code>--force</code> and <code>arg</code> must be a <code>WORD</code> (whatever that is, it doesn&rsquo;t matter). You would just write it with each rule being a function, sequences just being code after other code and choices (<code>|</code>) being an <code>if/else</code>: ```python def parse_action(parser): action = Action() parser.skip_whitespace() if parser.input.startswith(&ldquo;run&rdquo;): action.verb = &ldquo;run&rdquo; elif parser.input.startswith(&ldquo;stop&rdquo;): action.verb = &ldquo;stop&rdquo; else: return None while parser.has_input(): parser.skip_whitespace() flag = parse_flag(parser) if flag: action.flags.append(flag) continue arg = parse_arg(parser) if arg: action.args.append(arg) return action ``` Ignore some <code>parser.skip_whitespace()</code> I might have missed and pretend <code>parse_flag</code> and <code>parse_arg</code> look mostly like that. You don&rsquo;t even have to think much when you write a parser like that. There is no science here. You are just writing it out. And I didn&rsquo;t even need a <code>ROFL(69)</code> parser or whatever. If you don&rsquo;t know anything and are humble enough, you might think &ldquo;but maybe this is slow&rdquo; (because it&rsquo;s dumb/straightforward), but no, recursive descent is often one of the fastest ways to parse things. Maybe there are languages / grammars you can&rsquo;t parse with it? No, not even that. Recursive descent is one of the only popular algorithms that can parse context sensitive languages. Other parsers have problems with error recovery (mostly a meme as well btw. - <a href="#error-recovery">see below</a>) or with reporting good error messages. Even that is trivial with recursive descent parsers. And they are also comparatively easy to debug. It can do more, it can do it faster, it can do it better, it can do it easier. What the hell is everyone doing? <h3 id="expressions">Expressions</h3> The actual real downside of recursive descent parsers is parsing expressions. You can do it, but encoding operator precedence is tricky and non-intuitive (you need a separate parsing function for each precedence level). And if you do it that way, it tends to do a lot of backtracking, which ends up being slow (worst case exponential time - blegh). People have come up with <a href="https://en.wikipedia.org/wiki/Packrat_parser">packrat parsing</a>, which caches some of the intermediate results during parsing, which reduces it to linear time, but this honestly also makes my &ldquo;overcomplication&rdquo; senses tingle. The cool thing about recursive descent is that it is so ad-hoc that you can easily combine it with other algorithms, for example algorithms that are made specifically to parse expressions with different operators that have different precedence levels. One of the best (and coolest) algorithms to do this is &ldquo;<a href="https://en.wikipedia.org/wiki/Operator-precedence_parser#Pratt_parsing">Pratt parsing</a>&rdquo; (I recommend this blog post: <a href="https://eli.thegreenplace.net/2010/01/02/top-down-operator-precedence-parsing">link</a>). So if recursive descent is the best for almost anything and Pratt parsing is the best at parsing expressions and you can combine them well, why not just use those two? I wrote a parser for a small language recently in a couple days that does exactly this and it&rsquo;s great. I cannot find a single parser generator that does this and it confuses me to no end. <h3 id="actual-downsides">Actual Downsides</h3> The actual problem I am willing to acknowledge is that recursive descent parsers can remove ambiguity from an ambiguous grammar. Ambiguous grammars are grammars which allow that an input is parsed in different ways. This is mostly bad. And if you don&rsquo;t notice that your grammar is ambiguous and write a recursive descent parser for that grammar (which cannot be ambiguous), then you have kept the ambiguity in your grammar hidden and your grammar does not fit to the parser anymore! All the other fancy parsers that get CS students and academics excited have the cool property that you can derive a parser from an unambiguous grammar and if you have an <code>LL/LR</code> parser or whatever you can derive a grammar from it. Of course recursive descent does not have any nice theoretical properties like that. First of all I think it&rsquo;s silly to choose a parser based on whether it helps you find problems with your grammar. If you want, you can use an existing <code>LR</code> parser to verify that your grammar is unambiguous and then write the recursive descent parser (many people do this). People are rightfully obsessed with removing ambiguities in their grammar, but why not simply come up with a grammar that can&rsquo;t even have them? Like PEG! <a href="https://en.wikipedia.org/wiki/Parsing_expression_grammar">Parsing Expression Grammars</a> replace the choice operator (<code>|</code> before) with an ordered choice which implies a priority of the first (left hand side) option and thereby removes ambiguity. This is actually sick, because if you notice above I started with checking for <code>run</code> and then checked for <code>stop</code>. The grammar didn&rsquo;t actually imply a priority there, but my code did of course. There might be an expression that can get parsed by both <code>if</code>s after all. And if you use a PEG then those different branches in your code DO have a well-defined, correct order. In that way PEGs map to recursive descent parsers more directly. Parsing experts don&rsquo;t like them much and I don&rsquo;t really understand why. I read that it&rsquo;s somehow undesirable that PEGs are unambiguous because you can write stuff that looks correct, but actually isn&rsquo;t and no one complains. I think this is mostly (again) related to operator precedence. Also they use more memory, which I respect as a limitation in certain scenarios, but I have 32G of RAM and that&rsquo;s not even a lot, but more than plenty to parse a couple MBs of code. <h3 id="theory">Theory</h3> Generally I am all for theory. It&rsquo;s important that someone understand these things (and anything else) on a truly deep level. Someone needs to know what is possible and provable and correct, but what confuses me is the ratio of people doing pragmatic parsing and people swooning over parsing algorithms. I feel like the whole world is tricking me into thinking I actually need a lecture to parse text. I don&rsquo;t - you can literally just parse it. I acknowledge that there was a time where all this theory was actually necessary to compile C on an analogue toothbrush in some ancient sumerian assembly. And they did develop some great ideas that are useful to this day (like precedence climbing / Pratt parsing), but honestly I think most of this stuff is just not interesting to the majority of people that want to turn text into trees. <h2 id="lexing">Lexing</h2> I think everyone is overdoing it on lexing / tokenization. I get what lexing is for, but it also feels like it is mainly to make the parser more elegant, because it can just work on tokens and doesn&rsquo;t have to skip whitespace and comments itself. I am not convinced its actually solving a real problem. The fact alone that plenty of parsers do not lex as a separate step proves this. When I see &ldquo;struct&rdquo; in the right place, I know there has to be an identifier next and if there is not, I should error out right then and there. I don&rsquo;t need to figure out generally what could be at any point in the text. It feels like I now have to solve two problems (parsing and figuring out what kind of token a string might be without knowing where it is), when I can solve it in one go and effectively have FEWER problems this way. I heard that lexing separately can actually be faster, because it&rsquo;s cheap to do and it makes parsing simpler, but I heard more often that a &ldquo;scannerless parser&rdquo; (no separate lexing step) is usually faster. And even if having a separate lexing step is slightly faster, I am not convinced it&rsquo;s worth the trouble. <h2 id="parse-generators-existing-libraries">Parse Generators / Existing Libraries</h2> When I see YACC/Bison or Lemon all these other things I cannot believe how people started putting C in their grammar. And it&rsquo;s not just regular C, which is bad enough, but stuff like <code>$$ = system__create_ast_node_ternary(auxil, AST_NODE_TYPE_STATEMENT_IF_ELSE, range__new($0s, $0e), e, s, t);</code>. This is not code, this is garbage. You went the wrong way! Why do none of the popular parser generators just output a damn AST by default? If you write a parser generator, just give me the AST! Don&rsquo;t overdo it, just give me a bunch of these: <code>c++ struct Node { NodeType type; size_t start, end; Node* children; size_t num_children; };</code> Where is the &ldquo;common man&rdquo; tool/library that accomodates practical parsing needs and is not just for parsing nerds? I don&rsquo;t need your help, if I already know how to do it! Do I have to make one myself? I already don&rsquo;t have enough time. Honestly more than half the tools I come across I don&rsquo;t even understand! I have been programming for 20 years and done a bunch of different things. I have Bachelor&rsquo;s degree in Physics. I&rsquo;m stupid, but not completely stupid. Parsing text is really not difficult if you do it the right way, but somehow everything out there is super complicated anyways. I tried getting a grip on all this stuff multiple times through the years, eventually just gave up after a few hours (until recently) and ended up writing a recursive descent parser again, in the beginning not even knowing that is what they were called. <h3 id="expressions_1">Expressions</h3> EVERY LANGUAGE needs expressions. Why do we have to solve this a million times? Why can&rsquo;t I just define a set of sub-languages with their operators and precedences and the parser will work it out? If you build a parser generator there should be a built-in feature for operator expressions. What do you think people use your generators for? Everyone needs this. <h2 id="what-i-want">What I Want</h2> Like I said I gave up many times on parsing. The first tool that changed my mind was <a href="https://github.com/lark-parser/lark">lark</a>. It is so cool in large part because it eats almost any grammar you give it and because it spits out an AST right away, which you can just post-process (or you can customize the AST generation). The problem with lark is of course that it&rsquo;s a Python library and I don&rsquo;t want to call into Python to parse (lol). Also it&rsquo;s pretty slow. I wrote a grammar for the language I am working on and it takes about 10 seconds to parse less than 200 lines, which is, of course, absurd. The thing I want should generate a C parser, because you can integrate those into anything. It should have roughly these features: <ul> <li>PEG grammars with sub-grammars for a Pratt parser for expressions</li> <li>Generates recursive descent parser (with Pratt sub-parser for expressions)</li> <li>Easy way to skip whitespace and comments (you need that almost always - It has to be built-in!)</li> <li>Generates AST right away by default</li> </ul> Something like this (don&rsquo;t hold me to that - I already don&rsquo;t like a bunch of things about this): ``` start: assignment assignment: IDENTIFIER &ldquo;=&rdquo; expr &ldquo;;&rdquo; expr: { primary: IDENTIFIER | NUMBER | &ldquo;(&rdquo; expr &ldquo;)&rdquo; prefix &ldquo;-&rdquo; (10): expr infix &ldquo;-&rdquo; (1): expr infix &ldquo;+&rdquo; (1): expr infix &ldquo;&rdquo; (2): expr infix &ldquo;/&rdquo; (2): expr infix &ldquo;^&rdquo; right (5): expr } IDENTIFIER: [a-zA-Z_][a-zA-Z0-9_] NUMBER: [0-9]+ WS: [ \t\n\r] COMMENT: &ldquo;#&rdquo; (!&rdquo;\n&rdquo;.) &ldquo;\n&rdquo; !ignore COMMENT WS ``` I just want to say which operators I have, what operands they take, their binding powers (in parentheses in the example above) and whether they are prefix/infix/postfix and the parser will do the rest. I feel like a tool like that would be stupidly useful and allow people to prototype programming languages in hours. I really do not understand why there are not 10 of those already. Can someone please make this? I think by the time most people are capable of doing something like this, they already got nerd-sniped into parsing theory and lose sight of the forest for the trees. <h2 id="links">Links</h2> These links say some of the things I said here, but not as clearly distressed and much more measured and intelligent: <ul> <li><a href="https://matklad.github.io/2018/06/06/modern-parser-generator.html">Modern Parser Generator</a></li> <li><a href="https://tiarkrompf.github.io/notes/?/just-write-the-parser/aside1">Just write the #!%/* parser</a></li> <li><a href="https://utcc.utoronto.ca/~cks/space/blog/programming/WhyRDParsersForMe">Why I write recursive descent parsers (despite their issues)</a></li> </ul> <h2 id="foot-notes">Foot Notes</h2> <h3 id="error-recovery">Error Recovery</h3> Most people do not need it. You need it for parsers in IDEs for language servers and stuff like that, but most of the time you just want to error out on the first error. Whenever I get an error in any programming language, I just skip to the first one and doing anything else is (almost) always a waste of time. Imho error recovery is overrated. eBPF File System Monitoring https://joelschumacher.de/posts/ebpf-file-system-monitoring 2025-01-27T12:13:00+0000 This blog post is about something I am currently working on at work, so it will be partly about file system monitoring, somewhat about the file system code in the Linux kernel and a whole lot about eBPF, which is very cool. Say we want to monitor file system changes on a system that hosts thousands of containers for thousands of customers that can be accessed however they like. Most commonly files are modified through SSH/SFTP, but it could be anything they can come up with. It should also work fairly independent of the filesystem. And we want to know about such files changing rather quickly (sub-second would be great), so scanning once a day is not good enough. Of course we are on Linux, because this is where all the cool things happen and also where all the cool kids hang out. We don&rsquo;t just want to know about all operations happening on files, but operations that would have us read something else from them, i.e. that make their contents change. I.e. a write by itself is not interesting, but a write followed by a flush or a close is interesting! <h2 id="existing-monitoring-apis">Existing Monitoring APIs</h2> <h3 id="inotify">inotify</h3> This API is most commonly used for filesystem monitoring on Linux I would say (apart from possibly <a href="https://linux.die.net/man/2/stat"><code>stat</code></a>-ing in a loop), because it doesn&rsquo;t require any capabilities or privileges, it&rsquo;s easy to use and can do most things people care about. Unfortunately you have to watch every file and directory separately - you can&rsquo;t watch recursively! &ldquo;inotify&rdquo; stands for &ldquo;inode notify&rdquo;, because you monitor single inodes. Of course this scales very badly with a large (or very large - as in our case) number of files. On my personal computer with just my stuff on it I have seen Sublime Text exhaust the number of inotify file descriptors multiple times (the limit can be increased, I know). <h3 id="fanotify">fanotify</h3> Instead of monitoring single inodes, you can monitor mount points, which is much better, but with many thousands of containers, which all have their own mount namespace, it might not scale well either. It also requires <code>CAP_SYS_ADMIN</code> to use. <h3 id="ebpf">eBPF</h3> This is gaining popularity in recent years for all sorts of reasons and people have been doing many cool things with it (see <a href="https://github.com/bpftrace/bpftrace">bpftrace</a> and <a href="https://github.com/iovisor/bcc">bcc</a>). It allows you to attach small programs to hooks, which are existing functions in user space or in kernel space. These small programs (<a href="https://bughunters.google.com/blog/6303226026131456/a-deep-dive-into-cve-2023-2163-how-we-found-and-fixed-an-ebpf-linux-kernel-vulnerability">ideally</a>) do not risk the stability or security of your operating system, because they are compiled to a dedicated bytecode, which is verified by the Kernel when it is attached to a hook. The eBPF verifier makes sure that your program terminates in finite time, that stack size is bounded and that it does not access memory out of bounds. More on that in a later section. The most important aspects that make eBPF attractive for high performance file system monitoring is that we can pre-filter events in the Kernel, to reduce the amount of events we have to transfer. We can also mark data as soon as the syscall is issued or returned, which can reduce latency. Also we only require <code>CAP_BPF</code> and <code>CAP_PERFMON</code> to attach a eBPF program, but might require more depending on the program and attach point. After it is attached, we don&rsquo;t need any special permissions in user space to process the events generated by the eBPF program. Of course there is no state about watched items and way fewer layers than for example fanotify (vfs -&gt; fsnotify -&gt; fanotify -&gt; userspace). You can also optimize how exactly events are dispatched. And if you want to add additional information (like the mount namespace, the pid of the process that modified the file, etc.) you can (often) trivially add this information. So there are definitely good reasons it might be cool, let&rsquo;s try it! <h3 id="what-to-trace">What to Trace</h3> Maybe this is not a smart way to think about this, but I considered ways I can edit a file, which syscall sequence this corresponds to and then looked at functions that might be good candidates. I am sure I have forgotten many. <h4 id="operations">Operations</h4> These are the ones I could come up: <ul> <li><a href="https://linux.die.net/man/2/write"><code>write</code></a> a (non-zero) number of times and then <a href="https://linux.die.net/man/2/close"><code>close</code></a> (C API: <code>fwrite</code>, <code>fclose</code>)</li> <li><a href="https://linux.die.net/man/2/write"><code>write</code></a> a (non-zero) number of times and then <a href="https://linux.die.net/man/2/fsync"><code>fsync</code></a> (C API: <code>fflush</code>)</li> <li><a href="https://linux.die.net/man/2/write"><code>write</code></a> a (non-zero) number of times and the kernel does an automatic write-back</li> <li><a href="https://linux.die.net/man/2/writev"><code>writev</code></a></li> <li><a href="https://linux.die.net/man/2/mmap"><code>mmap</code></a>, modify the mapped memory and then call <a href="https://linux.die.net/man/2/msync"><code>msync</code></a></li> <li><a href="https://linux.die.net/man/2/mmap"><code>mmap</code></a>, modify the mapped memory and then call <a href="https://linux.die.net/man/2/close"><code>close</code></a></li> <li><a href="https://linux.die.net/man/2/mmap"><code>mmap</code></a> and the kernel does an automatic write-back</li> <li><a href="https://linux.die.net/man/2/sendfile"><code>sendfile</code></a></li> <li><a href="https://linux.die.net/man/2/splice"><code>splice</code></a></li> <li><a href="https://linux.die.net/man/2/truncate"><code>truncate</code></a></li> <li><a href="https://linux.die.net/man/2/rename"><code>rename</code></a></li> </ul> Now we need to find the functions in the Linux kernel that we need to trace to catch all these different ways a file can be modified. Ideally we want to pick functions that take parameters from which we can easily obtain the information we want to collect. In user space Linux programs tend to interact with files using file descriptors, which are just integers, but internally these will be mapped to <code>struct file</code> objects. File system related system calls then call into VFS (Virtual File System), which is an abstraction layer over all file systems with functions like <code>vfs_write</code>, <code>vfs_close</code> and many more. VFS functions then use <code>file-&gt;f_op</code> of type <code>struct file_operations*</code>, which is just a collection of function pointers like <code>read</code>, <code>open</code>, <code>flush</code> pointing to the actual functions that implement the specific file system (of which there are many). If for example we traced on the <code>close</code> system call itelf, we would just get the file descriptor in our eBPF program and we would have to get all the useful information from that ourselves, which is often not even possible in a restricted eBPF program, because you cannot call arbitray kernel functions. So it makes sense to trace on <code>vfs_close</code>, which takes a <code>struct file</code> instead. Unfortunately this function particular is inlined! And if there is no jump to that function, we cannot hook into it. The next interesting function in the call chain is <code>filp_close</code>, which calls into <code>f_op</code> directly, so it&rsquo;s the last function to trace that is not already part of a specific file system. This also means that flushing on close is not handled by VFS, but by the file system itself and we can&rsquo;t get away by just tracing the flush operation, but we have to trace <code>filp_close</code>. The next way to modify a file we consider is <code>fsync</code>. <code>fsync</code> calls into <code>vfs_sync</code>, which calls into <code>vfs_sync_range</code>, which then calls into <code>f_op</code>, so <code>vfs_sync_range</code> is what we want to trace additionally. Similarly the automatically triggered write-back of dirty pages and <code>msync</code> all end up in <code>vfs_sync_range</code>, but I won&rsquo;t go into detail for all of these. You may dig through the Kernel source yourself to verify. You would think <code>writev</code> just does <code>write</code>, but that&rsquo;s not true. It calls into <code>vfs_writev</code>! <a href="https://linux.die.net/man/2/sendfile"><code>sendfile</code></a> calls <code>do_splice_direct</code> if the output is a file and eventually ends up calling <code>do_splice_from</code>, which calls <code>fs_op-&gt;splice_write</code> directly without a VFS layer function anywhere inbetween. <a href="https://linux.die.net/man/2/splice"><code>splice</code></a> also ends up in <code>do_splice_from</code>. <code>truncate</code> and <code>rename</code> call into <code>vfs_truncate</code> and <code>vfs_rename</code> respectively. This leaves us with the following functions to trace: <ul> <li><code>filp_close</code></li> <li><code>vfs_sync_range</code></li> <li><code>vfs_writev</code></li> <li><code>do_splice_from</code></li> <li><code>vfs_truncate</code></li> <li><code>vfs_rename</code></li> </ul> I had hoped there were fewer. <h2 id="tricky-bits">Tricky Bits</h2> <h3 id="paths">Paths</h3> As a human I cannot deal well with the sort of data that is passed to e.g. <code>filp_close</code>. That&rsquo;s just a <code>struct file*</code>. I want more - I want paths. I started with <code>filp_close</code> and struggled immensely to get the absolute path from the <code>struct file*</code> only to learn later that this was one of the easy functions. Essentially a <code>struct path</code> in the Linux kernel is a <code>struct dentry *dentry</code> and a <code>struct vfsmount *mnt</code>. The <code>dentry</code> contains the name of the filesystem object and a pointer to its parent <code>dentry</code>. This is not enough, because the end of this linked list is not the filesystem root, but the root of the mount point. This is why you need the <code>struct vfsmount *mnt</code> as well to get to the full path. So essentially you have to read the kernel source and figure out ways to get to the data you want from the data you have. It&rsquo;s not impossible, which is why it&rsquo;s not listed under problems, but it is tricky, hence the section this piece of text is placed in. <h3 id="the-verifier">The Verifier</h3> A good chunk of the time, especially in the beginning, I spent on trying to please the eBPF verifier. After you learn how it works and some tricks, it stops being a constant annoyance. As you might know it is hard (read: <a href="https://en.wikipedia.org/wiki/Halting_problem">in general impossible</a>) to say whether a program terminates. So the eBPF verifier imposes some restrictions on the kinds of programs you write. As the verifier has to enumerate all possible paths of your program and analyze them individually, you may only produce a finite number of paths. In practice this number doesn&rsquo;t just have to be finite, but bounded (1 million instructions in total at the time of this writing). Essentially this means that you mostly have to use bounded loops and if you can&rsquo;t tell that your program is finishing in finite time, the verifier will not either. So instead of this: <code>c while ((node = node-&gt;next)) { // ... }</code> you should probably write this: <code>c for (int i = 0; i &lt; MAX_NUM_NODES; ++i) { /// ... node = node-&gt;next; }</code> I haven&rsquo;t had much trouble with this limiation and on top of that the verifier has become much smarter about unbounded loops in recent kernel versions. The really tricky bit are memory accesses though. In regular C code you would confidently write this: <code>c if (offset + size &lt;= BUF_SIZE) { bpf_probe_read_kernel(buf + offset, size, some_kernel_ptr); }</code> That&rsquo;s safe, right? Well, go ask the verifier - he will not agree. The verifier keeps track of every register and it&rsquo;s possible value range, but it will not track the range of a value in relation to another. So if for example the only thing the verifier knows about a value <code>offset</code> is that in all possible universes it might be in the range <code>[0, 4096]</code>, another value <code>size</code> is also in <code>[0, 4096]</code> and <code>BUF_SIZE</code>, the size of the buffer <code>buf</code>, is <code>4096</code>, then there is no way to encode the constraint from <code>if (offset + size &lt;= BUF_SIZE)</code>. Consequently it will complain that <code>bpf_probe_read_kernel</code> might cause an out of bounds memory access, because <code>offset</code> and <code>size</code> are both in <code>[0, 4096]</code>. After all it is possible that both <code>offset</code> and <code>size</code> are <code>4096</code>. In practice this means that you need to double your buffers a lot, so that e.g. if <code>offset</code> and <code>size</code> are in <code>[0, 4096]</code>, the size of <code>buf</code> should be <code>8192</code>. Then you can never have an out of bounds memory access. You also need to sprinkle your code with macros that explicitly limit the range of values, like this: ```c <h1 id="define-path_buf_size-4096">define PATH_BUF_SIZE 4096</h1> <h1 id="define-limit_path_buf_sizex-x-path_buf_size-1">define LIMIT_PATH_BUF_SIZE(x) ((x) &amp;= (PATH_BUF_SIZE - 1))</h1> // &hellip; LIMIT_PATH_BUF_SIZE(path_start); ``` When I started working on this I thought I could get away with just programming carefully, but you need to look at the eBPF assembly and see what&rsquo;s actually going on to understand some of the errors. For example look at this funny little thing: <code>c name_len++; if (name_len &gt; path_start) { break; }</code> <code>{ guess_lang = False } ; name_len++; 97: (07) r2 += 1 ; R2_w=scalar(umin=1,umax=4294967296, ; var_off=(0x0; 0x1ffffffff)) 98: (bf) r1 = r2 ; R1_w=scalar(id=6,umin=1,umax=4294967296, ; var_off=(0x0; 0x1ffffffff)) ; R2_w=scalar(id=6,umin=1,umax=4294967296, ; var_off=(0x0; 0x1ffffffff)) 99: (67) r1 &lt;&lt;= 32 ; R1_w=scalar(smax=9223372032559808512, ; umax=18446744069414584320, ; var_off=(0x0; 0xffffffff00000000), ; s32_min=0,s32_max=0,u32_max=0) 100: (77) r1 &gt;&gt;= 32 ; R1=scalar(umax=4294967295, ; var_off=(0x0; 0xffffffff)) ; if (name_len &gt; path_start) { 101: (25) if r1 &gt; 0x1000 goto pc+2051 ; R1=scalar(umax=4096,var_off=(0x0; 0x1fff))</code> Here <code>name_len</code> is stored in <code>r2</code>. In instruction 97 it is incremented. <code>path_start</code> is just <code>4096</code>. The <code>umin</code>/<code>umax</code> values in the comments indicate the known ranges of the registers and are just another way to display <code>var_off</code>. The problem with this snippet is that the verifier will not update the range of <code>name_len</code>, despite <code>if (name_len &gt; path_start) break</code>. In general it will exclude paths that you <code>break</code> from and update value ranges accordingly. This is because here <code>name_len</code> is a 32-bit unsigned int and <code>r2</code> is a 64-bit register, so the increment might push <code>r2</code> out of 32-bit range. To perform the check in instruction 101, the compiler has to bring <code>r1</code> back into 32-bit range by shifting it up (instruction 99) and down again (instruction 100). Unfortunately the compiler decided to use another register (<code>r1</code>) for this and as a result the verifier doesn&rsquo;t know that it should update <code>r2</code>. It&rsquo;s possible to program around these things, but it can be very annoying and it will frequently bug you. <h2 id="actual-problems">Actual Problems</h2> While those things above need getting used to and are managable, there are also problems that aren&rsquo;t solved well with elbow grease. <h3 id="paths_1">Paths</h3> I mentioned that <code>filp_close</code> was one of the easy functions, implying there might more difficult ones. The one I had in mind here is <code>vfs_rename</code>. It takes a <code>struct renamedata*</code>, which looks like this: <code>c struct renamedata { struct mnt_idmap *old_mnt_idmap; struct inode *old_dir; struct dentry *old_dentry; struct mnt_idmap *new_mnt_idmap; struct inode *new_dir; struct dentry *new_dentry; struct inode **delegated_inode; unsigned int flags; } __randomize_layout;</code> Just <code>dentry</code>s and <code>inode</code>s. But to build a full path I need a <code>vfsmount</code>! I haven&rsquo;t found a way to get it. I considered other functions to trace, but I haven&rsquo;t found a good solution. You cannot call into arbitrary functions in the kernel from eBPF, which is why it&rsquo;s not as easy as just doing what you would do in the kernel. My current workaround is to just traverse the <code>dentry</code> list and build a path up until the mount root. Then include this path in the event passed to userspace and the mount namespace id. As root, this should be enough to uniquely identify the file that was modified. I am sure I am still missing something for when the same filesystem is bind-mounted multiple times. <h3 id="unstable-abi">Unstable ABI</h3> Another reason I have to go with the mount root path is that to traverse up to the parent mount, I need to get the <code>struct mount</code> that contains the <code>struct vfsmount</code>. I use <code>container_of</code> for this, but <code>struct mount</code> is not in a public header, so I have to define a (sufficient) subset myself. This is what I did: <code>c struct mount { struct hlist_head mnt_hash; // list_head -&gt; hlist_head Al Viro 2014 (same size) struct mount *mnt_parent; struct dentry *mnt_mountpoint; // Al Viro 2011 inserted struct vfsmount mnt; // lots more fields, that are partially dependent on compilation flags };</code> I added some comments to note when these fields were last changed. Linux tries to have a stable API (<a href="https://lkml.org/lkml/2012/12/23/75">&ldquo;WE DO NOT BREAK USERSPACE&rdquo;</a>), but it does not have a stable ABI! This means such structs may change. This means your eBPF programs might break on every new kernel release and even if they don&rsquo;t change, your distro might have a good reason to change it. As unstable as the kernel ABI is, your eBPF programs are likely too. Inserting new fields or reordering them somehow is annoying enough, but it&rsquo;s possible refactors take away fields you depend on all together and then you don&rsquo;t just have to recompile your program, but solve your problem again from the beginning. This might incur a significant maintenance burden, especially considering this is usually not a particularly simple task. <h3 id="__randomize_layout"><code>__randomize_layout</code></h3> I purposely left it in the definition of <code>renamedata</code> above to get you guessing! Many structures in the Linux kernel are marked with <code>__randomize_layout</code> so that its members are shuffled during compile time. This makes certain exploits more difficult as the offsets to critical structures are unknown. Unfortunately we depend on some of those offsets! The full version of my small <code>struct mount</code> above is actually marked <code>__randomize_layout</code> as well. Meaning, if struct randomization is enabled in your kernel, there is no way to implement this full path traversal properly in an eBPF program. Fortunately (?) many distros (such as Debian and Ubuntu) disable struct randomization. You can check this by running <code>grep CONFIG_RANDSTRUCT /boot/config-$(uname -r)</code>, which should return <code>CONFIG_RANDSTRUCT_NONE=y</code> if it is disabled. But if you use a distro that has it enabled or have some requirements to enable it, then certain things will be very difficult (or impossible) to do in an eBPF program. I have later realized that smarter people are aware of this issue. You can use <a href="https://www.kernel.org/doc/html/latest/bpf/btf.html">BTF</a> and <a href="https://docs.ebpf.io/concepts/core/">CO-RE</a> (Compile Once - Run Everywhere) to generate headers with the actual struct layouts. I will only need the file system monitoring on distros that have struct randomization disabled, so I haven&rsquo;t looked into it yet. <h2 id="conclusion">Conclusion</h2> eBPF can be used for high performance file system monitoring, but it is possibly unstable and not very portable. If you control the target system very well, that might not be a huge problem. Also it is definitely some trouble to collect all the possible syscalls that might end up modifying a file on your disk and I am not sure I got them all. Even if I did, new ways could be added tomorrow or in a year. It seems efficient and flexible, but fragile, which is a difficult tradeoff. It&rsquo;s the kind of thing where the tradeoff might be worth it today, but with an ongoing maintenance burden, that tradeoff might slowly become less favorable. I am sure I should look into CO-RE and BTF because they alleviate some of my concerns, but I don&rsquo;t think it will clear them all up. Game Engine Requirements https://joelschumacher.de/posts/game-engine-requirements 2025-01-15T20:27:00+0000 <h2 id="introduction">Introduction</h2> A few days ago I found <a href="https://racenis.github.io/tram-sdk/why.html">Tramway SDK</a> on HackerNews and I thought its design decisions and motiviations really resonated with me. I have been keeping similar notes on why I want to make my own engine scattered across my hard drive for years and thought maybe I should write it down and extend it over time as a living document. Of course I know this is not how every engine should be and I certainly don&rsquo;t think there is a huge market for an engine as I am describing it here, but it is what I want. <h2 id="table-of-contents">Table of Contents</h2> <ul> <li><a href="#no-editor-code-first">No Editor / Code First</a></li> <li><a href="#no-intentionally-opaque-shit">No Intentionally Opaque Shit</a></li> <li><a href="#vcs-aware">VCS aware</a></li> <li><a href="#usability-over-architecture">Usability over Architecture</a></li> <li><a href="#target-audience-experienced-programmers">Target Audience: Experienced Programmers</a></li> <li><a href="#fast-iteration-time">Fast Iteration Time</a></li> <li><a href="#small-scope">Small Scope</a></li> <li><a href="#cross-platform">Cross-Platform</a></li> <li><a href="#single-executable">Single Executable</a></li> </ul> <h2 id="details">Details</h2> <h3 id="no-editor-code-first">No Editor / Code First</h3> I hate editors so fucking much. Of course you need to place your objects in the level with an editor, but I don&rsquo;t want to do literally everything in the damn editor. Whatever engine you are today, your UI is way too fucking full of stuff, it&rsquo;s dog shit slow and most of the time just plain weird. Digging through layers of options dialogs and settings makes me want to cry, turn off the computer and never turn it on again. Editor UIs just evolve horribly. Every new thing increases the complexity of everything else. And almost always they are so fucking buggy it makes you lose respect and faith in the software altogether. Instead everything should be handled in code, human-readable text(-ish) files and via CLI tools that can be automated and used in scripts at every step of the way. <h3 id="no-intentionally-opaque-shit">No Intentionally Opaque Shit</h3> I don&rsquo;t want your silly files that are not documented properly anywhere. Even if your stuff is open source, I do not care. I want official specifications or open file formats (even better)! If I need to do some modification to a file, I don&rsquo;t want to modify the engine, but write my own tool. This is how an engine is actually easily extensible! If the engine needs proprietary file formats (which it very likely does, that is normal), they should have proper specifications or ideally be so simple that a brief glance tells you all you need to know. Also I don&rsquo;t want any files I&rsquo;m not supposed to understand, because they should just work (or something). They don&rsquo;t - constantly. Every file should be for the developers to consume and produce, no other garbage please. <h3 id="vcs-aware">VCS aware</h3> When I say VCS aware of course I mean git. If your engine works great with version control and that means &ldquo;just Perforce&rdquo; or god forbid some proprietary thing, that doesn&rsquo;t count. I DO NOT CARE. Every file the engine tools write to or reads from the disk should be VCS friendly (i.e. doesn&rsquo;t diff randomly and auto-merges well). This sort of implies that all interesting files are either assets (like textures, sounds, meshes) or human-readable text files. I am human, let me read my files! This also ties into the previous section - I don&rsquo;t want to think about whether a file should or should not be added to VCS. It should be obvious. <h3 id="usability-over-architecture">Usability over Architecture</h3> I have wasted quite a few years building engines with grand concepts and beautiful architectures that ended up not being very useful or putting road blocks in my way while doing simple things, so that I could make complex things easier. There are plenty of architecture astronauts that build abstract temples made of archetypal ECSs with relationships, queries and schedulers and I think most of you are going too far. If you built the most amazing architecture and no one has made a real game with it in 5 years (there are multiple popular examples of this), then you should consider if you went too far as well. Abstractions make sense and are necessary, but they should solve a problem and too often they solve problems, that (approximately) no one really has yet. So the engine should have just as much abstraction as necessary. Imho if there is any grand, clever architectural idea in your engine you might brag about on your website, that is a big negative, not a positive! <h3 id="target-audience-experienced-programmers">Target Audience: Experienced Programmers</h3> So many engines are built for people that just get started and don&rsquo;t know what they are doing yet. It&rsquo;s great those engines exist, but if you have some general experience with game development, you tend to fight a lot against the &ldquo;noob-layer&rdquo; put on top of everything. My engine should be made for experienced programmers (not artists or designers), that have made (multiple) games before. You should know how to use a computer and you know how it works and you are not afraid of code or the terminal. <h3 id="fast-iteration-time">Fast Iteration Time</h3> It should never take more than one second (non-negotiable) to try out a change you have made to game code (engine code is not included). I have worked on some C++ games, where the iteration times went up to 30s and it killed curiosity to try out things, the joy in development and ultimately the fun. I have also worked on software that takes 15min to (incremental!) build and you just end up adopting weird habits to avoid rebuilding. As soon as you develop a hint of laziness about experimenting with your game, there is no way you will make it as good as you can. In practice this means you either do something crazy (shared libs with game code in C) or you use scripting (Lua is my strong preference). <h3 id="small-scope">Small Scope</h3> It&rsquo;s purposely made for game jam games and small/medium indie games. Making AAA stuff possible should never be a design goal or constraint. The engine is not does-it-all. If you make a bigger game or want to do something special, it&rsquo;s guaranteed you need your own build of the engine (with significant modifications). It should include a basic version of everything and a fancy version of absolutely nothing. Only the simplest usable thing is built-in. Everything gets sucky over time, because it grows and bloats and tries to solve every problem for everyone. So you have to commit to not add every feature that might be useful to someone and maybe not even fix every bug. You should try to keep it small aggressively, but of course add everything that almost everyone needs. Half the reason I hate most engines is because they are &ldquo;do everything&rdquo; tools. Even if you use something big like Unity or whatever, if you make a serious game, you eventually have to make your own specialized tools anyways and write your own libs and everything else. They include the feature-complete option already, but they are so &ldquo;feature complete&rdquo; (read: bloated), that it gets simpler to just remake the damn thing with 10% of the features, because you gain stability and speed. Making a single tool that works for ten different people is often less work than making a tool that works for a single person once. And if you have those ten tools, they are faster and easier to use and more tailored the users needs. Users of the engine should be expected to replace certain parts according to their needs. Reinventing the wheel exists, but people try to avoid it too much and they should &ldquo;reinvent&rdquo; it more often. <h3 id="cross-platform">Cross-Platform</h3> The engine can be built on Windows and Linux and both are equally important. And you can build your game for Windows, Linux, Mac and Web out of the box. <code>engine build --release mygame/</code> should give you distributables for all platforms without much fuss. Increasingly many game jams require all submissions to be playable in the browser, therefore a web export is very important. <h3 id="single-executable">Single Executable</h3> Yes, everything you want to do is in that executable. This makes for an easy download and easy experiments. You can run your debug builds with it, you can create your release builds with it, you can convert your assets. Everything the engine needs to do, is in that single executable. This is not in contrast to what I said in <a href="#small-scope">Small Scope</a>, because I am talking about a means of software distribution only. This is about the stuff that is part of the engine anyways. <h2 id="final-note">Final Note</h2> I didn&rsquo;t expect this to bring out such anger, but I think it stems from bad experiences I have made with existing engines and it is likely this anger that drives me towards making my own. I should also mention that <a href="https://love2d.org/">löve</a> comes very close to almost all of these requirements. I&rsquo;ve used and enjoyed it for a very long time, but it&rsquo;s mostly geared towards 2D games and most of the games I want to make today are 3D games. Also there are a few smaller things I wish it did differently (only user directory is writable, there should be official build tools, etc.). Yet Another C++ Coroutine Tutorial https://joelschumacher.de/posts/yet-another-cpp-coroutine-tutorial 2023-04-24T18:34:24+0000 <h2 id="introduction">Introduction</h2> I know, everyone has written one. My grandma recently published hers, after telling me how bad she finds the one her hairdresser wrote. I suspect many people have similar experiences of pouring a dozen of these resources into their heads and still wondering how the HELL these fucking things work and when they finally figure it out, they want to write it down too - &ldquo;properly&rdquo; this time. This is exactly what I will do in this blog post as well. Let me note that some of the sentences in this post are frankly just ridiculous. You have to read them carefully, word by word, and think hard about what they say. I think this topic is truly complicated and there is no other way but understanding that complexity. Hopefully this post can help you with that and lead to somewhere where you can operate coroutines in code that does something useful and interesting. If not and you find certain questions not adequately answered or entirely unadressed, feel free to write me an email and help me make this tutorial better. I first tried to write this tutorial with some small classes that fake doing async IO, but it became hard to extend the examples to something meaningful, so instead I will use my own async IO library <a href="https://github.com/pfirsich/aiopp">aiopp</a>. Reading along should be enough and this blog post is intended to be self-contained, but if you want to run some of the code, look around in the <a href="https://github.com/pfirsich/aiopp">aiopp</a> repository (especially <code>examples/</code>). In fact most of the snippets in this blog post are parts or simplified versions of code in that repository. For this tutorial I will assume that you have some idea of what coroutines are, but to summarize briefly, they are functions that can suspend themselves, meaning they stop themselves without returning, even in the middle of their body and then can later can be resumed, continuing execution at the point they suspended from earlier. So when in the past you would write something like this (lots of complications and error handling omited): <code>cpp io.read(fd, buffer.data(), buffer.size(), [](std::error_code ec, size_t readBytes) { buffer.resize(readBytes); handleReadData(buffer); });</code> With coroutines you would just do this: <code>cpp std::vector&lt;uint8_t&gt; buffer(1024, 0); const auto [ec, readBytes] = co_await io.read(fd, buffer.data(), buffer.size()); buffer.resize(readBytes); handleReadData(buffer);</code> It looks like synchronous code, but <code>co_await</code> will suspend the execution of the function (coroutine) that contains this code, your program does something else (like waiting for the read to complete) and when the read is finished, the coroutine will be resumed and the read can be handled. You don&rsquo;t really have to worry about whether the operation might outlive a buffer or about sharing ownership of the object that initiated the operation (a coroutine) with the IO operation. Of course you have to worry about it once, when you build the framework and that&rsquo;s bad enough, but after that it will not invite you repeatedly to mess it up, like callback based async code does. In a conventional function the variables live on the stack and when the function returns, the whole stack frame is thrown away. A stack frame of a function will never outlive the stack frame of the function that called it. With a coroutine of course this is not possible, so the coroutine state is typically allocated on the heap. <h2 id="my-first-coroutine">My First Coroutine</h2> Let&rsquo;s start with the minimal coroutine example I can come up with, explain it and build it up: ```cpp <h1 id="include">include <coroutine></h1> class BasicCoroutine { public: struct Promise { BasicCoroutine get_return_object() { return BasicCoroutine {}; } <pre><code> void unhandled_exception() noexcept { } void return_void() noexcept { } std::suspend_never initial_suspend() noexcept { return {}; } std::suspend_never final_suspend() noexcept { return {}; } }; using promise_type = Promise; </code></pre> }; BasicCoroutine coro() { co_return; } int main() { coro(); } <div class="codehighlight" style="background: #282828"><pre style="line-height: 125%;"><code>&lt;em&gt;(this example is self-contained)&lt;/em&gt;&lt;/p&gt; &lt;p&gt;Not very small, I know. And it doesn&amp;rsquo;t do anything either - Already love them! I promise I&amp;rsquo;ll build it up to something useful, but let&amp;rsquo;s first analyze this snippet. &lt;code&gt;coro&lt;/code&gt; is a coroutine simply because it contains the keyword &lt;code&gt;co_return&lt;/code&gt;. If a function contains &lt;code&gt;co_await&lt;/code&gt;, &lt;code&gt;co_return&lt;/code&gt; or &lt;code&gt;co_yield&lt;/code&gt;, it&amp;rsquo;s a coroutine. The return type, even though it doesn&amp;rsquo;t do much, has to be this complicated, because coroutines are rewritten by the compiler in a certain way. This is what a coroutine is rewritten as:&lt;/p&gt; &lt;p&gt;&lt;code&gt;cpp ReturnType someCoroutine(Parameters parameter) { auto* frame = new coroutineFrame(std::forward&amp;lt;Parameters&amp;gt;(parameters)); auto returnObject = frame-&amp;gt;promise.get_return_object(); co_await frame-&amp;gt;promise.initial_suspend(); try { &amp;lt;body-statements&amp;gt; } catch (...) { frame-&amp;gt;promise.unhandled_exception(); } co_await frame-&amp;gt;promise.final_suspend(); delete frame; return returnObject; }&lt;/code&gt;&lt;/p&gt; &lt;p&gt;So as we said earlier, the coroutine frame, called the &amp;ldquo;coroutine state&amp;rdquo; is allocated on the heap, the function parameters of the coroutine and later all local variables and temporaries are saved inside it. Another object is also allocated inside it, which is called the &amp;ldquo;&lt;em&gt;promise&lt;/em&gt;&amp;rdquo;. There are different ways the compiler can select a promise type for a coroutine, but what we will use for now is that it will look for &lt;code&gt;ReturnType::promise_type&lt;/code&gt; (see also the minimal example). It might be worth mentioning, that there are also &lt;code&gt;coroutine_traits&lt;/code&gt; which you can use to associate a promise type with any coroutine return type, e.g. standard library types. Most of the logic of the coroutine (apart from it&amp;rsquo;s body) is handled through the promise. When we later suspend the coroutine, the coroutine state will also contain information about the point we suspended from, so we can resume to that point later.&lt;/p&gt; &lt;p&gt;Then the return object of the coroutine is created through the promise. &lt;code&gt;get_return_object&lt;/code&gt; has to return something that is convertible to the return type of the coroutine.&lt;/p&gt; &lt;p&gt;The promise member functions &lt;code&gt;initial_suspend&lt;/code&gt; and &lt;code&gt;final_suspend&lt;/code&gt; return &lt;em&gt;awaitables&lt;/em&gt; and provide an opportunity to control the behavior of the coroutine at the start and the end. &lt;em&gt;Awaitables&lt;/em&gt; are just things you can &lt;code&gt;co_await&lt;/code&gt;. Depending on the return type of your coroutine, the coroutine itself might be awaitable, but other things can be awaitable as well. I would say that most things you &lt;code&gt;co_await&lt;/code&gt; are not coroutines (e.g. &lt;code&gt;io.recv()&lt;/code&gt; above is not). I will explain awaitables in detail shortly but for now know that &lt;code&gt;co_await std::suspend_never&lt;/code&gt; is pretty much a noop and terminates immediately, without suspending the coroutine - it seems the name has been chosen well! A common choice for &lt;code&gt;initial_suspend&lt;/code&gt; is also &lt;code&gt;std::suspend_always&lt;/code&gt;, which will result in your coroutine suspending immediately after creating the return object. This would correspond to a &amp;ldquo;cold-start&amp;rdquo; or &amp;ldquo;lazy&amp;rdquo; coroutine, which you will have to resume at least once first before it starts executing its body. The coroutine from our minimal example is analogously called a &amp;ldquo;hot-start&amp;rdquo; or &amp;ldquo;eager&amp;rdquo; coroutine.&lt;/p&gt; &lt;p&gt;Additionally upon reaching &lt;code&gt;co_return &amp;lt;expr&amp;gt;;&lt;/code&gt; &lt;code&gt;promise.return_void()&lt;/code&gt; or &lt;code&gt;promise.return_value(&amp;lt;expr&amp;gt;)&lt;/code&gt; is called, depending on whether &lt;code&gt;&amp;lt;expr&amp;gt;&lt;/code&gt; is void or not and which function is defined for the &lt;em&gt;promise&lt;/em&gt; type. Note that &lt;code&gt;return_value&lt;/code&gt; also returns &lt;code&gt;void&lt;/code&gt; and it is used to store the value that was &lt;code&gt;co_return&lt;/code&gt;ed inside the promise. If the coroutine returns void, &lt;code&gt;return_void&lt;/code&gt; is also called at the end of the body, even if the coroutine does not contain &lt;code&gt;co_return;&lt;/code&gt; explicitly.&lt;/p&gt; &lt;p&gt;Our next goal is to &lt;code&gt;co_await&lt;/code&gt; something!&lt;/p&gt; &lt;h2 id=&quot;awaitables&quot;&gt;&lt;em&gt;Awaitables&lt;/em&gt;&lt;/h2&gt; &lt;p&gt;Let&amp;rsquo;s start with half a UDP echo server first, that we will then coroutinize (sic): ```cpp&lt;/p&gt; &lt;h1 id=&quot;include-aioppioqueuehpp&quot;&gt;include &amp;ldquo;aiopp/ioqueue.hpp&amp;rdquo;&lt;/h1&gt; &lt;h1 id=&quot;include-aioppsockethpp&quot;&gt;include &amp;ldquo;aiopp/socket.hpp&amp;rdquo;&lt;/h1&gt; &lt;h1 id=&quot;include-spdloggerhpp&quot;&gt;include &amp;ldquo;spdlogger.hpp&amp;rdquo;&lt;/h1&gt; &lt;p&gt;using namespace aiopp;&lt;/p&gt; &lt;p&gt;class Server { public: Server(IoQueue&amp;amp; io, Fd&amp;amp;&amp;amp; socket) : io_(io) , socket_(std::move(socket)) { }&lt;/p&gt; &lt;pre&gt;&lt;code&gt;void start() { receive(); } &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;private: void receive() { receiveBuffer_.resize(1024, &amp;lsquo;\0&amp;rsquo;); io_.recvfrom(socket_, receiveBuffer_.data(), receiveBuffer_.size(), 0, &amp;amp;clientAddr_, &lt;a href=&quot;std::error_code ec, int receivedBytes&quot;&gt;this&lt;/a&gt; { if (ec) { spdlog::error(&amp;ldquo;Error in recvfrom: {}&amp;rdquo;, ec.message()); receive(); return; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt; receiveBuffer_.resize(receivedBytes); spdlog::info(&quot;received: &#39;{}&#39;&quot;, receiveBuffer_); receive(); }); } IoQueue&amp;amp; io_; Fd socket_; std::string receiveBuffer_; ::sockaddr_in clientAddr_; &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;};&lt;/p&gt; &lt;p&gt;int main() { setLogger(std::make_unique&lt;SpdLogger&gt;());&lt;/p&gt; &lt;pre&gt;&lt;code&gt;auto socket = createSocket(SocketType::Udp, IpAddress::parse(&quot;0.0.0.0&quot;).value(), 4242); if (socket == -1) { return 1; } IoQueue io; Server server(io, std::move(socket)); server.start(); io.run(); return 0; &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;} </code></pre></div> The full echo server is here: <a href="https://github.com/pfirsich/aiopp/blob/main/examples/echo-udp.cpp">echo-udp.cpp</a> I hope you can guess well enough what all the aiopp stuff does. Note that I simplified the ownership of some of the objects. Everything is in <code>Server</code> and as long as that lives longer than <code>io.run()</code> runs, we&rsquo;re fine. To build a coroutine version of this application, we need to create an awaitable for <code>recvfrom</code> operations. An awaitable is an object for which <code>operator co_await</code> is defined. There is also another way via <code>promise_type::await_transform</code>, but we won&rsquo;t discuss that in this blog post. Before I explain more, I will show you an awaitable for our <code>recvfrom</code> operation: ```cpp struct IoResult { std::error_code ec; int result; }; class RecvFromAwaitable { public: RecvFromAwaitable(IoQueue&amp; io, int socket, void buf, size_t len, ::sockaddr_in srcAddr) : io_(io) , socket_(socket) , buf_(buf) , len_(len) , srcAddr_(srcAddr) { } <pre><code>auto operator co_await() { struct Awaiter { RecvFromAwaitable&amp; awaitable; IoResult result = {}; bool await_ready() const noexcept { return false; } void await_suspend(std::coroutine_handle&lt;&gt; handle) noexcept { awaitable.io_.recvfrom(awaitable.socket_, awaitable.buf_, awaitable.len_, 0, awaitable.srcAddr_, [this, handle](std::error_code ec, int receivedBytes) { result = IoResult { ec, receivedBytes }; handle.resume(); }); } IoResult await_resume() const noexcept { return result; } }; return Awaiter { *this }; } </code></pre> private: IoQueue&amp; io_; int socket_; void buf_; size_t len_; ::sockaddr_in srcAddr_; }; ``` <code>operator co_await</code> must return an awaiter object, which should provide three methods: <code>await_ready</code>, <code>await_suspend</code> and <code>await_resume</code>. When an awaitable is <code>co_await</code>ed, the <code>operator co_await</code> is executed and the awaiter is obtained from it. Then <code>awaiter.await_ready()</code> is called. If it returns <code>true</code>, the coroutine does not suspend and <code>await_resume</code> is called immediately, producing the return value of the expression. This is how <code>std::suspend_never</code> works - It simply returns <code>true</code> from <code>await_ready()</code>! If <code>await_ready</code> is <code>false</code>, the coroutine is suspended, meaning all state necessary to resume it later is stored in the coroutine state and <code>await_suspend</code> is called, being passed a <a href="https://en.cppreference.com/w/cpp/coroutine/coroutine_handle"><code>std::coroutine_handle</code></a> to the coroutine being suspended. Among other things and most importantly this handle can be used to resume the coroutine later. When a suspended coroutine is resumed, <code>await_resume</code> is called, returning the result to be produced by the <code>co_await</code> expression. I keep a reference to the awaitable in the awaiter which should be fine, because both objects live until the end of the <code>co_await</code> expression and the awaiter will be destroyed before the awaitable. By the way you could also create an awaiter directly and <code>co_await</code> that (without the awaitable in between), but that is a less conventional approach and I would not have an opportunity to demonstrate <code>operator co_await</code> to you. And of course it would not work with this awaiter, because it needs a reference to an awaitable. We can use our new awaitable like this: <div class="codehighlight" style="background: #282828"><pre style="line-height: 125%;"><code>BasicCoroutine serve(IoQueue&amp;amp; io, const Fd&amp;amp; socket) { while (true) { std::string receiveBuffer(1024, &amp;lsquo;\0&amp;rsquo;); ::sockaddr_in clientAddr; const auto [recvEc, receivedBytes] = co_await RecvFromAwaitable( io, socket, receiveBuffer.data(), receiveBuffer.size(), 0, &amp;amp;clientAddr); if (recvEc) { spdlog::error(&amp;ldquo;Error in recvfrom: {}&amp;rdquo;, recvEc.message()); continue; } receiveBuffer.resize(receivedBytes); spdlog::info(&amp;ldquo;received: &amp;lsquo;{}&amp;rsquo;&amp;ldquo;, receiveBuffer); } }&lt;/p&gt; &lt;p&gt;int main() { auto socket = createSocket(SocketType::Udp, IpAddress::parse(&amp;ldquo;0.0.0.0&amp;rdquo;).value(), 4242); if (socket == -1) { return 1; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt;IoQueue io; serve(io, socket); io.run(); return 0; &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;} </code></pre></div> The full echo server is here: <a href="https://github.com/pfirsich/aiopp/blob/main/examples/echo-udp-coro.cpp">echo-udp-coro.cpp</a> Isn&rsquo;t that much prettier? There is much less noise and stupid lambdas and code that doesn&rsquo;t even tell you what the program actually does. It&rsquo;s much easier to understand too, ignoring the (high) complexity hidden in the <code>RecvFromAwaitable</code> and <code>BasicCoroutine</code> (more of that to come btw.). Let&rsquo;s recap: <ul> <li><code>main()</code> is called and calls <code>serve</code><ul> <li>The coroutine state is allocated, including the promise object</li> <li>The return object of the coroutine is created via <code>promise.get_return_object()</code></li> <li><code>co_await promise.initial_suspend()</code> is executed, which in our case is <code>co_await std::suspend_never{}</code>, i.e. it terminates immediately - the coroutine body starts executing</li> <li>A <code>RecvFromAwaitable</code> is constructed packaging the parameters of our async IO operation</li> <li>The awaiter is obtained from the <code>RecvFromAwaitable</code></li> <li><code>awaiter.await_ready()</code> returns false, so the coroutine is suspended</li> <li><code>await_suspend()</code> is called, which initiates the async IO operation (recvfrom), storing the handle to the suspended coroutine in the lambda callback.</li> </ul> </li> <li>Since <code>serve</code> suspended, it returned control to the caller and we are back in <code>main</code></li> <li><code>io.run()</code> is called and we wait to receive a message<ul> <li><code>io.recvfrom</code> completes and the callback is called</li> <li>We store the result inside the awaiter and call <code>handle.resume()</code> to resume the coroutine (<code>serve</code>)<ul> <li><code>await_resume</code> is called inside <code>serve</code> to produce the result of the <code>co_await</code> expression.</li> <li>The return value is processed and we print the received message</li> <li>We <code>co_await RecvFromAwaitable</code> again (see above)</li> </ul> </li> </ul> </li> </ul> <h2 id="tcp-echo-server">TCP Echo Server</h2> Please pretend that I built various other classes like <code>RecvFromAwaitable</code> for other methods of <code>IoQueue</code>. In reality I wrote some ugly template class that packages arguments into a tuple and then applies them to the lambda, inclusion of which in here would distract too much from the topic of this blog post. Eventually we have a few functions that return awaitables for other <code>IoQueue</code> methods that could be copy-pasted from the <code>RecvFromAwaitable</code> above (but aren&rsquo;t). In the following I will pretend we have such functions for <code>accept</code>, <code>recv</code> and <code>send</code>. Take a look at this simple TCP echo server: <div class="codehighlight" style="background: #282828"><pre style="line-height: 125%;"><code>BasicCoroutine echo(IoQueue&amp;amp; io, Fd socket) { while (true) { std::string recvBuffer(1024, &amp;lsquo;\0&amp;rsquo;); const auto [rec, receivedBytes] = co_await recv(io, socket, recvBuffer.data(), recvBuffer.size()); if (rec) { spdlog::error(&amp;ldquo;Error in receive: {}&amp;rdquo;, rec.message()); break; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt; if (receivedBytes == 0) { // Connection closed break; } recvBuffer.resize(receivedBytes); const auto [sec, sentBytes] = co_await send(io, socket, recvBuffer.data(), recvBuffer.size()); if (sec) { spdlog::error(&quot;Error in send: {}&quot;, sec.message()); break; } if (sentBytes == 0) { // Connection closed break; } } &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;}&lt;/p&gt; &lt;p&gt;BasicCoroutine serve(IoQueue&amp;amp; io, Fd&amp;amp;&amp;amp; listenSocket) { while (true) { const auto [ec, fd] = co_await accept(io, listenSocket, nullptr, nullptr); if (ec) { spdlog::error(&amp;ldquo;Error in accept: {}&amp;rdquo;, ec.message()); continue; } echo(io, Fd { fd }); } }&lt;/p&gt; &lt;p&gt;int main() { auto socket = createTcpListenSocket(IpAddress::parse(&amp;ldquo;0.0.0.0&amp;rdquo;).value(), 4242); if (socket == -1) { return 1; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt;IoQueue io; serve(io, std::move(socket)); io.run(); return 0; &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;} </code></pre></div> The final echo server is here: <a href="https://github.com/pfirsich/aiopp/blob/main/examples/echo-tcp-coro.cpp">echo-tcp-coro.cpp</a> With what we did so far, this should already work! You can interact with it by running <code>nc 127.0.0.1 4242</code>. Also we did something cool here, which might not be obvious right away. We <code>co_await accept</code> in a loop and call <code>echo</code> afterwards, which will handle a single connection. This makes it possible to handle multiple clients at once, without extra effort. When <code>accept</code> completes, i.e. we got a new connection and <code>serve</code> is resumed, we call echo, which will immediately <code>co_await recv</code> and return control back to us, so we can accept again, so that both <code>accept</code> and <code>recv</code> are issued at the same time. Pretty sweet! Here it becomes important that we chose to use <code>std::suspend_never</code> for <code>final_suspend</code>! You will often (later) see task types (like <code>BasicCoroutine</code>) that keep a handle to a coroutine and clean it up in the destructor, like <code>if (handle_) handle_.destroy()</code>. In this case you would have to change <code>final_suspend</code> to return <code>std::suspend_always</code> (or something else that suspends). With <code>std::suspend_never</code>, as we learned earlier (see above to remind yourself), the coroutine will not suspend after executing the coroutine body and the coroutine state will be destroyed immediately. This will invalidate all coroutine handles. So if we called <code>handle_.destroy()</code> in <code>~BasicCoroutine</code>, the coroutine would be long dead and our coroutine handle would be dangling, which would result in a use-after-free bug (or double-free if you will). Of course it is not just illegal (straight to jail) to destroy a coroutine through a dangling handle, but to do anything with it, including checking if the coroutine is done! Analogously if you use <code>std::suspend_always</code> for <code>final_suspend</code>, the coroutine state will not be destroyed automatically and you have to destroy the coroutine through its handle if you don&rsquo;t want to leak it. If we went for this approach, our <code>BasicCoroutine</code> destructor would have destroyed the coroutine and since <code>echo(io, Fd { fd })</code> returns a temporary <code>BasicCoroutine</code>, the <code>BasicCoroutine</code> would destroy the coroutine as soon as we suspend <code>echo</code> from the first <code>co_await recv</code> and return control to <code>serve</code>. In summary we want our coroutine to outlive the <code>BasicCoroutine</code> objects, because we don&rsquo;t use them and they get destroyed early. There is still a tiny problem with this. Since TCP is a stream protocol, send might not send all the data we have given it, i.e. <code>sentBytes</code> might be less than <code>recvBuffer.size()</code> and we should actually send in a loop until the whole buffer has been sent or an error occured (or the connection was closed). Ideally we would just introduce a new function <code>sendAll</code> which will do what I just explained, like so: ```cpp BasicCoroutine sendAll(IoQueue&amp; io, const Fd&amp; socket, const std::string&amp; buffer) { size_t offset = 0; while (offset &lt; buffer.size()) { const auto [ec, sentBytes] = co_await send(io, socket, buffer.data() + offset, buffer.size() - offset); if (ec) { spdlog::error(&ldquo;Error in send: {}&rdquo;, ec.message()); co_return; } <pre><code> if (sentBytes == 0) { co_return; } offset += sentBytes; } </code></pre> } ``` Unfortunately we cannot simply call this function from <code>echo</code>. If we did, the <code>send</code>s from <code>sendAll</code> and <code>recv</code>s from <code>echo</code> would get interleaved just like the <code>accept</code>s from <code>serve</code> and the <code>recv</code>s from <code>echo</code>. And since <code>sendAll</code> is holding a reference to the receive buffer, this interleaving might cause the receive buffer to get changed by the <code>echo</code> function under our feet! Even if we do a simple fix of copying the entire buffer and passing it into <code>sendAll</code> by value (dirty and lazy), another <code>recv</code> could in theory complete before the <code>sendAll</code> and initiate another <code>sendAll</code>, potentially destroying the stream property of the TCP connection by interleaving data that should not be interleaved. To be correct, we should pass control to <code>sendAll</code> until it finishes sending data and only then regain control in <code>echo</code> - or in other words: we want to <code>co_await sendAll()</code>! <h2 id="awaitable-coroutines">Awaitable Coroutines</h2> Let&rsquo;s just try to <code>co_await</code> a <code>BasicCoroutine</code>: ```cpp BasicCoroutine echo(IoQueue&amp; io, Fd socket) { while (true) { std::string recvBuffer(1024, &lsquo;\0&rsquo;); spdlog::info(&ldquo;recv&rdquo;); const auto [rec, receivedBytes] = co_await recv(io, socket, recvBuffer.data(), recvBuffer.size()); if (rec) { spdlog::error(&ldquo;Error in receive: {}&rdquo;, rec.message()); break; } <pre><code> if (receivedBytes == 0) { // Connection closed break; } recvBuffer.resize(receivedBytes); co_await sendAll(io, socket, recvBuffer); } </code></pre> } ``` Unfortunately we get the following error: <code>error: no member named 'await_ready' in 'BasicCoroutine'</code>. This is because, like I said earlier, a coroutine doesn&rsquo;t have to be an awaitable and ours isn&rsquo;t! Fixing this means, that we have to provide an <code>operator co_await</code> for our coroutine object. Instead of extending <code>BasicCoroutine</code>, we introduce a new <code>Task</code> to be returned by <code>sendAll</code> (more on why later). The name of this class is borrowed from many other libraries and aligns with a proposal for a type to be added to the C++ standard. Even programming languages like C# and Python use this name to represent an asynchronous operation that can be awaited, so we will use it too. Let&rsquo;s start with a copy of <code>BasicCoroutine</code>, but add a <code>operator co_await</code>, so we can <code>co_await</code> it: <div class="codehighlight" style="background: #282828"><pre style="line-height: 125%;"><code>class Task { public: struct Promise { Task get_return_object() { return Task { }; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt; void unhandled_exception() noexcept { } void return_void() noexcept { } std::suspend_never initial_suspend() noexcept { return {}; } std::suspend_never final_suspend() noexcept { return {}; } }; using promise_type = Promise; struct Awaiter { bool await_ready() const noexcept { return false; } void await_suspend(std::coroutine_handle&amp;lt;&amp;gt;) const noexcept { } void await_resume() const noexcept { } }; auto operator co_await() noexcept { return Awaiter { }; } &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;}; ```&lt;/p&gt; &lt;p&gt;What do we want our &lt;code&gt;Task&lt;/code&gt; to do really? We want to suspend the calling coroutine (&lt;code&gt;echo&lt;/code&gt; in this case), when we start &lt;code&gt;sendAll&lt;/code&gt; and only resume the calling coroutine, when &lt;code&gt;sendAll&lt;/code&gt; has finished. The best way to hook into &amp;ldquo;when a coroutine has finished&amp;rdquo; is the &lt;em&gt;awaiter&lt;/em&gt; returned by &lt;code&gt;final_suspend&lt;/code&gt;, because as we learned earlier that awaiter will be suspended, when the coroutine body has completed. That means we probably want to introduce a custom awaiter, which we will call &lt;code&gt;FinalAwaiter&lt;/code&gt;:&lt;/p&gt; &lt;p&gt;```cpp class Task { public: struct FinalAwaiter { bool await_ready() const noexcept { return false; } void await_suspend(std::coroutine_handle&amp;lt;&amp;gt; handle) noexcept { } void await_resume() const noexcept { } };&lt;/p&gt; &lt;pre&gt;&lt;code&gt;struct Promise { Task get_return_object() { return Task { }; } void unhandled_exception() noexcept { } void return_void() noexcept { } std::suspend_never initial_suspend() noexcept { return {}; } FinalAwaiter final_suspend() noexcept { return {}; } }; using promise_type = Promise; struct Awaiter { bool await_ready() const noexcept { return false; } void await_suspend(std::coroutine_handle&amp;lt;&amp;gt;) const noexcept { } void await_resume() const noexcept { } }; auto operator co_await() noexcept { return Awaiter { }; } &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;}; ```&lt;/p&gt; &lt;p&gt;Now we have all the pieces and we just need to draw the rest of the fucking owl.&lt;/p&gt; &lt;p&gt;If we want to resume the calling coroutine later, we need to save a handle to it and we need to save it somewhere we can get to from &lt;code&gt;FinalAwaiter&lt;/code&gt;, which will be the &lt;em&gt;promise&lt;/em&gt;. To get to the &lt;em&gt;promise&lt;/em&gt; from the &lt;code&gt;Awaiter&lt;/code&gt;, we need a handle to the coroutine that belongs to the &lt;code&gt;Task&lt;/code&gt;. Let&amp;rsquo;s start by passing that from &lt;code&gt;Promise::get_return_object&lt;/code&gt; to &lt;code&gt;Task()&lt;/code&gt; and keeping a copy in &lt;code&gt;Awaiter&lt;/code&gt; as well:&lt;/p&gt; &lt;p&gt;```cpp class Task { public: struct FinalAwaiter { bool await_ready() const noexcept { return false; } void await_suspend(std::coroutine_handle&amp;lt;&amp;gt; handle) noexcept { } void await_resume() const noexcept { } };&lt;/p&gt; &lt;pre&gt;&lt;code&gt;struct Promise { Task get_return_object() { return Task { std::coroutine_handle&amp;lt;Promise&amp;gt;::from_promise(*this) }; } void unhandled_exception() noexcept { } void return_void() noexcept { } std::suspend_never initial_suspend() noexcept { return {}; } FinalAwaiter final_suspend() noexcept { return {}; } }; using promise_type = Promise; Task() = default; struct Awaiter { std::coroutine_handle&amp;lt;Promise&amp;gt; handle; bool await_ready() const noexcept { return false; } void await_suspend(std::coroutine_handle&amp;lt;&amp;gt;) const noexcept { } void await_resume() const noexcept { } }; auto operator co_await() noexcept { return Awaiter { handle_ }; } &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;private: explicit Task(std::coroutine_handle&lt;Promise&gt; handle) : handle_(handle) { }&lt;/p&gt; &lt;pre&gt;&lt;code&gt;std::coroutine_handle&amp;lt;Promise&amp;gt; handle_; &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;}; ```&lt;/p&gt; &lt;p&gt;Note that we specified the promise type explicitly for the newly introduced coroutine handles. This is necessary to make the &lt;code&gt;.promise()&lt;/code&gt; method available.&lt;/p&gt; &lt;p&gt;Remember that a handle to the calling coroutine (the one that &lt;code&gt;co_awaits&lt;/code&gt; the &lt;code&gt;Task&lt;/code&gt;) is passed to &lt;code&gt;Awaiter::await_suspend&lt;/code&gt;. Let&amp;rsquo;s add a member variable to &lt;code&gt;Promise&lt;/code&gt; and save a handle to the calling corouting in there. We will call this member variable &lt;code&gt;continuation&lt;/code&gt;:&lt;/p&gt; &lt;p&gt;```cpp struct Promise { std::coroutine_handle&amp;lt;&amp;gt; continuation;&lt;/p&gt; &lt;pre&gt;&lt;code&gt; Task get_return_object() { return Task { std::coroutine_handle&amp;lt;Promise&amp;gt;::from_promise(*this) }; } void unhandled_exception() noexcept { } void return_void() noexcept { } std::suspend_never initial_suspend() noexcept { return {}; } FinalAwaiter final_suspend() noexcept { return {}; } }; struct Awaiter { std::coroutine_handle&amp;lt;Promise&amp;gt; handle; bool await_ready() const noexcept { return false; } void await_suspend(std::coroutine_handle&amp;lt;&amp;gt; calling) noexcept { handle.promise().continuation = calling; } void await_resume() const noexcept { } }; &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;}; ```&lt;/p&gt; &lt;p&gt;Now all that&amp;rsquo;s left to make it work, is to get the &lt;code&gt;continuation&lt;/code&gt; handle in &lt;code&gt;FinalAwaiter::await_suspend&lt;/code&gt;, which is called when the called coroutine (&lt;code&gt;sendAll&lt;/code&gt;) completes, and &lt;code&gt;.resume()&lt;/code&gt; it:&lt;/p&gt; &lt;p&gt;```cpp struct FinalAwaiter { bool await_ready() const noexcept { return false; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt; template &amp;lt;typename P&amp;gt; void await_suspend(std::coroutine_handle&amp;lt;P&amp;gt; handle) noexcept { handle.promise().continuation.resume(); } void await_resume() const noexcept { } }; &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;```&lt;/p&gt; &lt;p&gt;We are not quite done, but this should work already. Let&amp;rsquo;s iron out the kinks before we proceed.&lt;/p&gt; &lt;p&gt;First of all, since we now have a &lt;code&gt;FinalAwaiter&lt;/code&gt; which suspends, the coroutine state is not destroyed automatically anymore. To prevent a memory leak, we need to add a destructor to &lt;code&gt;Task&lt;/code&gt;:&lt;/p&gt; &lt;p&gt;&lt;code&gt;cpp ~Task() { if (handle_) { handle_.destroy(); } }&lt;/code&gt;&lt;/p&gt; &lt;p&gt;This might also be a good time to mention that a coroutine is &amp;ldquo;done&amp;rdquo;, when it &amp;ldquo;is suspended at its final suspended point&amp;rdquo;, so when the &lt;em&gt;awaiter&lt;/em&gt; returned by &lt;code&gt;final_suspend&lt;/code&gt; has been &lt;code&gt;co_await&lt;/code&gt;ed. The difference, again, between &lt;code&gt;await_ready&lt;/code&gt; returning &lt;code&gt;true&lt;/code&gt; or &lt;code&gt;false&lt;/code&gt; is whether the coroutine state is destroyed automatically, which is why we need to destroy it manually this time.&lt;/p&gt; &lt;p&gt;Also I find it a bit awkward, that when we start &lt;code&gt;sendAll&lt;/code&gt;, it starts doing its thing and suspends at &lt;code&gt;co_await send&lt;/code&gt;, control is passed back to us (&lt;code&gt;echo&lt;/code&gt;) and then we &lt;code&gt;co_await&lt;/code&gt; the &lt;code&gt;Task&lt;/code&gt; returned by it (&lt;code&gt;sendAll&lt;/code&gt;). If &lt;code&gt;co_await send&lt;/code&gt; would not have suspended &lt;code&gt;sendAll&lt;/code&gt;, it might have completed immediately, &lt;code&gt;co_await&lt;/code&gt;ed our &lt;code&gt;FinalAwaiter&lt;/code&gt; and we would attempt to resume the calling coroutine, before it has even been stored in the promise, because it was never suspended and &lt;code&gt;await_suspend&lt;/code&gt; has not been called yet! This is why this time, we want to return &lt;code&gt;std::suspend_always&lt;/code&gt; from &lt;code&gt;initial_suspend&lt;/code&gt;. This also means that when we first &lt;code&gt;co_await&lt;/code&gt; a &lt;code&gt;Task&lt;/code&gt;, we should resume it once, so it starts executing:&lt;/p&gt; &lt;p&gt;&lt;code&gt;cpp // In Awaiter (returned from operator co_await) void await_suspend(std::coroutine_handle&amp;lt;&amp;gt; continuation) noexcept { handle.promise().continuation = continuation; handle.resume(); }&lt;/code&gt;&lt;/p&gt; &lt;p&gt;This also invites us to do something called &amp;ldquo;symmetric transfer&amp;rdquo; in which we can suspend one coroutine and resume another without consuming additional stack space. Read this post by Lewis Baker to learn more: &lt;a href=&quot;https://lewissbaker.github.io/2020/05/11/understanding_symmetric_transfer&quot;&gt;Understanding Symmetric Transfer&lt;/a&gt;. To use it, we simply have to return a coroutine handle from &lt;code&gt;await_suspend&lt;/code&gt; instead of calling &lt;code&gt;resume()&lt;/code&gt; on it:&lt;/p&gt; &lt;p&gt;&lt;code&gt;cpp // In Awaiter (returned from operator co_await) auto await_suspend(std::coroutine_handle&amp;lt;&amp;gt; calling) noexcept { handle.promise().continuation = calling; return handle; }&lt;/code&gt;&lt;/p&gt; &lt;p&gt;We can also use this in &lt;code&gt;FinalAwaiter::await_suspend&lt;/code&gt;:&lt;/p&gt; &lt;p&gt;&lt;code&gt;cpp template &amp;lt;typename P&amp;gt; auto await_suspend(std::coroutine_handle&amp;lt;P&amp;gt; handle) noexcept { return handle.promise().continuation; }&lt;/code&gt;&lt;/p&gt; &lt;p&gt;The &lt;a href=&quot;https://github.com/andreasbuhr/cppcoro&quot;&gt;cppcoro&lt;/a&gt; library has a &lt;a href=&quot;https://github.com/andreasbuhr/cppcoro/blob/10bbcdbf2be3ad3aa56febcf4c7662d771460a99/include/cppcoro/config.hpp#L33&quot;&gt;check&lt;/a&gt; for whether a compiler supports symmetric transfer, but judging from my trials on &lt;a href=&quot;https://godbolt.org&quot;&gt;godbolt&lt;/a&gt; it seems that symmetric transfer works for all compilers that don&amp;rsquo;t complain about their respective C++20 flags (GCC 11 and later, MSVC 19.29 and later), so I will just use it.&lt;/p&gt; &lt;p&gt;Now that &lt;code&gt;initial_suspend&lt;/code&gt; returns &lt;code&gt;std::suspend_always&lt;/code&gt; our coroutine does not start executing until we &lt;code&gt;co_await&lt;/code&gt; it, meaning that it would make sense to make our &lt;code&gt;Task&lt;/code&gt; type &lt;code&gt;[[nodiscard]]&lt;/code&gt;, otherwise creating the coroutine would simply do nothing and that is likely a mistake we want to help catch: &lt;code&gt;cpp class [[nodiscard]] Task { // ... };&lt;/code&gt;&lt;/p&gt; &lt;p&gt;The last tweak we want to make to &lt;code&gt;Task&lt;/code&gt; is to &lt;code&gt;Awaiter::await_ready&lt;/code&gt;. It returns &lt;code&gt;false&lt;/code&gt;, which works for our code, but in case the coroutine is already done when we &lt;code&gt;co_await&lt;/code&gt; it, we don&amp;rsquo;t want to suspend, but rather return the value right away:&lt;/p&gt; &lt;p&gt;&lt;code&gt;cpp bool await_ready() const noexcept { return !handle || handle.done(); }&lt;/code&gt;&lt;/p&gt; &lt;p&gt;Putting it all together, we get this:&lt;/p&gt; &lt;p&gt;```cpp class [[nodiscard]] Task { public: struct FinalAwaiter { bool await_ready() const noexcept { return false; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt; template &amp;lt;typename P&amp;gt; auto await_suspend(std::coroutine_handle&amp;lt;P&amp;gt; handle) noexcept { return handle.promise().continuation; } void await_resume() const noexcept { } }; struct Promise { std::coroutine_handle&amp;lt;&amp;gt; continuation; Task get_return_object() { return Task { std::coroutine_handle&amp;lt;Promise&amp;gt;::from_promise(*this) }; } void unhandled_exception() noexcept { } void return_void() noexcept { } std::suspend_always initial_suspend() noexcept { return {}; } FinalAwaiter final_suspend() noexcept { return {}; } }; using promise_type = Promise; Task() = default; ~Task() { if (handle_) { handle_.destroy(); } } struct Awaiter { std::coroutine_handle&amp;lt;Promise&amp;gt; handle; bool await_ready() const noexcept { return !handle || handle.done(); } auto await_suspend(std::coroutine_handle&amp;lt;&amp;gt; calling) noexcept { handle.promise().continuation = calling; return handle; } void await_resume() const noexcept { } }; auto operator co_await() noexcept { return Awaiter { handle_ }; } &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;private: explicit Task(std::coroutine_handle&lt;Promise&gt; handle) : handle_(handle) { }&lt;/p&gt; &lt;pre&gt;&lt;code&gt;std::coroutine_handle&amp;lt;Promise&amp;gt; handle_; &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;}; &lt;code&gt;`` *The final&lt;/code&gt;Task` class is here: &lt;a href=&quot;https://github.com/pfirsich/aiopp/blob/main/include/aiopp/task.hpp&quot;&gt;task.hpp&lt;/a&gt;*&lt;/p&gt; &lt;p&gt;And here&amp;rsquo;s the application that uses it:&lt;/p&gt; &lt;p&gt;```cpp Task sendAll(IoQueue&amp;amp; io, const Fd&amp;amp; socket, const std::string&amp;amp; buffer) { size_t offset = 0; while (offset &amp;lt; buffer.size()) { const auto [ec, sentBytes] = co_await send(io, socket, buffer.data() + offset, buffer.size() - offset); if (ec) { spdlog::error(&amp;ldquo;Error in send: {}&amp;rdquo;, ec.message()); co_return; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt; if (sentBytes == 0) { // Connection closed co_return; } offset += sentBytes; } &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;}&lt;/p&gt; &lt;p&gt;BasicCoroutine echo(IoQueue&amp;amp; io, Fd socket) { while (true) { std::string recvBuffer(1024, &amp;lsquo;\0&amp;rsquo;); const auto [ec, receivedBytes] = co_await recv(io, socket, recvBuffer.data(), recvBuffer.size()); if (ec) { spdlog::error(&amp;ldquo;Error in receive: {}&amp;rdquo;, ec.message()); break; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt; if (receivedBytes == 0) { // Connection closed break; } recvBuffer.resize(receivedBytes); co_await sendAll(io, socket, recvBuffer); } &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;}&lt;/p&gt; &lt;p&gt;BasicCoroutine serve(IoQueue&amp;amp; io, Fd&amp;amp;&amp;amp; listenSocket) { while (true) { const auto [ec, fd] = co_await accept(io, listenSocket, nullptr, nullptr); if (ec) { spdlog::error(&amp;ldquo;Error in accept: {}&amp;rdquo;, ec.message()); continue; } echo(io, Fd { fd }); } }&lt;/p&gt; &lt;p&gt;int main() { auto socket = createTcpListenSocket(IpAddress::parse(&amp;ldquo;0.0.0.0&amp;rdquo;).value(), 4242); if (socket == -1) { return 1; }&lt;/p&gt; &lt;pre&gt;&lt;code&gt;IoQueue io; serve(io, std::move(socket)); io.run(); return 0; &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;} </code></pre></div> The final echo server is here: <a href="https://github.com/pfirsich/aiopp/blob/main/examples/echo-tcp-coro.cpp">echo-tcp-coro.cpp</a> <h2 id="returning-values-from-coroutines">Returning Values From Coroutines</h2> The final thing that bothers me about this echo server is that when <code>sentBytes</code> is <code>0</code> (the connection has been closed between receiving and sending), we attempt another <code>recv</code>, which will immediately fail. Ideally we would return <code>std::pair&lt;std::error_code, bool&gt;</code> from <code>sendAll</code> which contains a potential error and an EOF (&ldquo;end-of-file&rdquo;) flag, so we can return from <code>echo</code> early, if needed: <code>cpp const auto [sendEc, eof] = co_await sendAll(io, socket, recvBuffer); if (sendEc || eof) { break; }</code> and: ```cpp Task<std::pair\<std::error_code, bool>> sendAll( IoQueue&amp; io, const Fd&amp; socket, const std::string&amp; buffer) { size_t offset = 0; while (offset &lt; buffer.size()) { const auto [ec, sentBytes] = co_await send(io, socket, buffer.data() + offset, buffer.size() - offset); if (ec) { spdlog::error(&ldquo;Error in send: {}&rdquo;, ec.message()); co_return std::make_pair(ec, false); } <pre><code> if (sentBytes == 0) { // Connection closed co_return std::make_pair(ec, true); } offset += sentBytes; } co_return std::make_pair(std::error_code {}, false); </code></pre> } ``` I mentioned earlier that we can define <code>Promise::return_value</code> instead of <code>Promise::return_void</code> to do that. And like the continuation earlier, we will store the result in the <code>Promise</code> as well. Additionally we have to make <code>Awaiter::await_resume</code> return our result from the promise, to make the <code>co_await</code> expression return a result. To support arbitrary return types, I have made <code>Task</code> a template class and to make sure it still works with void, I specialized the <code>Promise</code> type for <code>Result=void</code>. I used <code>requires</code> expressions to define different <code>Awaiter::await_resume</code> functions for different Result template arguments. This is the whole thing: ```cpp template <typename Result = void> class [[nodiscard]] Task { public: struct FinalAwaiter { bool await_ready() const noexcept { return false; } <pre><code> template &lt;typename P&gt; auto await_suspend(std::coroutine_handle&lt;P&gt; handle) noexcept { return handle.promise().continuation; } void await_resume() const noexcept { } }; struct Promise { std::coroutine_handle&lt;&gt; continuation; Result result; Task get_return_object() { return Task { std::coroutine_handle&lt;Promise&gt;::from_promise(*this) }; } void unhandled_exception() noexcept { } void return_value(Result&amp;&amp; res) noexcept { result = std::move(res); } std::suspend_always initial_suspend() noexcept { return {}; } FinalAwaiter final_suspend() noexcept { return {}; } }; using promise_type = Promise; Task() = default; ~Task() { if (handle_) { handle_.destroy(); } } struct Awaiter { std::coroutine_handle&lt;Promise&gt; handle; bool await_ready() const noexcept { return !handle || handle.done(); } auto await_suspend(std::coroutine_handle&lt;&gt; calling) noexcept { handle.promise().continuation = calling; return handle; } template &lt;typename T = Result&gt; requires(std::is_same_v&lt;T, void&gt;) void await_resume() noexcept { } template &lt;typename T = Result&gt; requires(!std::is_same_v&lt;T, void&gt;) T await_resume() noexcept { return std::move(handle.promise().result); } }; auto operator co_await() noexcept { return Awaiter { handle_ }; } </code></pre> private: explicit Task(std::coroutine_handle<Promise> handle) : handle_(handle) { } <pre><code>std::coroutine_handle&lt;Promise&gt; handle_; </code></pre> }; template &lt;&gt; struct Task<void>::Promise { std::coroutine_handle&lt;&gt; continuation; <pre><code>Task get_return_object() { return Task { std::coroutine_handle&lt;Promise&gt;::from_promise(*this) }; } void unhandled_exception() noexcept { } void return_void() noexcept { } std::suspend_always initial_suspend() noexcept { return {}; } FinalAwaiter final_suspend() noexcept { return {}; } </code></pre> }; <code>`` *The</code>Task` class is here: <a href="https://github.com/pfirsich/aiopp/blob/main/include/aiopp/task.hpp">task.hpp</a>* You may ask if we should have or could have extended our <code>BasicCoroutine</code> to handle return values as well, but we would have to change so much about it, that the name would not be justified anymore and we would almost have <code>Task</code> anyways but without <code>operator co_await</code>. We would have to make <code>final_suspend</code> return <code>std::suspend_always</code> as well, so the coroutine frame lives long enough for us to get the result from the <code>Promise</code> through the <code>BasicCoroutine</code> object. And then we couldn&rsquo;t use it anymore to &ldquo;branch off&rdquo; with <code>echo</code> because it returns a temporary and now that the <code>BasicCoroutine</code> object has ownership of the coroutine, it would be destroyed in <code>~BasicCoroutine</code>. So we still have reasons to keep a type without a non-void return value, like <code>BasicCoroutine</code>, around. Also now that we have a <code>Task</code> class that owns the coroutine, it is possible that we destroy it (by destroying the <code>Task</code>) when it still has queued IO operations that essentially keep a reference to our awaiter (in the lambda capture). To avoid a use-after-free, it is necessary to cancel IO operations if the awaiter being referenced in the <code>IoQueue</code> dies. I did not include that in this tutorial yet, because I haven&rsquo;t figured out how to do that nicely myself. Maybe I&rsquo;ll edit it into this blog post in the future or maybe you figure it out and let me know if you find something that works well for you. <h2 id="where-to-go-next">Where To Go Next</h2> Lastly I have to mention that if you want to do anything multi-threaded or if you want to handle exceptions, there is extra work you have to do and you have to be a lot more careful everywhere. I&rsquo;m not going to do that here. Although I might have (sort of) figured out how to use coroutines, I don&rsquo;t really have experience with writing larger programs (beyond simple echo servers). It&rsquo;s possible that some of the solutions I have presented here might just be very annoying or problematic if used in larger code bases. I hope I get a chance and the inspiration to write about this in a later blog post. <h2 id="tips">Tips</h2> My number one, huge, seriously golden tip: use <a href="https://github.com/google/sanitizers/wiki/AddressSanitizer">AddressSanitizer</a>. It&rsquo;s very easy to build coroutine programs that use-after-free or leak memory without you realizing. I wholeheartedly suggest to turn on ASan and not turn it off again, as long as you are messing around with coroutines. At some point every code you write compiles, lots of it works and the only one telling you it&rsquo;s completely wrong is ASan. To help me understand which functions are called when and when objects are destroyed, I added logging to all objects on creation/destruction/copy with something like this: <a href="https://gist.github.com/pfirsich/e5e2c43ef8816f40bb8029ac8f7fb54a">DebugLogging</a>. I also added logs to every magic method - all the <code>await_*</code>s and all the <code>Promise</code> methods. It might seem silly dumping so many logs, but it really helps a lot to get an understanding of the control flow and I definitely needed it to get any understanding of coroutines in C++. <h2 id="sources">Sources</h2> <ul> <li><a href="https://lewissbaker.github.io/">Asymmetric Transfer</a> - Lewis Baker&rsquo;s blog. It&rsquo;s very detailed and it felt a bit overwhelming to me at the beginning, but it leaves few open questions and is a great read if you already sort of know how coroutines work.</li> <li><a href="https://github.com/andreasbuhr/cppcoro">cppcoro</a>. The de-facto helper library for coroutines in C++. The link points to a fork, which seems to be more maintained than the original repository by Lewis Baker, who seems to really know their coroutines.</li> <li><a href="https://en.cppreference.com/w/cpp/language/coroutines">Coroutines on cppreference</a>. Everyone knows this website is good, but this particular site is more helpful than I would have thought.</li> <li><a href="https://pabloariasal.github.io/2022/11/12/couring-1/">C++20 Coroutines and io_uring</a>. I tried working with coroutines a few times and just got confused without ever really getting anywhere. This helped me get started more than any other resource and I modeled my post after it. I just tried to go a bit further in some directions and explain some things that did not get clear to me reading that blog post.</li> </ul> Why Is There No Invisible Database? https://joelschumacher.de/posts/why-is-there-no-invisible-database 2023-04-19T21:09:09+0000 In this post I&rsquo;ll write about the exploration of an old idea of mine, which didn&rsquo;t lead very far, so it might not be the most informative piece. If you have time to spare, go ahead. This will also serve as a reminder to myself when I revisit the idea in a few years. <h2 id="the-idea">The Idea</h2> For a few years I have been thinking about a database that requires little to no explicit storage and loading. I imagined simply marking a <code>std::vector</code> for example as &ldquo;in the database&rdquo; and it would all just magically happen. I wouldn&rsquo;t have to create wrapper classes and or write tedious serialization/deserialization code. Most developer dislike writing plumbing code, right? I wanted a system that automatically retrieves the newest version whenever I access a data structure and sends a modified version back to the server, when I modify it. And ideally it would work with arbitrary data structures, without any extra work. Something like this: ```cpp struct TestStruct { std::string str; unsigned int num; }; // You need some sort of key, so you can identify that vectors across // different machines belong together. InTheDatabase<std::vector\<TestStruct>> vec(&ldquo;vec&rdquo;); vec-&gt;push_back(TestStruct { &ldquo;Joel&rdquo;, 69 /nice/ }); // &ldquo;Joel&rdquo; is now in &ldquo;vec&rdquo; and available on every connected client machine // This might print a bunch of <code>str</code>s, that we did not put into &ldquo;vec&rdquo;, // but another client might have put there for (const auto&amp; elem : *vec) { fmt::print(&ldquo;{}: {}\n&rdquo;, elem.str, elem.num); } ``` <h2 id="the-missing-piece">The Missing Piece</h2> Intuitively I thought that <code>InTheDatabase</code> would somehow inject a custom allocator that would track all dynamic allocations the type might be doing and keep track of that memory. It seemed to me the biggest problem would be detecting all modifications done to these tracked memory ranges. And because there was no way to do something like that (there is), I never really spent much time on the idea. Then, a few days ago, I learned about the &ldquo;<code>PROT_NONE/SIGSEGV</code> trick&rdquo;. It involves using <a href="https://man7.org/linux/man-pages/man2/mprotect.2.html">mprotect</a> to change the access protections of a range of memory to <code>PROT_NONE</code> (for this use case <code>PROT_READ</code> would work too), which disallows any accesses and installing a signal handler for SIGSEGV, which would be triggered whenever the protected region would be accessed. You could then detect these memory accesses and note down the page that was accessed and restore the access protections to whatever you had before (probably <code>PROT_READ | PROT_WRITE</code>). This was my missing piece (so I thought)! Maybe I could use this to implement my invisible database?! Shortly (seconds actually) after that I learned, that there is also <a href="https://www.kernel.org/doc/html/latest/admin-guide/mm/userfaultfd.html">userfaultfd</a>, which is a Linux feature that is dedicated to detecting page-faults in user space in a clean way, which can do this as well in a &ldquo;proper and more optimal&rdquo; way. <h2 id="a-revised-sketch">A Revised Sketch</h2> This idea made me engage with this invisible database more seriously. The first thing I realized is that in order to maintain invariants of the classes being stored in the database, we have to introduce transactions. If we simply synchronize every modified memory region as soon as we can, we can easily bring data structures into invalid states. Data structures may also span multiple pages (esp. if they do many allocations) or cross page boundaries. For example if we <code>push_back</code>-ed an element into a vector, we might update the size first and then actually allocate the new element (which might be on a different page). Of course it could be the other way around as well, which would be better for us, but we should not rely on implementation details we cannot control (or sometimes even know) to maintain valid data structures. Very similar to multi-threaded programming, you have to protect the integrity (invariants) of these data structures with some sort of mutex. This invites the introduction of another abstraction of a memory region in which we can place as many data structures as we want and that we can lock and unlock to modify it transactionally: ```cpp // the last argument is region size in pages invisibledb::SyncedMemoryRegion region(server, &ldquo;region&rdquo;, 4); struct TestStruct { std::pmr::string str; unsigned int num; }; // This will lock the region on the server preventing any other client // from locking it for a while, and read the current version from the server. // It might also actually lock a mutex, so you can use the SyncedMemoryRegion // from multiple threads. region.lock(); const auto vec = region.getObject<std::pmr::vector\<TestStruct>>(&ldquo;vec&rdquo;); // Here we will send all modified pages to the server and the region will be unlocked. region.unlock(); // &hellip; region.lock() vec-&gt;push_back(TestStruct { &ldquo;Joel&rdquo;, 42 }); for (const auto&amp; elem : *vec) { fmt::print(&ldquo;{}: {}\n&rdquo;, elem.str, elem.num); } region.unlock(); ``` There are also some other small changes in this snippet. To inject the allocator, we used the standard library types from the <code>std::pmr</code> (polymorphic memory resource) namespace. Also of course we need some object that represents the database server, <code>server</code>, which we need to pass into the <code>SyncedMemoryRegion</code>. Where it comes from, how we connect and anything like that is not important for our sketch. The <code>getObject</code> factory function either creates the vector on the heap, inside the memory region or returns the instance, if an object with that name is already in the region. It was already like that in the first sketch, but it should be mentioned that the vector itself must not live on the stack, or it would not be synchronized properly (duh). Additionally because we can track memory accesses already, we can do cool things like <code>assert</code>ing when the memory is accessed without the region being locked, because sometimes that&rsquo;s the only way they&rsquo;re gonna learn. Note that it&rsquo;s very easy to be less than extremely careful here and use the wrong allocator by accident. For example like this: <code>cpp std::pmr::string str = "Joel"; vec-&gt;push_back(TestStruct { std::move(str), 42 });</code> Here the <code>str</code> field in our <code>TestStruct</code> object is not allocated in the memory region managed by the database! That&rsquo;s a very easy mistake, hard to catch and the behavior exhibited will be maddeningly weird. Bad sign. Also we need to consider that the allocator for the <code>SyncedMemoryRegion</code> will have to keep it&rsquo;s state inside the memory region as well. That means all variables that control the allocator behavior have to be inside this memory region and have to be synchronized. Otherwise we might allocate from the start of the region multiple times on different clients and overwrite things by accident. For that reason alone we have to roll our own allocator (or multiple, depending on use case). But there is also another, smaller reason, which is that we require perfectly predictable allocator behavior. This almost gives us no other choice but to roll our own. Imagine that on one client an allocator might reuse a previously deallocated region and on another client a different allocator (maybe from a different standard library implementation) might reuse another. A more practical reason is also, is that the pmr containers keep a pointer to the allocator, which of course should not point outside the synchronized memory region as well. And thinking about this it also becomes clear that not just all the allocators have to have the same behavior, but all the data structures involved to. So to be sure, you probably want to use exactly the same standard library version for each client. A small corollary, which might be even more obvious, is that we cannot mix little and big endian clients. That might be a bit too restrictive to be useful. <h2 id="the-prototype">The Prototype</h2> This is about where I started implementing this thing. If you were smarter than me, you might have figured this out before starting, but pretty much all the interesting data structures have pointers. And even though they will point into the synchronized memory region, if you do everything right, those addresses might (very likely) not be correct on other clients, because the base address of the memory region will be different on different clients. I could not figure out how to fix this properly, except by walking through the whole modified memory region and finding everything that is aligned like a pointer and contains a value that looks like a pointer (i.e. an address inside of this memory region). After finding those addresses, I make them relative, save their offsets and restore them with the proper base address on reception. Yes, this could give false positives, which should be unlikely, but would lead to the most confusing bugs. You can check out my proof-of-concept implementation here: <a href="https://github.com/pfirsich/invisibledb">pfirsich/invisibledb</a> In my implementation I took some shortcuts for the first (and likely last) prototype. The <code>Server</code> object does not actually connect anywhere and <code>Server::sendPage</code> and <code>Server::receivePage</code> simply <code>memcpy</code> some memory to and from a buffer. Similarly <code>Server::lockRegion</code> and <code>Server::unlockRegion</code> simply set a boolean flag. The <code>TrackedMemoryRegion</code> class, which was supposed to do the cool stuff - tracking writes using <code>usefaultfd</code> - currently only uses <code>memcmp</code> to check whether a page was modified. I wanted to keep the complexity low whereever possible until I implemented the tricky bits. That turned out to be wise, because, as I have explained above, they were very tricky. Also the allocator is a very simple linear allocator that can not reuse freed memory and simply increases an offset from which it gives out memory. In a real world application you definitely want something smarter and likely different allocators for different use cases and data structures. You would possibly need to support multiple different allocators in the same region. Of course there are a couple more ways to improve this, which are not really relevant at the level of seriousness of this project. <h2 id="conclusion">Conclusion</h2> Now that I have finally implemented a (not so) invisible database, I know why it didn&rsquo;t exist before or at least why it&rsquo;s not very popular. It was tricky to build and a fun exercise, but there are some downsides to using it, which are the answer to &ldquo;Why is there no invisible database?&rdquo;: <ul> <li>Visiblity: The resulting system is not nearly as invisible as we would like it to be (explicit locking, specific ways of creating objects, &hellip;).</li> <li>Usability (arguably still &ldquo;visibility&rdquo;): It&rsquo;s extremely easy to misuse and the resulting bugs are very confusing. Having a variable on the stack, forgetting to inject the right allocator or forgetting to lock the region, etc. can cause significant headaches.</li> <li>Performance: While you do save the serialization/deserializion logic there is significant overhead. Intercepting every memory access is likely inefficient. As is the scanning of memory for pointer-like values during synchronization. There are probably some access patterns where it might be faster than any other database, but I cannot come up with them and I definitely did not prove it.</li> <li>Portability: You cannot mix clients with different endianness or even standard library versions and maybe even compilers. And, of course, <code>userfaultfd</code> is currently limited to Linux only! You can only use it with C or C++ and those languages are not considered very cool or 🚀 blazingly fast 🚀 anymore. Even if you use exactly the same software on every client, you probably want to upgrade at some point. Since you are storing memory regions, an ABI break in some library you use might break your weekend.</li> <li>Complexity: Despite ignoring many complications (a real server, proper allocators, proper write tracking), the resulting piece of software is quite the monstrosity and I am sure you can measure my heart rate increase, when I look at it. Introducing complexity usually implies future costs.</li> </ul> And even if any of these are no concern to you, you don&rsquo;t really know if it will stay like that forever. I would not feel comfortable introducing something like this into my project. Even if I had made it! If we finally (ever?) get reflection in C++, another approach might be to generate serialization/deserialization logic for <a href="https://en.wikipedia.org/wiki/Passive_data_structure">POD</a>s and provide some wrappers for standard library classes to simply interface with any of the popular databases. This would essentially be just a fancy client library for an existing database. And if we ever do get reflection, I will probably try that. Expectedly my idea did not lead to a groundbreaking conclusion, but I found it very worthwile to explore this topic more deeply and the better I understand the remaining problems, the more likely I am to find solutions for them in the future. Ray Casting in 2D Grids https://joelschumacher.de/posts/ray-casting-in-2d-grids 2016-02-10T21:21:48+0000 Ray casting is used in many different areas of game development, most commonly for visibility, AI (often for visibility though) and collision detection and resolution. If you are making a game, chances are you need ray casting somewhere. I started this blog post over a year ago, but decided to not write it up fully and publish it, since I thought that it might be a little embarassing, since most if it is too trivial. But I still find it fairly hard to find any good information or example code on this and on the weekend of Global Game Jam 2016, I forgot how to do it properly and didn&rsquo;t find any good resources on this, ending up debugging ray casting code for a little over 3 hours (unnecessarily, I think). So if this actually helps no one then this shall remain as a memorial of shame for myself. Or as a handy reminder, so history does not repeat itself in a way that leaves me debugging ray casting code for hours. The relevancy of grids is given by the prevalence of tile based games and even if the game is not tile based, because of optimization structures for collision detection/physics or other gameplay elements that are often in place, being either directly (simple uniform grids) or indirectly (e.g. quadtrees) related to grids. Throughout this article, our grid will be a uniform grid with cell size <code>grid.cellSize</code>. The ray will be a tuple of a point (it&rsquo;s origin) and a vector (it&rsquo;s direction), namely <code>ray.startX, ray.startY</code> and <code>ray.dirX, ray.dirY</code>. The set of points on the ray is then described by: <code>{ray.start + t*ray.dir | t is a real number}</code> and we can &ldquo;address&rdquo; every point on the ray with a value <code>t</code>. All methods presented here on out are implemented in this GitHub repository (rather badly most of the time, since I started this project over a year ago so this is a little patchwork-ey, but I put some work into refactoring, so it should be bearable): <a href="https://github.com/pfirsich/ray-casting-test">https://github.com/pfirsich/ray-casting-test</a> <h2 id="the-naive-method">The Naive Method</h2> The simplest, but often times an adequate way to do it, is stepping uniformly in world coordinates. In our case, where our ray direction is not normalized, we can not just step uniformly in <code>t</code>, but must use our normalized direction vector multiplied by the step size (again: in world coordinates). Even if often times this is sufficient, it is never perfect in the sense that this method finds all cells that intersect with the ray and with finite dt, there will always be corners that could be overstepped. An example of this from the example löve program (function <code>castRay_naive</code>) can be seen in this screenshot: <a href="/images/naive_ray_casting.png"><img alt="Naive Ray Casting" src="/images/naive_ray_casting.png" title="Naive Ray Casting" /></a> The boxes inside the cells indicate missing or superfluous cells. Of course this method could be &ldquo;exact&rdquo; if we always choose the step size to be corresponding to the size of a single pixel on screen, meaning we would not be able to distinguish exact and inexact at this point, but this would imply a huge number of samples, which is most likely not efficient enough. <h2 id="line-rasterization-methods-eg-dda-bresenham">Line Rasterization Methods (e.g. <a href="https://en.wikipedia.org/wiki/Digital_differential_analyzer_%28graphics_algorithm%29">DDA</a>, <a href="https://en.wikipedia.org/wiki/Bresenham's_line_algorithm">Bresenham</a>)</h2> This might seem like the obvious choice and depending on your game (or the problem at hand) it might very well be, but in general this is also not sufficient, since these algorithms operate in (in our case) tile coordinates only, so that there will be no difference in a ray connecting the upper left corner of a tile and the lower right of another tile and a ray connecting the lower right corner of the same, first tile and the upper left corner of the other one. In fact most commonly line rasterization algorithms will rasterize every pixel, when a diamond shape in the pixel&rsquo;s center intersects the line connecting the centers of the start and end pixel (the &ldquo;diamond rule&rdquo;). An example of this, including a visualization of the diamonds mentioned before, can be found in this screenshot (also from the example program, which has both DDA and Bresenham available): <a href="/images/ray_casting_bresenham.png" title="Ray Casting with Line Rasterization - Bresenham"><img alt="Ray Casting with Line Rasterization - Bresenham" src="/images/ray_casting_bresenham.png" /></a> Here you can see that we lose the information of where exactly inside the tile our start and end points are and also don&rsquo;t treat the the tiles as axis aligned squares. If you just need any line connecting your start and endpoint somehow, you might use a line rasterization method since they can be implemented very efficiently - and probably already have been, since they are so important for computer graphics in general. Also if your game has everything snapping to the tile grid and everything centered, this becomes an exact solution (a modern example would be Crypt of the Necrodancer)! In my example program the functions implementing this are called <code>castRay_DDA</code> and <code>castRay_Bresenham</code>. <h2 id="exact-ray-casting">Exact Ray Casting</h2> <h3 id="simplified-case-direction-positive">Simplified Case (direction positive)</h3> In a more (algorithmically) efficient method we would love to not visit cells twice, i.e. make no unnecessary steps and never miss anything. So if you want to find all intersections of the line with the grid and don&rsquo;t miss any, at every step of the algorithm, you just have to find the first one, given your current position and direction. <a href="/images/ray_casting_tile.png" title="Ray Casting a Tile"><img alt="Ray Casting a Tile" src="/images/ray_casting_tile.png" /></a> If we consider the current position <code>curX, curY</code>, we can easily find the the tile coordinates of the tile <code>curX, curY</code> is in by dividing by the cell size and truncating the floating point part. Beware that, because we are using Lua for example code (since the example program is in Lua), arrays and tiles will be 1-indexed, i.e. start with index 1. So we will have to add 1 to both tile coordinates to get the final coordinate: <code>lua function tileCoords(cellSize, x, y) return math.floor(x / cellSize) + 1, math.floor(y / cellSize) + 1 end</code> To get the next edges (assuming our direction is positive in x and y direction), we only have to add <code>grid.cellSize</code>, or in our cases, since we already added 1, do nothing but transform back to world coordinates, by multiplying with <code>grid.cellSize</code>. The distance to these edges can then be divided by the corresponding components of our direction vector to give us these distances in units of <code>t</code>! <code>lua local tileX, tileY = tileCoords(grid.cellSize, curX, curY) local dtX = ((tileX)*grid.cellSize - curX) / ray.dirX local dtY = ((tileY)*grid.cellSize - curY) / ray.dirY</code> To find the first intersection, we then only have to increment our current <code>t</code> by the smaller <code>dt</code> of the two and update our current position. Implemented as the function <code>castRay_clearer_positive</code>, the results are exactly what we wanted! <a href="/images/ray_casting_accurate.png" title="Ray Casting - Accurate"><img alt="Ray Casting - Accurate" src="/images/ray_casting_accurate.png" /></a> Though, as mentioned before, negative directions (any axis) still give us some problems: <a href="/images/ray_casting_negative.png" title="Ray Casting Negative - Problems!"><img alt="Ray Casting Negative - Problems!" src="/images/ray_casting_negative.png" /></a> The reasons for this are obviously our flawed calculation for the next edge, since in the current case, we add 1 to the tile coordinates. In the negative cases, we would have to add zero, since the edges we want to hit are the ones of the current tile. This is intentionally formulated to hint at the other problem, which is that if we jump to the next intersection, we will still be in the current tile and revisit the first intersection indefinitely! This is because <code>math.floor</code> maps values from <code>[x, x+1)</code> to <code>x</code>, so that a whole integer maps to itself, which is very sensible for a floor function, but is not entirely what we want. <h3 id="complete-case-all-directions">Complete Case (all directions)</h3> The problem of finding the next edge is trivial to solve, by introducing an offset of the tile coordinates that depends on the direction: ```lua &ndash; NOTE: &lsquo;cond and x or y&rsquo; is a common idiom that is mostly equivalent to the &ndash; ternary operator i.e. equals x if cond evaluates to true and y otherwise local dirSignX = ray.dirX &gt; 0 and 0 or -1 local dirSignY = ray.dirY &gt; 0 and 0 or -1 &hellip; local dtX = ((tileX + dirSignX)grid.cellSize - curX) / ray.dirX local dtY = ((tileY + dirSignY)grid.cellSize - curY) / ray.dirY ``` The <code>floor</code>-problem has multiple ways to solve it. One would be to rewrite our floor function so that it takes an additional parameter, which indicates which side of the interval should be open (i.e. not include the integer). Another one is to add an epsilon to our <code>t</code>, when incrementing it by either <code>dtX</code> or <code>dtY</code>, so that we will barely sneak into the next cell. Choosing this epsilon is a little tricky though since by introducing it, our algorithm will start to have edges it could overstep, just like the naive method. Any non-vanishing epsilon will have this as a result and even if we make it really small (of the order of a screen pixel size), we might run into issues when it just disappears because of floating point normalization, when ray casting in really big grids. This method might then resemble a naive method, with a really small step, which intelligently skips some samples, if possible. The best way though would probably be to introduce our own tile coordinates, which are not calculated from the current position, but kept track of alongside it. Then we can just decide to increment/decrement these at the right times - <code>tileX</code> when <code>dtX &amp;lt; dtY</code> and <code>tileY</code> otherwise. This is implemented in the function <code>castRay_clearer_alldirs_improved</code>. <h2 id="final-solution">Final Solution</h2> The hawkeye programmer might have noticed that our <code>castRay_clearer_alldirs_improved</code> is a little more complicated than it should be. You have to watch a little more closely but some terms in the calculation of <code>dtX</code> and <code>dtY</code> might be constant and maybe we can eliminate the division. In fact we can and with a little more transforms and noticing that all our constants are symmetric in x and y, so that we might as well write a helper function that calculates these independent of axis, we arrive at our new function <code>castRay_clearer_alldirs_improved_transformed</code> (thank god this article is going to end soon, these names are getting long). The necessary transformations to arrive at this function are shown in the following images: The unmodified version of <code>castRay_clearer_alldirs_improved</code>: <a href="/images/raycast_trafo_1.png" title="Ray Casting Function Transformation #1"><img alt="Ray Casting Function Transformation #1" src="/images/raycast_trafo_1.png" /></a> The part with the central logic is replaced so the actual modification of the central variables happens outside. This is so we can identify the delta of <code>dtX</code> and <code>dtY</code> more easily: <a href="/images/raycast_trafo_2.png" title="Ray Casting Function Transformation #2"><img alt="Ray Casting Function Transformation #2" src="/images/raycast_trafo_2.png" /></a> Again we simplify to ease the identification of the delta of <code>dtX</code> and <code>dtY</code> by wirting the change in <code>curX</code> and <code>curY</code> with <code>dt</code> instead of <code>t</code>: <a href="/images/raycast_trafo_3.png" title="Ray Casting Function Transformation #3"><img alt="Ray Casting Function Transformation #3" src="/images/raycast_trafo_3.png" /></a> Here we &ldquo;pull out&rdquo; the first assignment of <code>dtX</code> and <code>dtY</code> and only apply deltas in the main loop of the algorithm, which can be identified by subtracting the <code>dtX</code> and <code>dtY</code> of two successive iterations: <a href="/images/raycast_trafo_4.png" title="Ray Casting Function Transformation #4"><img alt="Ray Casting Function Transformation #4" src="/images/raycast_trafo_4.png" /></a> At this point we don&rsquo;t need <code>curX</code> and <code>curY</code> anymore and can just put it into our mark-function, which is only used for visualization: <a href="/images/raycast_trafo_5.png" title="Ray Casting Function Transformation #5"><img alt="Ray Casting Function Transformation #5" src="/images/raycast_trafo_5.png" /></a> To simplify further, we prepare by putting all assignments after the if into the cases, so that we can optimize them separately and for their special cases): <a href="/images/raycast_trafo_6.png" title="Ray Casting Function Transformation #6"><img alt="Ray Casting Function Transformation #6" src="/images/raycast_trafo_6.png" /></a> We just plug in the known values of <code>dtileX</code> and <code>dtileY</code> and reorder a little: <a href="/images/raycast_trafo_7.png" title="Ray Casting Function Transformation #7"><img alt="Ray Casting Function Transformation #7" src="/images/raycast_trafo_7.png" /></a> Now we use that all intermediate variables defined at the top are symmetric in x and y and can be calculated by a common function. We also take out the division from the main loop, since the summand is constant through all iterations (<code>ddtX</code> and <code>ddtY</code>) and rename <code>dirSignX</code>/<code>dirSignY</code> to <code>dtileX</code> and <code>dtileY</code>: <a href="/images/raycast_trafo_8.png" title="Ray Casting Function Transformation #8"><img alt="Ray Casting Function Transformation #8" src="/images/raycast_trafo_8.png" /></a> One branch and 5 additions per step and a setup function (a little trickier though, but constant time) is probably as good as it can get, though it is definitely not as apparent what is actually happening in this function. If we decide that we have no use for t and therefore the intersection points themselves (and just want to visit some cells, or want to know if there is a possible path at all), then we can eliminate t all together and because we don&rsquo;t care about the actual values of dtX and dtY, we can decide to omit the &ldquo;<code>- dt</code>&rdquo; term, because the comparison in the next iteration will still yield the same result, so that our algorithm would be reduced to one branch and two additions. This is in fact the algorithm described in <a href="http://www.cse.yorku.ca/~amana/research/grid.pdf">&ldquo;A Fast Voxel Traversal Algorithm for Ray Tracing&rdquo;</a> by John Amanatides and Andrew Woo, which inspired the transformations done to <code>castRay_clearer_alldirs_improved</code>. All in all, I showed you what the problems of ray casting in 2D grids are and what kind of solutions for them exists or seem to exist, but turn out to be a fluke. And also made a nice connection to another algorithm, which many use without understanding it properly, by deriving it from an easier to understand one and applying some transformations. Preparation for Indie Speed Run 2015 - Screwdriver, SparklEd, Lighting https://joelschumacher.de/posts/isr2015-preparation-screwdriver-sparkled-lighting 2015-11-06T01:35:12+0000 In preparation for Indie Speed Run 2015 (which went well and for which I will post our game when the voting period starts) me and a long time programming buddy, <a href="http://shellfishgames.com/">Markus</a>, decided to prepare by building a handful of tools that are going to make some things a little easier. We find ourself discarding the same ideas time and time again, because we don&rsquo;t deem them feasible in the often severly limited amount of time or not discarding them but spending too much time on them, so we decided to prepare them beforehand. We had a lot of stuff in mind including a post-process library, realtime and precomputed 2D lighting, pre-built character controllers and smooth collision detection and response solutions for different scenarios that we might need, finally a proper particle system and a corresponding editor, an editor and a library for skeletal animations, and a level editor that supplements the awesome <a href="http://www.mapeditor.org/">Tiled</a> for games that are not tile based (but either polygon or simply sprite based). But as it becomes glaringly obvious this is quite a lot of work and of course we did not manage to finish everything we had on the list, although still a considerable part of it. Markus worked on the animation editor/library and manged to finish it in time. Sadly we didn&rsquo;t have any use for it in the jam, just as we didn&rsquo;t for everything I worked on. We wanted to implement the best idea we had and coincidentally (and unexpectedly) there was no need for the tech we developed beforehand (exluding a small exception elaborated on later). The projects I took on and managed to finish mostly before the jam were the lighting system, the particle effect editor and the level editor. <h2>The Level Editor (Screwdriver) - <a href="https://github.com/pfirsich/Screwdriver">GitHub</a></h2> This level editor uses my GUI, kraidGUI. It&rsquo;s a GUI made for being modified in a sense. It only serves as a slim core for handling/passing events and handling properties. Look and feel are implemented in a &ldquo;theme&rdquo;, which makes use of a very simple backend, which, as of yet, is only implemented for löve. Every part of it can be exchanged and extended easily. For Screwdriver I had to implement and color picker and was surprised how well kraidGUI did it&rsquo;s work by making this very easy. Both the GUI and the Editor were meant as an exercise in writing actual software. I feel like, lacking a formal education and professional experience, I tend to &ldquo;make things work&rdquo; first and foremost often leaving proper software design behind. This certainly also has a lot to do with me participating in game jams a lot. But GUIs and Editors (in this case especially) are meant to be used a lot more than once. kraidGUI has some things I&rsquo;m still not completely content with and Screwdriver is also not free from a few sketchy ways to do things, that I would consider hacks, but all in all I think that it&rsquo;s still mostly quite readable, maintainable and hack free. Both of them lack documentation, but I think it&rsquo;s very common for programmers to not be particularly fond of creating these. Anyways I will try to still spend some time on documenting kraidGUI soon, since I think it&rsquo;s a nice piece of software and other people might deem it useful too. Regarding the editor itself, I made it for usage with many different games in mind. Every type of entity in the game world has an entity description, which contains Metadata (modifiable or not), so a seemingly fairly custom fitted editor can be prepared for any game without much effort. The system corresponding to the components that make up the entities are implemented in the components itself, which kind of defeats the purpose of ECSs, since special cases still have to be handled inside the components, but the editor was supposed to be modular and capable of being extended with rather complicated edit modes without treating the main components special. Therefore I&rsquo;m borrowing terminology more than anything from entity component systems. Thankfully I didn&rsquo;t regret the way of doing this yet, since there was no special case except shared behaviour that could be resolved by properly breaking up functionality and using inheritance. The editor remains very modular and can be extended fairly easily with rather comprehensive editing capabilities. An example of such a entity description file can be seen here (and should be self-explanatory): <script src="https://gist.github.com/pfirsich/4e9e7fb9bf626e3085d4.js"></script> The editor itself can be seen used in this video: <iframe width="560" height="315" src="https://www.youtube.com/embed/x0ldw3uBklg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> <h2>The Particle Effect Edtior (SparklEd) - <a href="https://github.com/pfirsich/SparklEd">GitHub</a></h2> After collecting some notes about the particle system I wanted to implement for this editor, I remembered briefly that I might have heard of löve having a particle system already. I didn&rsquo;t check, but a properly tested system that is potentially a lot more efficient (instancing, geometry shaders) should be preferred to anything I could implement in löve itself. So I could reduce the amount of work I had to do to making an editor for löve&rsquo;s partice system. I made the editor in a day plus a few days fixing bugs while using it. I borrowed the idea of using the mousewheel for numeric value editing to reduce the amount of GUI related work (didn&rsquo;t want to bloat a project that small with kraidGUI). Using the mousewheel while hovering a property reduces/increases it, while holding shift decreases the amount of change. Loading is done by using the mousewheel above a property (serving as a radio button) and saving via shortcut. It supports multiple emitters to make many-layered effects (explosions for example need smoke, debris, fire, etc.). It supports continuous effects (e.g. fire) and bursted effects (you can press space to emit &ldquo;Emit amount&rdquo; of particles). And has a brief built-in documentation in the form of tooltips. It can be seen used in this video: <iframe width="560" height="315" src="https://www.youtube.com/embed/FxFmrirFCmA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> <h2>Lighting</h2> When I started on this I was very short on time, so I had to make some suboptimal decisions. I would probably do a lot of things differently if I had to do it again, so I will not go into much detail. Every light in the world has an own lightmap (normally a lot smaller than the in-game pixels it affects, because of the soft shadows that alleviate a lof of these visual impact of lower resolution lightmaps), which is only update if the light or if the occluders the light affects change (if it throws shadows). I also implemented soft shadows by appending fins to the shadow geometry on both sides and scaling them appropriately. Hard Shadows, 1 Light: <a href="/images/lighting_hard_shadows_1_light.png"><img alt="Hard Shadows, 1 Light" src="/images/lighting_hard_shadows_1_light.png" /></a> Soft Shadows, 1 Light: <a href="/images/lighting_soft_shadows_1_light.png"><img alt="Soft Shadows, 1 Light" src="/images/lighting_soft_shadows_1_light.png" /></a> Hard Shadows, 11 Lights: <a href="/images/lighting_hard_shadows_11_lights.png"><img alt="Hard Shadows, 11 Lights" src="/images/lighting_hard_shadows_11_lights.png" /></a> Soft Shadows, 11 Lights: <a href="/images/lighting_soft_shadows_11_lights.png"><img alt="Soft Shadows, 11 Lights" src="/images/lighting_soft_shadows_11_lights.png" /></a> Old Videos Fixed https://joelschumacher.de/posts/old-videos-fixed 2015-03-10T16:24:42+0000 Some of you may have noticed, that most of the posts tagged with &ldquo;from old blog&rdquo; have broken videos. It turns out that it is not super easy to get a high quality video from unpublished blogspot blog posts (that&rsquo;s what I did with my old blog), but I managed to find most of the old ones. The following posts now have proper videos and I encourage you to watch them in their full HD glory: <ul> <li><a href="http://theshoemaker.de/2013/04/spacewalk-part-1/">Spacewalk</a></li> <li><a href="http://theshoemaker.de/2013/12/metroidlike-2d-water-bone-animations-catching-up-part-1/">Metroidlike</a></li> <li><a href="http://theshoemaker.de/2013/01/feedback-effect/">Feedback-Effect</a></li> </ul> Also I found videos of a few demoscene-related things I started when I was a little younger. I never really participated in the demoscene, but was a long time admirer and of course tried to do similar stuff myself, but the results were rather lame. One of these things is an attempt at a 4KB intro, which you can see here: <iframe width="560" height="315" src="https://www.youtube.com/embed/NIl6jnO0uYE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> In retrospect I think it&rsquo;s quite neat and I probably should have finished it. But I felt a little intimidated and inadequate next to these things: <a href="http://www.pouet.net/prodlist.php?type[0]=4k&amp;page=1&amp;order=views">pouet.net - 4k - sorted by popularity</a> &nbsp; SudoHack update & Replay system https://joelschumacher.de/posts/sudohack-update-replay-system 2015-03-08T14:21:34+0000 So I really have to write another blog post. 120 of the 173 total commits for my Git repository have been after the last blog post and I start to feel bad about putting anything new in the game, because I should have documented all the other stuff already. <h2>SudoHack update</h2> Notably I started putting my <a href="https://gist.github.com/pfirsich/56fc805d9ca7ea62f645">Changelog online</a>, drastically changed the game once, should have posted about it and changed it again. I am currently in the process of changing it another time. The first drastic change was losing your Bits a lot faster over time and gaining a lot more for every enemy kill. This essentially meant that you could not stand still or not kill enemies without dying, which was a lot closer to my creative vision I had from the start. I started feeling rather confident with the game and received good feedback, but I realized: to make a full fledged game out of it that offers the scope that I had in mind, I had to make a lot more levels. Taking in to account that the player only sees a subset of all the possible levels in every run, I estimated I had to make about 300 maps. In conjunction with the fact that it took me a least a week to make 6 I kind of liked, I had to take a different approach if I ever want to finish it (which is one of the main goals of this project), prompting the second radical change. So I revisited the two level generators I already started working on and dismissed pretty fast. And like always you just have to throw enough sweat and man hours at something to make something work and I arrived at something I rather like at the moment. Screenshots: <a href="/images/sudohack_level_generator_1.png"><img src="/images/sudohack_level_generator_1.png" alt="SudoHack Level Generator Example #1" height="240"/></a> <a href="/images/sudohack_level_generator_2.png"><img src="/images/sudohack_level_generator_2.png" alt="SudoHack Level Generator Example #2" height="240" /></a> <a href="/images/sudohack_level_generator_3.png"><img src="/images/sudohack_level_generator_3.png" alt="SudoHack Level Generator Example #3" height="240" /></a> <a href="/images/sudohack_level_generator_4.png"><img src="/images/sudohack_level_generator_4.png" alt="SudoHack Level Generator Example #4" height="240" /></a> <a href="/images/sudohack_level_generator_5.png"><img src="/images/sudohack_level_generator_5.png" alt="SudoHack Level Generator Example #5" height="240" /></a> <a href="/images/sudohack_level_generator_6.png"><img src="/images/sudohack_level_generator_6.png" alt="SudoHack Level Generator Example #6" height="240" /></a> <a href="/images/sudohack_level_generator_7.png"><img src="/images/sudohack_level_generator_7.png" alt="SudoHack Level Generator Example #7" height="240" /></a> <a href="/images/sudohack_level_generator_8.png"><img src="/images/sudohack_level_generator_8.png" alt="SudoHack Level Generator Example #8" height="240" /></a> <a href="/images/sudohack_level_generator_9.png"><img src="/images/sudohack_level_generator_9.png" alt="SudoHack Level Generator Example #9" height="240" /></a> <a href="/images/sudohack_level_generator_10.png"><img src="/images/sudohack_level_generator_10.png" alt="SudoHack Level Generator Example #10" height="240" /></a> The algorithm is essentially a random walk in 2D (a little more fancy than that, but not too fancy) and as you can see in the limit of many steps we can observe a gaussian distribution (as expected with a random walk)! I just put them in to show that this algorithm works rather nicely for small maps (they are very similar to the ones I have built myself) and also quite okay for the medium sized ones. With being able to have maps that are a little bigger than the ones before, I could, of course, not resist and also changed the scale of the maps a little. Sadly though I fairly quicky lost the stupid grin I acquired after implementing feature after feature and seeing the simulation time per frame steadily sticking at about 1ms. Apparently things aren&rsquo;t always O(1) and sometimes even worse than O(n). But that&rsquo;s why I kept a tidy list of any optimizations I could do if need ever be in my Trello, which I honestly couldn&rsquo;t even wait to tackle. A lot of the collision was optimized by only checking collision with the level geometry for the tile the object is currently on and surrounding tiles, but player-enemy and enemy-enemy collision never had a broadphase detection step, which I then implemented. After considering numerous approaches and comparing the necessary gain with the amount of work required I settled with the easiest solution, which is a grid/spatial hash. The whole grid effect was re-tweaked and optimized for only being applied to a little more than the visible tiles. Also other parts of the collision detection were optimized a little, I replaced a lot of if shape.type == &ldquo;whatever&rdquo; with table lookups, which proved to be exceptionally better. But I don&rsquo;t even know why I didn&rsquo;t do it from the start, since the code also got a lot smaller and prettier by doing it. It seems like I reached the point where I grew out of the code I wrote during the beginning of the project, meaning that I know more about the language now and get shivers running down my spine reading the early stuff. Also I implemented collision detection filtering for &ldquo;idle&rdquo; objects and enemy AI is now divided between &ldquo;thinking&rdquo; and &ldquo;acting&rdquo;. The former here represents the current state (and substate) of the enemy and it&rsquo;s transitions between them, usually involving the more costly computations (ray casts, a lot of distance calculations) and the latter the behaviour, mostly consisting of accelerating towards something or shooting players (outrageous!). <h2>Replay system</h2> Something I spent a lot more time on than I should have is definitely gameplay recordings. I can&rsquo;t spend a 40 hour week (or more) on this game and I certainly don&rsquo;t have huge testing capacities, so I wanted to maximize the information I could get from the few testers I have. As far as I see it, there are two fundamental approaches to doing this: <ul> <li>recording user input and just "playing it back" afterwards</li> <li>recording snapshots of the whole game state and relevant events</li> </ul> As far as my analysis went the pros for each method are as follows: <ul> <li> State snapshots <ul> <li>will work even if you're simulation isn't deterministic, therefore less error prone considering differing floating point behaviour for different architectures (and potentially different binaries). But most networked games already have this requirement and deterministic simulations can be achieved without bending over backwards.</li> <li>probably smaller if the dimension of your state space is rather small (few enemies, bullets, etc.) <ul> <li>Also you can choose arbitrary precision for most of your recorded data. It is not important that the player looks exactly in the direction it did while recording if it looks good enough, while it can be dangerous to truncate precision with keypress recording.</li> </ul> </li> <li>a lot less dependent on mini-tweaks. Changing a value a little in enemy behaviour might lead to digressions that eventually destroy the recording</li> <li>you can record only a few seconds in the middle of the game without knowing the full history. This could also be done with keypress-recording but you would have to make a full state snapshot at the start, which is almost half the work of doing it completely with state snapshots.</li> <li>Rewinding and seeking are relatively easy to implement. Using keystrokes this is only possible with making snapshots every N frames and skipping to them.</li> </ul> </li> <li>Keypresses <ul> <li>probably smaller if the dimension of your state space is rather high (only a few inputs for potentially thousands of enemies/bullets)</li> <li>could potentially be coded once and reused for multiple projects (modulo extra meta data), given your simulations are always deterministic</li> <li>can be 'stapled on' later rather easily (again, determinism provided)</li> <li>could be used as a tool to reproduce bugs that are often difficult to produce</li> <li>the user can be recorded in the menu or during the pause screen and observing unwanted keypresses (the user trying out different keys because she/he doesn't know how something is supposed to work) or just how well the user can navigate those menus is essentially for free (but also mandatory). In essence: unwanted behaviour or behaviour not specifically expected might be observed, which is a useful tool for debugging.</li> <li>Gives you another reason to use a fixed timestep, which you definitely should (for framerate independent behaviour/making sure that approximation in the integrator is valid for the used dt, because physics might get weird with too small or too big dt/because it's crucial for networked multiplayer), though in-between frame interpolation might be needed.</li> </ul> </li> </ul> In this case I thought that for my purpose, which is not actual ingame replay of something that is visible to the player but rather a being tool for me, the Keypress recording should be my method of choice. My implementation is Open Source an can be found here: <a href="https://github.com/pfirsich/loveDemoLib">loveDemoLib</a> (see main.lua for a usage example) My googling showed (I could have made a histogram or something and tested it myself) that regular gamepads mostly have a precision of 10-12 Bit for every axis, so I tried using fixed precision for these floats by writing floor(val*GAMEPAD_AXIS_FP_PREC) and reading val/GAMEPAD_AXIS_FP_PREC. Apparently GAMEPAD_AXIS_FP_PREC being 10^5 - 1 or 10^6 - 1 is still enough to throw the simulation off rather quickly. I assume this is because of binning problems e.g. non-homogeneous density of float numbers in the interval -1 to 1 in contrast to homogeneous density of fixed point numbers in this interval, so that some fixed point numbers correspond to multiple floats and some to none. Also I really wanted to implement binary writing, but it is far too much of a hassle in Lua if you want to write numbers (extracting every byte yourself for integers and mantissa/exponent for floats). It&rsquo;s certainly not inconceivably hard, but considering that I wrote the uploader in Python, zipping the file before uploading guarantees that the file to upload is seldom bigger than 1MB, which is in this day and age totally acceptable, I think. <h4>Integration</h4> First I added a pre-commit to my git repository that calls a Python script which then replaced certain variables in my code. For now it&rsquo;s just a timestamp of the commit (I would really like to have a hash, but I can&rsquo;t get the hash of the commit before commiting of course and later commiting the changed file is also suboptimal) so I can (almost) uniquely identify different versions of the game and can always checkout the necessary version to play back a specific recording. Then I added an awesome duo of an uploader, written in Python: <a href="/images/sudohack_demo_uploader.png"><img alt="SudoHack Demo Uploader" src="/images/sudohack_demo_uploader.png" /></a> and a Web interface for adding keys and overviewing and downloading the recordings uploaded by the testers (also I love Bootstrap): <a href="/images/sudohack_demo_webinterface_1.png"><img alt="SudoHack Demo Web Interface 1" src="/images/sudohack_demo_webinterface_1.png" /></a> <a href="/images/sudohack_demo_webinterface_2.png"><img alt="SudoHack Demo Web Interface 2" src="/images/sudohack_demo_webinterface_2.png" /></a> <a href="/images/sudohack_demo_webinterface_3.png"><img alt="SudoHack Demo Web Interface 3" src="/images/sudohack_demo_webinterface_3.png" /></a> They are far from great and there is a lot of stuff I would change if I could justify putting any more work into them, but the only people that will ever use them are probably people I personally know quite well, so I don&rsquo;t feel super bad about it. Of course I also did a bit of other stuff in 120 commits, but I want to write them up in a later blog post with a little more context (and screenshots and videos of course). Tweaking Values in SudoHack https://joelschumacher.de/posts/tweaking-values-in-sudohack 2015-02-22T19:00:44+0000 It is fairly known that just tweaking values in a game a few percent can make or break awesome gameplay and create the difference between an iconic franchise and the joy of gameplay you often get from games that are made by some 13-year olds and uploaded for testing on a game developement board. Not that I don&rsquo;t like games by 13 year olds, just that it took me a ridiculously long time to get the basic hang of making games remotely fun (I&rsquo;m still very far from being good it). And let&rsquo;s face it: if you are a game developer and you&rsquo;re tweaking the friction that is applied to the player when the boss knocks him back using his extra-power-attack, you often stop after 20 minutes and call it a day. It&rsquo;s tedious and even if it that&rsquo;s not always the case, it is seldom any fun. Especially with a game like SudoHack I discover a new game almost every day by just trying out different parameter spaces for all the configuration values in it. Until this morning, for every value I wanted to tweak, I had to make changes in the code and restart the game to view it&rsquo;s effects. There was a system in place, which let me press ctrl+r, generating a tweaks.lua-configuration file (it looked something like this: <a href="https://gist.github.com/pfirsich/854145171298f3175ecf">https://gist.github.com/pfirsich/854145171298f3175ecf</a>) and giving me the opportunity to change these values, hit ctrl+r again and have the changes show up in the game in real time. Even though this is almost neat, it is way too complicated to have actually been used (like once or twice maybe). I remember having something like this in one of the dozen C++ game engines I started developing, when I was a little younger, where I would define a <code>TWEAK(value)</code> macro, which would use <code>__FILE__</code> and <code>__COUNTER__</code> to uniquely identify every call of this macro. The actual implementation would look up the key-pair and upon triggering a reload (by pressing ctrl+r for example) parse all the source files for all occurences of <code>TWEAK</code> and update it&rsquo;s values, therefore giving me the opportunity to tweak everything in the source and only tell the game to reload the tweaks. And I thought it was awesome. So I set out to do something like this in Lua/löve. The first implementation worked fairly quickly using the already introduced tweak(defaultValue, name) functions all over the code, I update the values by parsing the source for these function calls. Because I fear that it might not fully be clear what I&rsquo;m actually doing with this, here is an example work flow: <code>lua -- &lt;implement new feature (for example: bit rendering)&gt; love.graphics.setColor(tweak(40, "bits: colR"), tweak(150, "bits: colG"), tweak(200, "bits: colB"), 255) -- &lt;change the values in the source code, while leaving the game running&gt; love.graphics.setColor(tweak(255, "bits: colR"), tweak(100, "bits: colG"), tweak(100, "bits: colB"), 255) -- &lt;hit ctrl+r&gt; -- =&gt; The changes are visible in the game</code> This is already pretty neat, but one might think: disregarding potential future uses, why do you even need names? Isn&rsquo;t the nth-occurence of a tweak()-function call and the source file enough to uniquely identify the values? And you&rsquo;re right, they are actually quite useless, but if you have a look at my implementation: <script src="https://gist.github.com/pfirsich/51a5a51543da40a3c2fa.js"></script> Then you might formulate this problem as: can we generate the tweak-value names ourselves? And the answer is: I don&rsquo;t know. I asked in #lua on freenode and people recommended me looking at the lua debug module, which has the capability of building a stacktrace at any point in the file, getting the source file name and the current executed line number, but this is not enough if you want to allow multiple tweak values in a single line. I implemented a package loader, which will be invoked if another module is required in a source file that replaces every occurence of a specific <code>func()</code>-call with <code>func(x)</code>, where <code>x</code> is the filename and the number of occurences before it concatenated in a string. For the interested: <script src="https://gist.github.com/pfirsich/549fc8cb38949feef665.js"></script> Despite it working I don&rsquo;t really feel comfortable using it, since I don&rsquo;t like changing the actual entry point off my game to a different file, only to have it be included right after the tweak-loader. Especially if what I gain seems minimal. Also I really don&rsquo;t like messing with things like this, since love&rsquo;s filesystem sandbox might make this a little more complicated if it is run from a zip-archive (which is, in a shipped version, the default mode of operation), but I&rsquo;m not entirely sure about this. Finally I will just use the system that was in place today 5 hours ago and see how much entering names will annoy me pretending that they might have an actual use in the future. All the approaches discussed here are, of course, only actually usable in small teams or, more concretely, teams where everyone, who should have the possibility of tweaking game values, knows her/his way around the code. Otherwise a system where code and data are more separated is probably the way to go, since the amount of overhead can be justified with expanding the access to configuration values. Also in that case I would really like some Quake/Source engine like console, which makes easy changes in the game possible and gives the opportunity to spawn graphical elements like sliders/spindowns to edit them visually. For me though, the edit-quick and almost no-overhead solution is more desirable. Exam Preparation Avoidance Game Jam https://joelschumacher.de/posts/exam-preparation-avoidance-game-jam 2015-02-19T17:28:57+0000 Last week <a title="Jonas Lieb" href="http://www.jonas-lieb.de/">Jonas</a> and I knew that we had to start preparing for our upcoming condensed matter physics exam. Instead we jammed a little game, which should initially only take up one single day of our time, but actually took two. It&rsquo;s a blatant copy of this game: <a href="http://www.bildspielt.de/tipps/fluglotsen-training-fur-jeden-airport-madness-3-109441/">Attention, German!</a> Copying games is really interesting and it&rsquo;s always fun figuring out how you can implement something, while maybe even improving it a little or at least giving it a bit of your own twist. I could go on more about this, but I actually just want to keep this blog more complete than comprehensive and actually interesting. If someone will ever read this and maybe even contemplates contacting me about something, then please do so without hesitation. Until then: Screenshot: <a href="/images/fluglotse_screenshot.png"><img alt="Screenshot - Game" src="/images/fluglotse_screenshot.png" /></a> Editor Screenshot (No actual graphical interface, just a plethora of keyboard shortcuts, a &ldquo;command line&rdquo; and only about 3-4 hours of work): <a href="/images/fluglotse_editor_screenshot.png"><img alt="Screenshot - Editor" src="/images/fluglotse_editor_screenshot.png" /></a> Video: <iframe src="https://www.youtube.com/embed/jN1GjQzoKJM" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"></iframe> <a href="https://github.com/jojonas/fluglotse">Github link</a> <a href="http://download.theshoemaker.de/Fluglotse/">Downloads</a> Don&rsquo;t expect it to be particularly fun. It was way more fun to make than it is to play it. &nbsp; Global Game Jam 2015 - "Falling Nukes + Running People" https://joelschumacher.de/posts/global-game-jam-2015-falling-nukes-running-people 2015-01-28T22:49:58+0000 Last weekend my notorious partner in crime, <a href="http://jonas-lieb.de/">Jonas Lieb,</a> his artist friend Lukas and I attended the Global Game Jam 2015 in Cologne. After watching the keynote, which was not only slightly cringeworthy and partly in a language no one on the site could understand, but also way too long and obviously tinted by Anita Sarkeesian ideology (she was also listed in the section for &ldquo;Special Thanks&rdquo;) the theme was announced as &ldquo;What do we do know?&rdquo;I initially expected something way to artsy and narrow, like one could expect from the GGJ organizers, but I was very pleasantly surprised. My first idea was making a game with no objective, similar to Proteus, but local mulitplayer and, of course, 2D, but somehow one of the guys had the idea of a game, which was also explorative, but rather a quest of finding something specific rather than finding anything. We decided to make a game in which the players are placed into a world with an obvious threat, initially during brainstorming referred to as a comet, but later implemented as the impending impact of nuclear missiles. It is then the players&rsquo; objective to find a way out of this situation by exploring his/her surroundings and finding a way to escape the impending death. We planned on placing numerous different items in the world, which only had capacity for a single player, therefore leaving the rest of the group behind posing the theme question and giving the rest of the team the incentive to step up their efforts. The first major dispute was about the underlying tech of the game either being tile based and only featuring full tiles (i.e. squares) or (my stand point) using arbitrary polygons, which would be edited in a editor specifically made for this game and full images for the whole map rendered above these polygons (which were used for collision). Of course this is a very crude approach and far from efficient, but it was the easiest way this could have been done without major preparatory work. The reason for me rooting for the polygon approach was the fact that, for the first time, we could facilitate having an actual artist who knows what he&rsquo;s doing and leave him a lot of freedom in designing the levels and all the layers above and below it. Jonas though was of the opinion that a tile based approach was not just a lot easier to implement and was something we did numerous times before, but also gave us the opportunity to create levels a lot faster, without Lukas being the obvious bottleneck of the level creation progress. Of course he was right, but I always prefer trying to make a cool game people will find cool rather than doing something mediocre, that will more probably pan out successful. Thankfully Lukas was on my side and we went with the fancy approach. After about 24 hours (without any sleep by the way, because I really can&rsquo;t sleep well if I really want to work on something) we had most of the engine running, a test level and could try out a first prototype of the game we had in mind. Quickly we noticed that our idea was flawed in a way or not feasible in such a short amount of time because it seemed way to easy to find a way out, since the objects were clearly stood out as special/objects to interact with. It would have needed a lot more objects, which were actually useless. And we think this wouldn&rsquo;t have changed the feel of the game to an explorative one, but rather to a game which is full of useless stuff, you have to filter out by mashing the &ldquo;use&rdquo; button. We didn&rsquo;t want that and were trapped in a kind of design bubble. I started integrating a little &ldquo;shove&rdquo;-mechanic which pushed and stunned the player for a short period of time and after testing it we noticed that this is probably the most fun element of the game so far and decided we should focus on it. We then made the decision to shift the game into a lot more competitive platformer, with a lot of punching and maybe other fighting mechanics and a lot more linear levels, that forced the players to initiate confrontations trying to reach the escapes. Sadly this sounds as stereotypical and bland as it gets but it was a lot more fun than our previous ideas would have been, I think (at least if we only had 48 hours to invest). The final result can be seen here: <a href="http://globalgamejam.org/2015/games/falling-nukes-running-people">Falling Nukes + Running People</a> All in all I am very satisfied with the result. The crowd seemed to like the game and the testers had a lot of fun beating up each other with a little timer in the upper right corner. I&rsquo;m also very happy that technically it turned out a lot easier than Jonas expected. Also as a very nice side effect another jammer might have joined our ranks. And even better: an (in my opinion very competent) artist! As a conclusion to the GGJ this year I learned that it is very important for multiple team members to clearly state their goals in making a game to ensure to make the right decisions along the way (unfortunately this was a little problem in our group, as we discovered afterwards). Also I should spend a little time (Hah! &ldquo;little&rdquo;&hellip;) to work on a proper animation editor, since Lukas uttered some dislike in having to draw numerous frames for a single character. At first I thought about a bone animation editor, like <a title="Metroidlike &amp; 2D-Water &amp; Bone-Animations – Catching up Part 1" href="http://theshoemaker.de/2013/12/metroidlike-2d-water-bone-animations-catching-up-part-1/">the one I started for Metroidvania</a>, but I&rsquo;d much prefer a hybrid like in the beginning of this video: <a href="https://www.youtube.com/watch?v=y-chi097uV4">Youtube: How Rayman Legends is Made</a>. Furthermore I really miss the collision detection library I&rsquo;m planning to make once I have time to finally get rid of the bullet through paper problem, that also proved to be quite annoying during the global game jam. But first: exams, not screwing up my Bachelor&rsquo;s and SudoHack. &nbsp; New Website! https://joelschumacher.de/posts/new-website 2015-01-20T15:51:39+0000 I&rsquo;m using wordpress on a new webspace now. This is the result! Wohoo! Because googling <a title="my name" href="http://en.wikipedia.org/wiki/Joel_Schumacher">my name</a> will never result in anything related to me, I had to choose this domain. I&rsquo;ll be participating in the global game jam this weekend and will surely inaugurate this website properly shortly after it and when I find time, I will make a short &ldquo;projects&rdquo; page with a little overview of the stuff I&rsquo;ve done. But what will probably take a while. SudoHack - Bits, AI, Weapons, Movement and a Lot More Prettiness https://joelschumacher.de/posts/sudohack-bits-ai-prettiness 2015-01-13T16:07:00+0000 SudoHack is a multiplayer roguelike where you and your friends, playing as computer programs, try to infiltrate a computer system. You gain &ldquo;Bits&rdquo; by destroying the guardian programs that are put in your way, but also lose them for every hit you take and every second that passes by. Your goal is to reach the end of the room before you run out of Bits and get destroyed yourself, therefore gaining access to deeper levels of the system, until you reach the core. So this week really has been rather productive (sadly just regarding SudoHack. Video (now on Youtube!): <iframe width="560" height="315" src="https://www.youtube.com/embed/C2Jum5neMJU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> I added Bits (the little blue things you can see in the video), which serve as a HP/time countdown hybrid. They decrease over time and you lose them by taking damage, representing your integrity as a little computer program guy, because, you know, that&rsquo;s exactly how that works (#1337hackz0r). I plan on making the games a little more swingy, by picking up Bits being a lot stronger, but also losing a lot more over time. Awesomely though they dont&rsquo;t just have an effect on a dandy little bar you can fill and empty, but also trigger a (placeholder) death screen, giving you the opportunity to start again. Which leads me to the <a href="http://en.wikipedia.org/wiki/JMP_%28x86_instruction%29">jmp</a>-tiles, which don&rsquo;t just enable you to restart the game, but advance you to other maps once you cleared the room of all enemies, being a key part of the game and also finally implemented! Also the random weapons have been changed, so I don&rsquo;t have to restart the game ten times before I can make a video ;) Now they are a lot more even in power level (by equalizing metrics like damage per second), but I don&rsquo;t have the impression that they feel that diverse at the moment, which may change after I implemented a lot more special powers or I might just drop the random weapon idea alltogether and stick with weapon upgrades or something. Testing will tell! Also I integrated ShotSynth, which I mentioned (and showcased) in <a href="http://theshoemaker.de/2015/01/sudohack/">a previous blog post</a>. Some of you might have noticed (definitely those of you who had the gift of being asked to test the game) that the enemies are not as clever as they could be and often get randomly stuck or behave a little strange. I tweaked this a little and removed two huge bugs I&rsquo;m still ashamed to even think about, let alone mention to any living person. For days they collided with dead enemies, and significant parts of their behaviour were never executed (still did it). Formerly the movement was just a regular euler integration with friction and keypresses corresponded to forces that were being exerted on the player mass. But I think that it was really hard to dodge bullets and all in all the movement looked and felt very floaty, albeit smooth. I changed it to partly setting the velocity and partly interpolating into a target velocity, which should (and did) give the game a little more of a Gradius/R-Type feel regarding movement. I really like the change, though some have expressed dislike (In my opinion, mostly because of the significantly different feel).&nbsp; Also I invested a little time to make the game compatible with the 360 controller and added the traditional twinstick shooting controls, so you don&rsquo;t have to also press a button to shoot. Finally the part that was definitely the most fun and the stuff that interests me the most in game programming: the graphics part. At first I added the grid effects, which can be explained if there is any need for it (just tell me), but certainly help to give the game it&rsquo;s cyber feel I was aiming for and also make the level a lot more dynamic and lively. To enhance these effects and because no cyber space game can live without copious amounts of it I added bloom/a glow effect, which still might be a little too much at the moment and will probably be toned done, but not until I got completely sick of it, which might take longer than anyone likes (but please tell me if you think that I should tackle my severe case of bloom addiction and kick it down a notch). Also I added a scanline and a rgb-shift shader, which should give the game the look of an old monitor, because again: cyber space and stuff. I plan on adjusting their parameters using the current amount of Bits, compromising the image more and more with less Bits being left. The bullets look a little prettier too, players have awesome, yet still unexplicable cyber-tails and on top of it all I managed to squeeze in approximately 60 hours of research and implementation for a neat little aim line. I also tried a few level generators, that I really, really didn&rsquo;t like and at the moment I plan on not having random maps, but rather prebuilt and randomly chosen rooms. I will definitely put more time into a generator first, after I built a few to get a feel of what works and what doesn&rsquo;t. The immediates milestones are a semi-working prototype for the <a href="http://globalgamejam.org/">Global Game Jam</a>, which I will attend from the 23rd to the 25th this month. For this the next things on my ToDo-list are a lot of minor changes, new enemies, a few maps and a new (wordpress) blog! SudoHack - Enemies and Prettiness https://joelschumacher.de/posts/sudohack-enemies-and-prettiness 2015-01-07T00:09:00+0000 SudoHack is a multiplayer roguelike where you and your friends, playing as computer programs, try to infiltrate a computer system. You gain &ldquo;Bits&rdquo; by destroying the guardian programs that are put in your way, but also lose them for every hit you take and every second that passes by. Your goal is to reach the end of the room before you run out of Bits and get destroyed yourself, therefore gaining access to deeper levels of the system, until you reach the core. I really should have spent the last few days on university stuff, but I really couldn&rsquo;t resist, I integrated two enemy types (a static turret and a &ldquo;moving turret&rdquo;, as it&rsquo;s called in the code) and a background for the &ldquo;floor&rdquo;, which was previously just black. Also the bullets look a little different now and being hit by enemy fire decreases your shared HP (&ldquo;Bits&rdquo;!) now, just as the general passage of time does. Looking at a diff from gitk tells me that I made a lot more changes, but most of them are not interesting or tangible enough (for non-programmers) to talk about. Also: screen shakes, flashes, explosions - the cool stuff. I think a video is the best I can do now, suspense curve-wise: <iframe width="560" height="315" src="https://www.youtube.com/embed/Lb_SQ-z_g94" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> Here my roommate (thanks, Jonas!) and I tried a little round in one of the first (and highly unrefined) maps. The weapons are already randomly generated and trying them out or just trying to figure out the AI (which I programmed myself, but sadly still can&rsquo;t fully predict) is already quite fun. One of the biggest problems I encountered during this developement session is visual design in general. I had a really hard time figuring out proper palettes for the game and am still not really convinced by them. Also I&rsquo;m not sure if the current general art style of the game is what I have in mind for it. The general atmosphere is also still very much uncertain, but, I think, largely influenced by music and can easily be shaped with it. If anyone has good ideas regarding the design (colors, graphics, whatever it is), please let me know! The AI also still has it&rsquo;s weaknesses (I think at the moment the AI is only a weakness) and it proved a lot harder than I thought to design proper state machines for convincing enemy behaviour. I also realized that this game is very, very dependant on proper balancing. Changing just a few values even a tiny bit has huge impacts on the enjoyability of the game and I think that it will take many, many hours to get it right. I&rsquo;m happy that it&rsquo;s still a while until I have to seriously tackle these problems, but it also makes me happy that a very fun game might just be a digit after the decimal place away! The next steps I&rsquo;m going to tackle are various changes that, ideally should have no effect on the game dynamics themselves, but tidy up the code base a little and optimize some things that require it at the moment. After that I will change the way the game handles configuration files and make the first distributable version for the readers of this blog to enjoy! I hope that it will take me a few days (or a week), because that means that I spent my time on important stuff. Let&rsquo;s see how that goes.. SudoHack https://joelschumacher.de/posts/sudohack 2015-01-05T01:05:00+0000 During this holiday season I started to work on a &ldquo;new&rdquo; project. I actually started it quite a few times, but never really worked more than a few hours on it before I lost interest. But this time I don&rsquo;t just want to try a few things and code a small game, I want to know if I can finish one. Making a game from start to finish. With a menu and all (I never made menu). Also I often fantasized about about taking a year off after I got my Bachlor&rsquo;s degree (if it comes to that) and be an indie dev - this is how I want to find out if I could actually do it. SudoHack is going to be a a multiplayer roguelike, inspired by the numerous recent roguelikes, like The Binding of Isaac, Rogue Legacy, FTL: Faster than Light, which I played a lot of. It is set in cyber space and you (and your mates) control little programs that try to infiltrate a system (which explains the name, if you ever used Linux, where the sudo command executes another command as the superuser). The key element is (and of course I&rsquo;m not sure if it will actually end up that way) getting from room (which are prebuilt, but randomly chosen) to room without losing too many &ldquo;bits&rdquo;, which you will pick up as loot from your defeated enemies and also lose by taking damage. They are essentially a ticking timer and HP combined. The goal is to finish the room in time, get to another room and go from there, defeating bosses along the way and hopefully infiltrating the system completely at the end. So far I have a lot of boilerplate code, a mini collision library and shooting with bouncy bullets! <div style="clear: both; text-align: center;"></div><div style="clear: both; text-align: center;"></div><div style="clear: both; text-align: center;"></div><div style="clear: both; text-align: center;"></div><div style="clear: both; text-align: center;"></div><div style="clear: both; text-align: center;"><object width="320" height="266" id="BLOG_video-35f59830c90472bb" class codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0"><param name="movie" value="//www.youtube.com/get_player"><param name="bgcolor" value="#FFFFFF"><param name="allowfullscreen" value="true"><param name="flashvars" value="flvurl=http://redirector.googlevideo.com/videoplayback?id%3D35f59830c90472bb%26itag%3D5%26source%3Dblogger%26app%3Dblogger%26cmo%3Dsensitive_content%253Dyes%26ip%3D0.0.0.0%26ipbits%3D0%26expire%3D1423912765%26sparams%3Did,itag,source,ip,ipbits,expire%26signature%3D3FDFDDBE58501BDEDA2DE4C9D74F73FB4ADED127.AA9C6B5C813403505562093BFC6062F9396BC236%26key%3Dck2&amp;iurl=http://video.google.com/ThumbnailServer2?app%3Dblogger%26contentid%3D35f59830c90472bb%26offsetms%3D5000%26itag%3Dw160%26sigh%3DclBYM312c03Swe4RQMq9882zRpg&amp;autoplay=0&amp;ps=blogger"><embed src="//www.youtube.com/get_player" type="application/x-shockwave-flash" width="320" height="266" bgcolor="#FFFFFF" flashvars="flvurl=http://redirector.googlevideo.com/videoplayback?id%3D35f59830c90472bb%26itag%3D5%26source%3Dblogger%26app%3Dblogger%26cmo%3Dsensitive_content%253Dyes%26ip%3D0.0.0.0%26ipbits%3D0%26expire%3D1423912765%26sparams%3Did,itag,source,ip,ipbits,expire%26signature%3D3FDFDDBE58501BDEDA2DE4C9D74F73FB4ADED127.AA9C6B5C813403505562093BFC6062F9396BC236%26key%3Dck2&iurl=http://video.google.com/ThumbnailServer2?app%3Dblogger%26contentid%3D35f59830c90472bb%26offsetms%3D5000%26itag%3Dw160%26sigh%3DclBYM312c03Swe4RQMq9882zRpg&autoplay=0&ps=blogger" allowFullScreen="true" /></object></div>&nbsp;Also, as you can see, local multiplayer is already implemented (the green guy is controlled using my controller and the red one using my keyboard and mouse). It is planned to also integrate networked multiplayer, but since I have never done this before (in contrast to everything else I have planned for this game) I&rsquo;m a little afraid of how it will turn out. I&rsquo;m confident that I can at least hack together something, that probably works only on LAN, but I hope that I can do just a little bit more. Also this time I&rsquo;m using <a href="https://love2d.org/">LÖVE</a> again, which is, as usual, a pleasure. Next up are enemies, which I already designed a few of and really can&rsquo;t wait to integrate into the game. One thing I already worked on way too much for a prototype is random weapons, which will be dropped as loot in the game. Every player starting the game will already have a randomly generated weapon (about 14 parameters). And for this feature in particular I started working on a shot sound synthesizer. Most of it&rsquo;s features can be seen in the description text in the video below. Obviously it also contains a few sounds I generated just pressing the space key a few times. <div style="clear: both; text-align: center;"><object width="320" height="266" id="BLOG_video-3d1b73b9e269d9fa" class codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0"><param name="movie" value="//www.youtube.com/get_player"><param name="bgcolor" value="#FFFFFF"><param name="allowfullscreen" value="true"><param name="flashvars" value="flvurl=http://redirector.googlevideo.com/videoplayback?id%3D3d1b73b9e269d9fa%26itag%3D5%26source%3Dblogger%26app%3Dblogger%26cmo%3Dsensitive_content%253Dyes%26ip%3D0.0.0.0%26ipbits%3D0%26expire%3D1423912765%26sparams%3Did,itag,source,ip,ipbits,expire%26signature%3D5746C07FE79D3C59650A53D3F51FBF00DDF5D1FA.A0D67504075F27A400C97C8708CFB297375F506A%26key%3Dck2&amp;iurl=http://video.google.com/ThumbnailServer2?app%3Dblogger%26contentid%3D3d1b73b9e269d9fa%26offsetms%3D5000%26itag%3Dw160%26sigh%3DeI2XHNWHlgfaduoZ14yLn5fD7IU&amp;autoplay=0&amp;ps=blogger"><embed src="//www.youtube.com/get_player" type="application/x-shockwave-flash" width="320" height="266" bgcolor="#FFFFFF" flashvars="flvurl=http://redirector.googlevideo.com/videoplayback?id%3D3d1b73b9e269d9fa%26itag%3D5%26source%3Dblogger%26app%3Dblogger%26cmo%3Dsensitive_content%253Dyes%26ip%3D0.0.0.0%26ipbits%3D0%26expire%3D1423912765%26sparams%3Did,itag,source,ip,ipbits,expire%26signature%3D5746C07FE79D3C59650A53D3F51FBF00DDF5D1FA.A0D67504075F27A400C97C8708CFB297375F506A%26key%3Dck2&iurl=http://video.google.com/ThumbnailServer2?app%3Dblogger%26contentid%3D3d1b73b9e269d9fa%26offsetms%3D5000%26itag%3Dw160%26sigh%3DeI2XHNWHlgfaduoZ14yLn5fD7IU&autoplay=0&ps=blogger" allowFullScreen="true" /></object></div>&nbsp; Both SudoHack and the ShotSynth in their current versions are available on GitHub: <a href="https://github.com/pfirsich/ShotSynth">https://github.com/pfirsich/ShotSynth</a> <a href="https://github.com/pfirsich/SudoHack">https://github.com/pfirsich/SudoHack</a> If anyone in the world ever reads this (highly improbable) and doesn&rsquo;t know how to get this game to run on his computer using the GitHub repository I linked here, but really wants to try, comment this blog post and I&rsquo;ll upload a binary :) The Delightful Reunification With Modify-Assign in Lua and Per Torpedum ad Astram https://joelschumacher.de/posts/modify-assign-in-lua-per-torpedum-ad-astram 2014-12-30T12:42:00+0000 Jonas, who already worked on Spacewalk with me, and another fellow student of mine recently became roommates after Jonas and third said roommate spent a year abroad. Jonas and I decided to visit the Global Game Jam in Cologne in January and felt it was necessary to do some kind of warm up to get comfortable with the technology we plan on using and produce some useful code for loading <a href="http://www.mapeditor.org/">Tiled</a> (which just keeps getting better and better, geez!) maps and general boilerplate. After a while I again noticed the painful lack of <code>+=</code>, <code>-=</code>, <code>*=</code> and <code>/=</code> operators in Lua. I remembered writing a Python script (which is bad) that compiled &ldquo;.mond&rdquo; to Lua only adding said assignment operators in functionality and significant whitespaces instead of those ugly <code>then</code>/<code>do</code>/<code>end</code> blocks. I now know though, that it is possible to add loaders to Lua that will be invoked once a module will be &ldquo;require&rdquo;d in a source file. So I wrote a loader that gives me those missing operators (replaces them before passing the source to loadstring)! The loader and an example of this using löve2D can be found here: <a href="https://github.com/pfirsich/ModifyAssignInLua">https://github.com/pfirsich/ModifyAssignInLua</a> If I or you (if anyone would ever read this) would like to use it, you just have to rename your actual <code>main.lua</code> to <code>realmain.lua</code> (or however you want to name it. But don&rsquo;t forget to change the name in <code>main.lua</code>!) and include <code>mond.lua</code> and <code>main.lua</code> in your project folder (without löve the usage is very similar to this). As you can see in <code>mond.lua</code>, line 33 ff. I also kept the option to parse completely different source files and do something like I did in the Python version of Mond (replacing the Lua blocks with significant whitespaces (actually the other way round of course)), but I think that this is, though nice and fun, a little of a waste of time, since I should actually be working important things. The last 5 posts were part of the TAG Jam, which I got to host, since I won TAG Jam 5: <a href="http://blog.ratalaika.com/tag/tagjam/">http://blog.ratalaika.com/tag/tagjam/</a> (here you can find my game too of course! Read the readme!) (Update 7th of April 2015: This link seems to be broken. You can <a href="https://www.dropbox.com/s/gvb3vomavw5lrxu/PerTorpedumAdAstram.zip?dl=0">download it from my Dropbox</a> instead. Or <a href="https://www.youtube.com/watch?v=723aAV4LAd8">watch a video of it on Youtube</a>). I think the version that can be downloaded on Ratalaika&rsquo;s blog is not the most recent one and not the one with all known bugs fixed and all typos corrected, but this game is already almost one year old, so I don&rsquo;t really know what is what. Screenshot for completeness&rsquo; sake: <a href="/images/per_torpedum_ad_astram.png"><img alt="Per Torpedum Ad Astram" src="/images/per_torpedum_ad_astram.png" /></a> This time, as stated in the readme, I worked on the game with a lot less pressure. I didn&rsquo;t want it to be good. I did it myself and I just wanted to spend a little time programming a game that I think would be cool. I purposely kept the scope of the game as small as I could imagine and this time it was just right for 48 hours. Apparently I still have problems keeping the scope small enough and not trying to do too much. This was easily the most fun game jam yet. Also the game being a game emulating the Starfox gameplay helped a lot, since it is one of my favourite franchises and one I have the fondest memories of. I&rsquo;m also a little proud of the unique graphics style. By that I don&rsquo;t just mean the exceptionally sub-par artwork, but the pseudo 3D visuals and the clouds and the graphics feel in general. TAG Jam #6 - Results https://joelschumacher.de/posts/tag-jam-6-results 2014-01-08T00:01:00+0000 Hi guys! I found some time and decided to playtest/write a resumé. First of all I found it really interesting that there are not that many multiplayer games (just one) and that one also tried to make a game without an objective. I&rsquo;m really happy that all of you tried to make something different :) <h3 id="clouds">Clouds</h3> This is exactly what I meant! Games like this. I think this one fits what I had in mind best. Many exploration games (like they were made here and playground-games) kind of have an objective in a sense that you want to do something. This one though is a game where you are finished the second you start playing, but you choose to keep on going. Like a scenery or a photograph, but dynamic and interactive. The music (which is really good by the way!) changes are a little abruptly and is disrupting the mood in my opinion, which is super important to avoid in a game like yours. All in all it&rsquo;s, sadly, just not very much and even if you decide to keep on playing, it loses it&rsquo;s appeal fairly soon. Technically: Maybe you could have spent some time investigating how to visualize rain nicely. Even rain drops with different speeds would have been a little more realistic. So it would have if you paid more attention on the tiling of the rain texture. I also don&rsquo;t like that the rain clouds get repelled and move away with constant speed. Maybe only repulsion would&rsquo;ve looked nicer - not entirely sure. <h3 id="you-only-need-one">You Only Need One</h3> First of all the title really reminds me of the last Ludum Dare theme (You Only Get One). At first I was a little confused since no one told me how to control the game! This is something that you should pay attention to. After figuring out to use the arrow keys though I must say that I really liked the setting and it really awakened my interest. The pacing of finding out what&rsquo;s happening in the beginning is really good in my opinion! I like using the word &ldquo;odd&rdquo; as an inspiration and am generally delighted by the setting. It&rsquo;s continually more interesting! I don&rsquo;t like though that the game actually has an objective. You want to get to the end, because you know there is one. It&rsquo;s just not explicitly stated. The suspense curve really takes it&rsquo;s time to bottom out. The last part is not very rewarding considering the growing platforming-difficulty. Technically: I&rsquo;m not sure how easy it is to do this in flash, but I would have liked it if the speech bubbles faded out. Also the music loops very strangely and starts again when you just forgot about it. Constantly playing music would have improved the experience, I guess. <h3 id="go-figure">Go Figure</h3> I really like the name! It&rsquo;s a nice thing to play around, but loses it&rsquo;s appeal rather quickly. It&rsquo;s more of a toy than a real &ldquo;game&rdquo;. I don&rsquo;t want to be mean, but just because a game doesn&rsquo;t have a game, it should still be enjoyable! That was part of the challenge. Something where it&rsquo;s all about the experience and the mood. At least that&rsquo;s what I had in mind and because it&rsquo;s just me, I should probably not criticize your game for it! <h3 id="revel">Revel</h3> I really like the mood and the music and the graphics! Atmosphere-wise you did a really, really good job. The location is intriguing and the character as well (4 arms, when I first jumped! Very cool moment). Execution-wise this is probably the best game (it has to be mentioned though that you are a significant amount of people) and fits the Outsider theme very well. I really like that the game doesn&rsquo;t stop to give you something to explore. What I don&rsquo;t like is that it eventually does and not in a very elegant fashion. I probably just got stuck, because I didn&rsquo;t know that to do, but that&rsquo;s not just my fault. I collected the two artifacts (which I think is a little silly of you to write directly on them :D) and found the little cave below the starting room which you can&rsquo;t see much in. I&rsquo;m actually pretty sure I&rsquo;m stuck, but I don&rsquo;t know that to do. So I&rsquo;m not really sure if it&rsquo;s really without an objective, since having a goal and being able to finish still kind of qualifies as one. Also I think the game would have benefited a lot from parallax effects in the background! <h3 id="uphero">Uphero</h3> (At least I hope that it&rsquo;s called that way). The first 4 times I started the game the player apprently spawned in the level geometry, which really confused me and made me restart the game until it did not. After I&rsquo;ve seen him and played the game I of course realized that I could set him free by deleting the blocks. And when I saw him I was happy to find such a cute little guy. He&rsquo;s awesome. And the music is too. It&rsquo;s not less groovy than you said (which is uber). I generally don&rsquo;t like that you just took an already existing game and tried to clone it. In general this is not a bad thing, and it doesn&rsquo;t make the game worse, but is a minus point when it comes to winning (which is probably understandable). The sun and the clouds look just a nice as the alien, but I don&rsquo;t really like that it&rsquo;s so easy to place blocks. I think that Minecraft and Terraria make building things rewarding, because it&rsquo;s work. It&rsquo;s just too easy to do something. Also I struggle with rather low FPS once I move. When moving the FPS really fluctuate quite a bit, which is really disruptive. PS: Sorry for pointing out the positive stuff at the beginning and the negative stuff at the end. I don&rsquo;t want to rewrite it, but don&rsquo;t think that I disliked any single one of your games! All of them were a pleasure to play! I&rsquo;m really happy that five people took their time on a weekend to participate in a game jam I had the pleasure to host! Thank you all for participating! I hope you had a lot of fun and learned equally as much! <h3 id="to-cut-to-the-chase-the-winner-is-revel">To cut to the chase, the winner is: Revel!</h3> After playing all your games I didn&rsquo;t find it easy to decide which is the best, but I think that it&rsquo;s actually just the one which yielded the most enjoyable experience during the time playing (even though there a some things to criticize, as with all the games), which was your team&rsquo;s, Charles! Congratulations! TAG Jam #6 - Time's Up! https://joelschumacher.de/posts/tag-jam-6-times-up 2014-01-06T00:00:00+0000 The jam time has ended! I hope you had a lot of fun during the last 48h+! Please upload your games now and link to them in the comments below. You will then have 12 hours to fix bugs and package everything. The final deadline for uploading your games is Monday 12:00 GMT/Monday 04:00 PST. I try to write up a little review for all the games and determine a winner by the end of the week. I sadly didn&rsquo;t finish my game (not even remotely) due to a rather busy weekend, but I am super excited about what you came up with :) TAG Jam #6 - The Jam Time Begins! https://joelschumacher.de/posts/tag-jam-6-the-jam-time-begins 2014-01-04T00:00:00+0000 And the second, even more special, moment has finally come too. The jam time begins! <a href="http://theshoemaker.de/2014/01/tag-jam-6-themes/">- The Themes - </a> <a href="http://theshoemaker.de/2013/12/tag-jam-6-welcome-post/">- The Rules -</a> The deadline is now in 48 hours, respectively Sunday 24:00 GMT/Monday 00:00 GMT/Sunday 16:00 PST. Then you will have 12 hours submission time. Feel free to add the 12 hours yourself and not make me write down all the different times in all the different time zones again. Your submissions can be linked to under another post I will create when the jam time ends. Have fun! :) TAG Jam #6 - Themes https://joelschumacher.de/posts/tag-jam-6-themes 2014-01-03T00:00:00+0000 Okay! The special moment has come; the magic begins. No screwing around - the themes: <ul><li>Outsider - &ldquo;a person who does not belong to a particular organization or profession&rdquo; / &ldquo;a competitor, applicant, etc. thought to have little chance of success&rdquo;</li><li>Drawback - &ldquo;a feature that renders something less acceptable; a disadvantage or problem&rdquo; </li><li>Affection - &ldquo;a gentle feeling of fondness or liking&rdquo;</li></ul>(Definitions from oxforddictionaries.com) There they are. All (semi-)randomly (I picked them from a larger pool of words) generated by <a href="http://www.wordgenerator.net/random-word-generator.php">http://www.wordgenerator.net/random-word-generator.php</a> Your game has to do with one or more of these themes. The jam time will officially start in 24 hours which is Friday 24:00 GMT/Saturday 00:00 GMT/Friday 16:00 PST/4pm PST (now I hope that everybody knows when it is) and will last for 48 hours. Don&rsquo;t forget the special rules for TAG Jams half-birthday! - <a href="http://theshoemaker.de/2013/12/tag-jam-6-welcome-post/">Look here!</a> TAG Jam #6 - Welcome Post https://joelschumacher.de/posts/tag-jam-6-welcome-post 2013-12-23T16:26:00+0000 Hi everyone! Sorry for the delay, but here&rsquo;s the countdown/welcome-post for #TAGJam6! I won&rsquo;t be using twitter, but for all you interested in doing so, the aforementioned one is the hashtag to use. Ticktakashi pointed out that with this one TAG is celebrating it&rsquo;s half-birthday (which totally is a thing)! Therefore we will have special rules and since I&rsquo;m not really big on decision making, we will just include a set of two additional rules from which you will have to choose one (or more, which is two and also all of them). These rules are as follows: <ul><li>Create a multiplayer game (local multiplayer is of course fine too and probably the way to go with such time constraints)</li><li>Create a game without an objective (and (extra sugar) without enemies)! Whether it may be explorative in nature or just some kind of interactive experience or even something else. Make something different!</li></ul>The themes (2 random ones and one of my choice) will be announced on Thursday January 2nd 24:00 GMT/16:00 PST and the game jam will start 24 hours later on Friday 3rd at 24:00 GMT/16:00 PST (e.g. Saturday 00:00 GMT), so you have 24 hours to ponder on the themes. Feel free to comment this blog entry, so I know who of all you awesome people I can expect to participate! The full rules are here: <a href="http://www.thearbitrarygamejam.com/">http://www.thearbitrarygamejam.com/</a> Edit: I did a mistake on the dates. I changed my Windows-calendar to January without changing the year -.- Metroidlike & 2D-Water & Bone-Animations - Catching Up Part 1 https://joelschumacher.de/posts/metroidlike-2d-water-bone-animations 2013-12-20T16:58:00+0000 I have a huge list of stuff that I should have put on my blog (just to document what I&rsquo;m programming and maybe have a look at the progression afterwards) and today is the day I try to catch up. I&rsquo;m thinking about making big posts with multiple things I worked on, so I don&rsquo;t have to create a myriad of posts today and also because some of them are related. Especially the three in this post, since they were intended to be part of the same game. <h2>Metroidlike</h2> I love the Metroid franchise. Almost every metroid game is one of my favourite games ever (except the Wii-Titles since I haven&rsquo;t played them). And for anyone wondering which I think are the best: My number one is Super Metroid without a doubt. I adore this game indefinitely. Prime and Fusion in no particular order are also among the better ones. So all along my time of programming games I started many Metroid 2D or Metroid Prime clones (though I never finished one) and I felt that it is again time to make one, since I know a whole lot more and had nothing to to do and a huge urge to start a project. Although I&rsquo;m doing mostly 2D-stuff lately I&rsquo;m a really graphics-affine guy and I really like to spend time making effects and I&rsquo;m really interested in graphics techniques, so naturally I spent most of my efforts on making the game look nice. First a video: <center><iframe width="560" height="315" src="https://www.youtube.com/embed/dJSdzbMAj7Y" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></center> As you will see again I tried to make the content-creation-pipeline for this game really programmer-friendly, so part of the look has to do with that. I know that tiles are a really easy way to make level geometry, but I think that they often just look a little jaggy and not as next-gen as they could, so I decided to make &ldquo;high resolution tiles&rdquo;, which are tilesets with very many different tiles. I wrote a program that generates such tile maps for me, passing the resolution of the resulting tile map and the number of points per side. Here is an example of the map used for this level and with 5 points per side: <img alt="" src="/images/metroidvania_tilemap.png" /> This makes it possible to make level geometry which can have very irregular shapes as you can see in the video (but only at the beginning of the level since I got a little lazy after that). These mono-coloured tile maps are then masked in a shader with a texture. In this case I used a rock texture. In the main program the tile map is rendered black on a white backdrop which as a whole is then blurred. This image is then used as a light-map. Also there are raytraces done from the top side of the map to genereate shadows and a sense of &ldquo;inside and outside&rdquo; (regarding inside the cave and outside the cave). Also the gradient in the background is generated that way. I also used a form of deferred rendering, but in multiple passes since löve doesn&rsquo;t support multiple render targets and multi-texturing. The normal maps and an image consisting of the lights (in the video: player and bullets) are rendered seperately and the resulting lighting is calculated in a final composition-pass. Especially the specular reflections make it look very plastic. I must say, that I really like the look of the game. <h2>2D-Water</h2> I intended the game to start deep down in a cave, probably with the main character being unconscious lying in a little puddle or a little mini-lake. So I got started with programming water! First the video: <center><iframe width="560" height="315" src="https://www.youtube.com/embed/6onBYjnob7w" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></center> Excuse the waves, which seem to be splashing way to high. This has to do with the fact that I didn&rsquo;t implement fixed timesteps in this test program and Fraps having a huge impact on framerate, making the forces way more impactful than at a high framerate. Just a little while ago a new löve-version was released which allows user defined geometry with user defined texture coordinates (but I don&rsquo;t know how much control is given over the way the geometry is stored. Since we want to update it every frame, one needs the possibility to make the VBO of type <code>GL_STREAM_DRAW</code>, but at the time I did this I had to find another way, which then consisted of an array of values, which represent the height of the water surface. In the water-shader these values are interpolated for every x and the water surface is drawn accordingly. The actual geometry drawn is a box that encloses the water and it&rsquo;s waves. This also makes it rather easy to not only make a sharp border, but also give the color a gradient (which I did) and highlight the surface, which improves the realism a lot. Water getting darker when it&rsquo;s deeper is one of the things that a lot of water-implementation that don&rsquo;t seem right lack. Also there is obviously a distortion-texture and the previously rendered scene passed to the shader (so this has do be done in a separate pass) to achieve the distortion effect, which also has a huge impact it looking plausible and is also not hard to implement. The values passed to the shader are determined in the main program using collision-checks of points on the water surface with the game world&rsquo;s geometry. The forces acting on the points are consisting of a force that always puts the point to the zero-level of the water surface (a harmonic force), damping (linear) and a relative harmonic force between the points to couple the oscillators so waves can pass through them. To make the simulation more stable I&rsquo;m using a form of verlet integration and multiple iterations per frame. Also there is a force which is responsible for the box to be able to &ldquo;push&rdquo; water, that is calculated by checking the velocity of every object that collided with the surface and pushing the water up in front of the moving object and pushing it down behind it (in front and behind are obviously determined by the velocity) This water effect alone makes mit a little sad that I decided to stop working on this project.. But there is even more cool stuff, like: <h2>Bone-Animations</h2> As I said before, I wanted to make the pipeline very programmer-friendly. And since drawing is probably my worst skill I just had to have bone animations. It is also something that I wanted on multiple occasions for a very long time. And after searching the net for a freeware 2D bone animations editor with an easy to read format and not finding anything, I decided to roll my own. This is also just semi-finished and not very sophisticated feature-wise. The only thing missing is a proper way to specify interpolation curves for the frames. There is not an easy way to approach this by using only one type of interpolation (linear, smoothed) because different things need very different types of interpolation. So there is an editor missing for that kind of stuff and this is actually the point where I started to stop the project for other stuff I&rsquo;m busy with. Writing this though made me really want to keep on working on it. I have a feeling that a well-working animation editor might not just be something nice I can find use of a few times, but something other people might find useful too. There are some things that make it not very useful for other people though, which is mainly löves policy of not making it possible to write to the file system outside of <code>C:\Users\user\AppData\Roaming\LOVE</code> or <code>%appdata%\LOVE</code>. Maybe I should start a rewrite in C++ or Python or similar and this is certainly too much work to be still interested in continuing. <h3 id="pictures-video">Pictures / Video:</h3> RiggEd, which is for rigging (making bone skeletons and attaching images): <img alt="" src="/images/rigged_guy_1.png" /> <img alt="" src="/images/rigged_guy_2.png" /> <img alt="" src="/images/rigged_plant_1.png" /> <img alt="" src="/images/rigged_plant_2.png" /> AnimEd (has two files to deal with. A <code>.skel</code>-file which is the output of RiggEd and contains a skeleton with images and an <code>.anim</code> files which contains keyframes for the skeleton. The <code>.anim</code> files can be used on multiple skeletons, so the animations can be reused for different characters and different image sets: <img alt="" src="/images/animed_1.png" /> <img alt="" src="/images/animed_2.png" /> <img alt="" src="/images/animed_3.png" /> <img alt="" src="/images/animed_4.png" /> <img alt="" src="/images/animed_5.png" /> BaconGameJam5 & löve https://joelschumacher.de/posts/bacongamejam5-love 2013-07-03T12:30:00+0000 <h3 id="love">löve</h3> About a month ago I participated in the fifth BaconGameJam, which is a game jam that started as &ldquo;RedditGameJam&rdquo;, but had to be renamed due to copyright issues, so it&rsquo;s the former actually unofficial official reddit game jam and now more of a official unofficial reddit game jam. To get to the point I have to mention that I wasn&rsquo;t really sure what technology to use, because using Kiwi is way too stressfull since C++ is not a very productive language for very short amounts of developement time (My opinion at least.), so Pywi (the Python version of Kiwi) was actually my weapon of choice just before the jam started, but I knew that Pywi still had a few problems with sound and that I was not very keen of distributing a 48h-game with a rather huge Python interpreter + Pymedia, Pygame, numpy (we used this in collision detection and a few other parts), so I spontaneously decided to use <a href="https://www.love2d.org/">löve</a>, which is 2D game engine, which is implemented in C++, but used in embedded <a href="http://www.lua.org/">Lua</a>. And I must say, that löve is probably the most awesome framework I&rsquo;ve ever used. I already tried using some other engines rather than my own, but every time I had the impression that there was something about them that bugged me so much, that I didn&rsquo;t really want to start any kind of project with them, but löve is almost exactly like I would like a game engine to be (API-wise). After using it for a while I noticed quite a few things that were not entirely to my liking, but most of these things are going to be changed in the next major release 0.9.0 and the rest is either not very important or something I could fix myself, since löve is open source. One of these more important things I decided to tackel myself was the integration of some proper games-oriented network library (in my case: <a href="http://enet.bespin.org/">ENet</a>). Thankfully a bunch of people already made some significant progress towards integration ENet into löve, namely leafo with <a href="http://leafo.net/lua-enet/">lua-enet</a> and Martin Felis with <a href="https://bitbucket.org/MartinFelis/love-enet">love-enet</a>, which my integration <a href="https://bitbucket.org/schmirsich/love-enet-new/overview">love-enet-new</a> is largely based. The differences are mostly things regarding the &ldquo;how&rdquo; rather than the &ldquo;what&rdquo;. The API is more &ldquo;lövely&rdquo; (commonly used as &ldquo;fitting with the löve-API stylewise) and the integration of the added enet-module is mostly just like and other löve module is integrated in the engine. A working version of this was proposed about two weeks ago in the issue tracker (<a href="https://bitbucket.org/rude/love/issue/645/enet-integration-take-two">link</a>), but was yet not responded to, which is kind of a bummer, but I hope that this will workout somehow and someday ENet will be available in löve officially. <h3 id="bacongamejam">BaconGameJam</h3> Of course there is still game to talk about (BaconGameJam), which was nameless until the very end. This story is equally unimpressive since again, but this time even worse, I didn&rsquo;t really finish in time. I wasn&rsquo;t really enthusiastic about the theme or my idea from the beginning and actually only chose the idea from my list of ideas because there wasn&rsquo;t anything else I even remotely wanted to do. (By the way: The theme was &ldquo;Lights Out&rdquo; and I decided to make a game in which you would wake up and walk around town discovering dead people and devastation, collecting clues and talking to people uncovering that the guy you play actually became a monster last night (when the lights went out), which is then replayed as kind of a &ldquo;reconstructed memory&rdquo; backwards &ldquo;creating&rdquo; the clues you collected and ending at the point the game started as a human, but as a monster.) I don&rsquo;t intend to show you what I managed to finish, but rather summarize what I learned about participating in a game jam. I don&rsquo;t have very much to be proud of then, but I have learned a lot more, which does not feel as good, but is way more valuable. <ol><li>It&rsquo;s not a good idea to make a game with a strong emphasis on story, because.. 48 hours. How am I am I supposed to make a game and invest at least twice as much time in content (levels, dialogues). Instead I should make games which center around a core mechanic or are very easy to make content for.</li><li>I&rsquo;m alone and a programming guy, so I should make games that can be done by a programming guy alone. Needing tile sheets, character sprites etc., which I am not capable of making in a game is just not a good requirement to start with and bound to not work out very well. I should stick to simple sprites, stylised graphics, which are easy to make or boxes/circles/lines.</li><li>Probably the most important: Accepting defeat. If I don&rsquo;t have a good idea or it&rsquo;s probably not very realistic to transform the ideas I have into a enjoyable game in 48 hours I should probably not even start and waste a weekend, feeling bad afterwards. Maybe I should even deign to make the most obvious and easy game to a theme I can think of because in the end I would have an obvious game that no one is exceptionally ecstatic about, which is not so great, rather than having no game, which no one will be at least a bit ecstatic about, which is not even great at all.</li></ol>Yep. That&rsquo;s it. The next Ludum Dare is next month (actually just a little less than two months), so I will see how much these lessons actually taught me.&nbsp; Spacewalk - Part 1 https://joelschumacher.de/posts/spacewalk-part-1 2013-04-07T19:02:00+0000 During my practical course in experimental physics I decided not to use C++ this time, but rather a language which has a way more extensive standard library and is way easier to debug. Just a language.. that is easier, in a sense. Long story short: I chose <a href="http://python.org/">Python</a> and fell in love. I then decided in a few free days to write up a little engine which is very similar to Kiwi (my C++ engine) now called Pywi. A buddy of mine and me then decided to put this engine to test, so we can maybe use it for the upcoming <a href="http://www.ludumdare.com/">Ludum Dare</a>. At first we struggled a bit with finding an idea we both would like to implement, but after looking at one of my idea-lists, we decided to 2D-ify one, named &ldquo;Spacewalk&rdquo;. It essentially explores multiplayer deathmath and capture the flag in space, but with level geometry consisting of materials with a very high mass. We actually had a few videos which were a little more impressive and capturing the mind-bending-fast-paced-style we would like to achieve in the end more appropriately (just more appropriately but in fact not at all), but somehow Fraps broke these files and neither WMP, nor VLC and MPlayer could play them back. We made these videos almost one and a half weeks ago and were not able to work on this game together since, so it&rsquo;s just this! <center><iframe width="560" height="315" src="https://www.youtube.com/embed/_4EkPm3LLYg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></center> <h3 id="polygons-and-gravity">Polygons and Gravity</h3> Technically we first weren&rsquo;t sure if the level should consist of predefined elements, a huge image that&rsquo;s used as a gravity-source or something polygony-vectory. Inspired by <a href="http://soldat.pl">Soldat</a> we decided to use polygons. We used Inkscape to draw these curves and export them to svg, which is then parsed, triangulated and rasterized. The rastered image will be interpreted as a cloud of point masses (every pixel drawn) and the resulting gravitational potential is stored in an numpy-array and saved (an image essentially). In-game we take the numerical pseudo-gradient <code>grad V = (dV/dx, dV/dy) with dV/dx = (V(x + h) - V(x - h)) / 2h</code> and <code>h</code> being one pixel and use this as the gravitational force. Also we calculate <code>1/(r + r_0)</code> instead of <code>1/r</code> to make the potential less steep, so we can choose a fairly high gravitational constant (because <code>1/r^2</code> decays quite fast) without the player sticking to the level geometry too hard. <h3 id="net">Net</h3> This was <a href="http://jonas-lieb.de/">Jonas</a>&lsquo; (said buddy) first and my semi-first venture into the realms of networked programming. Both of us already used some high level network classes (e.g. Qt) and used BSD-Sockets a few times. I actually worked on a LAN-multiplayer-game recently, but didn&rsquo;t get very far (moving and shooting and infrastucture without any optimizations or compensations). Just to start off we decided not to use ANYTHING remotely hard or an own reliabilty layer (to any extent whatsoever) (this has to be done some time though), client side prediction, lag compensation or proper interpolation and not surprisingly and very much to our liking this is more than appropriate over LAN. Very rarely though essential initilization packets don&rsquo;t reach the client (or the server respectively) (for example handshake packets or map changes) so we somehow have to implement a reliable layer on our own or switch to ENet, which I used in mentioned former networked game. Apart from this we tried to reduce the rather jaggy movement by exponential interpolation (I&rsquo;m not sure if it&rsquo;s called like this - <code>position += c * (target - position)</code> with <code>c in [0,1]</code>). <h3 id="collision">Collision</h3> Fortunately we had the triangulated polygons at hand and just had to implement Seperating Axis Theorem. There was only one thing we really struggled with for quite a long time, which is minimal translation vectors that, for reasons we didn&rsquo;t really understand, pointed in the wrong direction. We decided to just flip them if they pointed along the vector between the two polygons center of masses (cosine of inner angle &lt; 90° &lt;=&gt; dot product &gt; 0) Sorry for not really telling you interesting stuff, but this should be more of a documentary, rather than lessons. Feedback Effect https://joelschumacher.de/posts/feedback-effect 2013-01-29T12:00:00+0000 I played around with an old demoscene effect often called &ldquo;feedback&rdquo;. It probably happend to you, that you forgot the glClear (or equivalent clear screen commands) at the beginning of your renderloop which lead you to very funny results. This effect is essentially this but kicked up a notch. A video first: <center><iframe width="560" height="315" src="https://www.youtube.com/embed/AiWi9A39LxU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></center> The code is a mess, so I just present some pseudo-code: <ul> <li>Set render target to texture "fboTex"</li> <li>Add a translation to the transforms (I used my own "continousRandom", which returns an interpolated value from a buffer of random values and polar coordinates (an angle and a magnitude), so the translation changes, but randomly and continuous)</li> <li>Add a scaling to the transforms with a little more than screen size (in my example 1.011 times)</li> <li>Set color (you have to keep in mind, that having 0.5 as the red channel for example results in having half of the red in the next frame, and fourth in the next, and eighth in the next and so fort, so the values are very close to one and very hard to tweak) (I used: 0.999, 0.995, 0.1, 1.0)</li> <li>Render a quad with fboTex (the last frame)</li> <li>Just render the image you want to effectify (In my case an image which said "screensaver" using a cool font)</li> <li>Render fboTex to screen</li> </ul> Global Game Jam - Cardiac Murmur https://joelschumacher.de/posts/global-game-jam-cardiac-murmur 2013-01-28T23:04:00+0000 Last weekend I was attending the global game jam in cologne and I have to say that it was a really cool experience! Sadly my team mostly consisted of people from the design school this event took place in (they didn&rsquo;t study game design though) and they had to take part in this to receive certain credit points or something (in essence: My team was 60% people that have never made a game, don&rsquo;t play games and have no skills whatsoever to actively partake in making one) and we weren&rsquo;t lucky to incorporate an artist into our team, so we essentially ended up with two programmers. But otherwise it was still very cool! I was overwhelmed by all these productive talented people and all the awesome ideas. The theme of the game jam was the sound of a heartbeat (the actual sound. Not the description I just gave) and after a bit of brainstorming we decided to make a game where you had to listen to your heart. A kind of parallel-world-gameplay where you had to switch you view to see what your heart was telling you. We wanted to partake in the &ldquo;Games for Change&rdquo;-challenge and make a game that raises awareness of us often overlooking problems of the people surrounding us (not just by being ignorant, but because they try to hide). At first we wanted to focus on characters and their developement and stories. People you are friends with that secretly hide depression or family problems, but we ended up with not enough time and just made a cool platforming-level and one level that merely touched what we had planned. Still I am very content with what we did. I look at it as a prototype of a game that we might make some time. Link: <a href="https://web.archive.org/web/20130522133503/http://globalgamejam.org/2013/cardiac-murmur">Global Game Jam - Project Site - Cardiac Murmur</a> (Update 7th of April 2015: this link seems to be broken. I added Screenshots and a Download-Link at the end of this page) Technical stuff: The paper-like look is achieved by a multiplied gradient from top-left to bottom-right and a &ldquo;stencil-sketch&rdquo; texture multiplied, which essentially is a noise-texture that is stretched in one direction to have little lines. The normal world is encoded in the red-channel of the sprites and the heart-mode-version is encoded in the green channel. In a shader these channels are linearly interpolated with a game-defined value and converted to greyscale. We made the maps with <a href="http://www.mapeditor.org/">Tiled</a>, which is a great tool and I really recommend you using it (I certainly will use it again) and a little python script to convert them map files to a binary format and splice the tilemaps. Update 7th of April 2015: <a href="https://www.dropbox.com/s/4gsz7hi6dk1q4ro/CardiacMurmur.zip?dl=0">Download link</a> <a href="https://youtu.be/M3AxFZOEyYc">Youtube link</a> <img alt="Cardiac Murmur Light" src="/images/cardiac_murmur_screenshot_light.png" /> <img alt="Cardiac Murmur Dark" src="/images/cardiac_murmur_screenshot_dark.png" /> Species X https://joelschumacher.de/posts/species-x 2012-09-01T19:52:00+0000 I tried to make a game for ludum dare #24. The theme was evolution and I decided to make a game about some sort of species that evolves and you would have to fight. Because I was in a hurry and I&rsquo;m not very good at such things I ended up with this game. It features a cellular automaton resembling a fungus that spreads through the level and sort of &ldquo;zombifies&rdquo; people and hopefully other living beings (not yet implemented. Neither in my entry nor in the current version), making them slaves to spread the infection further. The goal would be destroying the fungus and surviving the attacks of the evil fungus-slaves surrounding it. Unfortunately the game around this idea emerged to a be a top-down shooter and all in all just way to complex to choose for a 48 hour competition. In short: The game I entered was extraordinarily bad. Really, really bad. I recommend trying it though for comparison: <a href="http://www.ludumdare.com/compo/ludum-dare-24/?action=preview&amp;uid=14778">http://www.ludumdare.com/compo/ludum-dare-24/?action=preview&amp;uid=14778</a> (Screenshot and download link inside) I was so unsatisfied and unhappy with my entry that I decided to see it as my honour to make this game an enjoyable experience. It proved to not be that trivial on numerous occasions but I think I made quite a bit of progress up until now. You can find it here: <a href="https://dl.dropbox.com/u/9961608/SpeciesX_new_and_improved_alpha.zip">https://dl.dropbox.com/u/9961608/SpeciesX_new_and_improved_alpha.zip</a> Screenshot: <img alt="Species X Screenshot" src="/images/species_x_screenshot.png" /> (Note: Space triggers a flame-thrower-effect that has no impact on the game world but looks nice and I&rsquo;m thinking about adding a flame-thrower as a new weapon (or 30-seconds-ish super power up)) At first you might notice, that the first version was WAY harder. I am pretty sure that it&rsquo;s safe to say that it&rsquo;s impossible to complete a level. Mainly because the enemies would spawn everywhere constantly. So right after sleeping and waking up the next morning I only allowed enemies to spawn if they would spawn on a tile that&rsquo;s currently visible and adjusted the rate of enemies spawning, which changed the experience a lot. You feel way less overwhelmed and have the feeling of having more overview what&rsquo;s actually happening in the level. I then changed the controls so you would throw grenades with G instead of changing the weapon, throwing and changing back which appeared to be very inconvenient mid-fight. After adding film grain and various particle effects (blood, goo (when shooting the fungus roots) and little sparkles (spores, I&rsquo;d say) over the fungus and a cheap fire during a grenade explosion) the next big problem would have been the absolute lack of fun when destroying the fungus. There had to be some sort of satisfaction from destroying it. Until then killing the fungus only meant having to destroy some other fungus somewhere else now. There was no reward. So I decided to change the cellular automaton drastically. I decided that I want to have the regular fungus connected to it&rsquo;s roots. Turning the spot light on the roots, because they deserve it! In the end this connection was implemented in the way that the root fungi dispatched waves through the fungus matter, which get carried on by the regular fungus. Additionally the fungi would die if they weren&rsquo;t &ldquo;freshened up&rdquo; by a wave recently. Also the fungi would now only be able to spread if they are just freshened up. This made the fungus seem way more like something organic and added to the atmosphere drastically. Still there seems to be a problem with the game being horribly repetitive and just not very interesting. Maybe I have to blow up this game idea with actual progress (different levels, gradually introducing weapons, doors, keys you have to find, generators you have to turn on, stuff like this) to make it interesting, but maybe this is equivalent to saying that this might be a good foundation for top-down 2D horror-shooter. Let me know what you think and how I could improve on this! Edit: This is all programmer art! If anyone is interested in replacing some of the images with better ones, I would really appreciate your help! First post! https://joelschumacher.de/posts/first-post 2012-09-01T18:24:00+0000 Hello! I&rsquo;m Joel. I&rsquo;m currently studying physics in Aachen (Germany) and I intend to use this blog as a journal of the games I make in my sparetime (since the next semester starts soon it will probably take quite a while until I post again).