May 2023 #WritingWonders

Recently, I stumbled across the Mastodon hashtag #WritingWonders, a series of short questions by Alina Leonova, Branwen OShea, and Amelia Kayne. It’s an excuse for writers to share fun facts about your story, and see how other writers tackled the same topics.

31 writing prompts for May 2023

On a whim, I decided to join in for the month of May, answering questions about the story and backstory of Run 3. Here are all 31 of my answers in one place.

Note: the questions often refer to a main character (“MC”), side character, and antagonist. Run 3 doesn’t have clearly-defined roles like that, but for this month I decided to treat the Runner as the main character, the Skater as the side character, and the Angel as the antagonist.

May 1: Intro Day. Share 3 fun facts about your favorite side character.

Original post

Fact about the present: of the main cast, the Skater is the one who most enjoys the game-like nature of the Tunnels. The moment-to-moment experience of timing his jumps, sticking the perfect landing, and replaying the same level over and over to find ever more optimal routes. In other words, he’s the one most invested in the core game loop. For comparison, the Runner is more into the meta loop of exploring and completing her map, and the Angel only cares about the meta meta loop: getting out of this place.

Fact about the past: the Skater will tell you that he’s awful at STEM, though in reality, his grades were just mediocre. He just has bad memories of the class laughing when he got an answer wrong, overriding the facts of the matter.

Fact about the future: the Tunnels are merely the start. The Skater intends to become a Planet-famous gymnast, capable of turning any obstacle course into a spectacle of color and motion.

May 2: What is your favorite side character’s relationship to the MC? Do they get along?

Original post

(Clarification: I don’t actually have favorite characters, the Skater is just one who’s had a side role so far.)

The Skater and Runner get along just fine, thanks for asking. I guess they got off to a bit of a rocky start when the Skater bowled the Runner over and made her drop her map and stayed only long enough to help retrieve it before skating off. He was just very excited about this new environment. And he did stay to help!

Later on (not shown yet), they start a friendly game where each tries to “prove” the superiority of their means of locomotion, by finding a level they can cross that the other can’t. Basically it’s an excuse to hang out and practice obstacles.

From the Runner’s perspective, the Skater is one of the least stressful characters to be around. Everyone else asks tough questions, or is abrasive, or otherwise requires mental effort. The Skater only requires physical effort (usually).

May 3: What is the criminal justice system like in your world?

Original posts

Disorganized.

If you’re accused of a crime, the outcome depends largely on where (and whether) it goes to court. Some courts have a reputation for favoring businesses, others are said to be harsh to everyone, etc.

Punishments vary wildly depending on the whim of the judge. You might pay a fine, lose your job, lose your name, get banned from a city, or get something akin to an RFID implant warning people about what you did. Imprisonment is only considered for repeat offenders.

Appeals go straight to the High Court, which almost always rejects them. For those lucky(?) accepted appeals, the Lawyer makes a point to air both sides’ dirty laundry for the world to see.

If you don’t trust the courts, you can hire a mercenary (often a ninja) instead. Careful: most ninjas do their own investigation before enacting their subtle and barely-legal vigilante justice. If they decide you’re wrong, good luck tracking them down for a refund!


The character limit kept me from going into detail on some things, like what I meant by “dirty laundry.” @Ninpan commented that “The lawyer sounds like a bit of a twat,” so I figured I’d give the Lawyer a chance to explain himself.

As a representative of the High Court, I have sworn to uphold not merely the letter of the law, but the concept of justice itself. This requires the utmost transparency.

I ask you, could a jury deliver justice with only some of the facts? Could you negotiate a fair settlement if either side kept secrets?

No, you most certainly could not. Thus, all facts relevant to the case must be brought to light, including details about the character of those involved.

And yes, concluded cases must be disclosed to the public, to help them fully understand the jury’s decision. Justice requires vindication; if the public were allowed to believe the High Court had made a mistake, the case’s winner would be denied vindication.

May 4: MC POV: What is your occupation? Do you like your job? Why/why not?

Original post

I liked my job.

Don’t get me wrong, I still enjoy cartography, but I can’t enjoy doing it as a job. Not since everyone decided to put me on a pedestal.

At last count, four people hurt themselves by trusting my maps too much. Including my own brother! I’ve added five errors to school curricula that I know of. And I upset the Pastafarian Church that one time, but we worked that out.

I love the process of map-making. The challenge of exploring and documenting things. Distilling places and concepts down into the simplest useful representation. Figuring out which details I can omit, and which ones people need to recognize the shape of the map.

I love using my own maps. The pure satisfaction of seeing how simple tasks become once I have maps for them.

I do like making other people’s lives easier. If only that didn’t come with so much risk…

(The Runner abdicated her job years ago.)

May 5: What scents and sounds can be found in your MC’s workplace?

Original posts

In her very first workplace, as few sounds as possible.

Not that you can ever truly achieve silence. You can pad the walls and turn off the music, but there’s still the ticking of a clock, the hum of a ventilation fan, the scratching of a stylus, the shrrpt of a paper cutter, and so on.

There are also smells of paints and dyes, new and old books, the faintest waft of fresh air from the ventilation, and if you’re lucky, the aroma of fresh baked goods.

Near one door, you might make out the hum of a heating element or the bubbling of a chemical solution. From the crack in the door might waft the scent of acids and bases, experimental polymers, rusted metal, and more.

Sometimes, there would be the squeak of that door opening, followed by a quiet conversation lasting minutes or hours. Or rarely, a not-so-quiet argument, about topics such as if it’s appropriate to disassemble your father’s clock to make the ticking stop.

Following one such argument, the building was very nearly silent for a time. No notes being written on paper, no chemical solutions boiling, no ticking, no fan. Just one set of footfalls on soft carpet.

This silence was broken briefly, first by a girl’s voice, then by a boy’s.

“Well maybe I’ll just find somewhere else to live.”

“Could you, please? That’d benefit us both.”

This may have been followed by a grunt of frustration from the girl (having failed to get the response she wanted), but no one can prove such a thing happened.

After another, shorter, period of silence, sounds of packing began. Shifting boxes, zippers opening and closing, stacks of items falling over, and the occasional exclamation of delight at the discovery of a lost possession. Several hours later, a faint buzz and the smell of ozone trailed a vehicle leaving the premises, and silence reigned once more.

The Cartographer’s second workplace featured instrumental music and the smell of the sea.

Credit where credit is due: some of my descriptions were heavily inspired by Betsy Lee’s Calamity Observes: The Silence.

May 5 fan art

[The Runner stares aghast at the Chemist, who appears to have taken apart a clock] / Runner: Well, maybe I'll just find somewhere else to live. / Chemist: Could you, please? That'd benefit us both. / [A picture of some clouds] / [A boat is docked behind the Runner, who stands in front of a doorway] / Sailor: Sis? / Runner: Yeah, I need a place to stay... for a while / Sailor: oh

Original post by imanoutlawontherun includes the full-size image

Disclaimer: fan art always involves the artist’s personal interpretations. I featured it here (with permission) because I like it, not to confirm or deny that these interpretations are canon.

May 6: If your opening scene had a theme song, what would it be?

Original post

Since my story is a video game, it already has a soundtrack! Here’s what plays in the first level of each game.

May 7: What does your MC look like? Share art, pics (face claims), and/or mood boards.

Original post

The Runner, a round gray alien with a single green eye, examining a map.
The Runner in her "Skier" costume, wearing red skis and a red hat with a large white pompom, carrying two ski poles. Partial art credit: Hasan Odom.The Runner in her "Jack-O-Lantern" costume, appearing to be a large pumpkin carved to resemble an angry eye and toothed mouth. Art credit: James "Jimp" Pearmain.The Runner wearing a bright orange wide-brim hat with a vine wrapped around it. The vine has a blue flower at the top and appears to pass through the hat's brim at the bottom. A rolled-up map is glued or otherwise attached to the brim.

The lines in this drawing are shakier than in the others, because it was drawn quickly for a stream and never got polished to the level of in-game artwork. Nevertheless, the hat is popular among fans.

May 8: Describe your MC’s laugh.

Original post

The Runner is pretty restrained, so it’s rare to hear more than a sensible chuckle from her.

There are a small number of things that can make her totally break down laughing, and it’s never the sort of thing that you (or she) would expect. When it happens, it comes out as a kind of wheezing guffaw, generally accompanied by a few gasped-out words, an attempt to get herself under control, and then at least one relapse.

She finds this totally embarrassing, and tries not to let it happen.

May 9: MC POV: When was the last time you laughed? Why?

Original post

The other day the Student and I spent a while hanging out, and she asked if I thought humans could really exist.

If you haven’t heard of humans, they come from these old pop sci-fi stories, which describe them as tall, hairy, reddish-brown aliens. Usually with a bunch more strange traits and abilities.

So I said yeah, almost anything could exist, as long as you leave out the blatantly supernatural stuff. Super strength and speed could come from advanced biology. Future vision and water breathing, probably not.

When I mentioned these examples, the Student immediately pulled up a story on her e-reader. Apparently, not only does the protagonist use some kind of precognition for danger, not only do they dive into a lake for protection, they claim their body contains over 50% water. So the two of us shared a laugh at the absurdity of the writing.

…Hey, if you specifically wanted a joke, you should have said so.

May 10: How much humor is there in your story? Share a funny snippet if you want!

Original posts

I never focus on writing jokes. Sometimes I have a silly idea and decide to write it in, but rarely as just a joke. There always has to be some character development or plot progression or something, so that if other people don’t find it as amusing as I do, they can still enjoy the story.

Beyond that, I think this ties back to the lack of confidence I mentioned in my earlier writing.

For instance, early in the story, the Duplicator accuses the Runner of stealing a planet. I think this is a funny concept, but instead of exploring it, I had the Runner shut the idea down.

Maybe it’s still a funny scene, but less so than I think it could be. Also a bit out of character for the Runner. Nowadays she never passes up a teaching opportunity.


As an experiment, I want to try rewriting the scene.

The Duplicator Hey! Runner!

The Duplicator Bad news: our planet is missing! It’s gone, like *poof*!

The Runner Oh, uh, yeah.

The Duplicator Just “yeah”? [narrows eye] Did you already know about this?

The Runner It left us behind the moment we landed. You didn’t watch it go?

The Duplicator I watched- uh, never mind that.

The Duplicator What do you mean it left us behind? Planets don’t just leave!

The Runner Hear me out. [pulls out map]

The Runner The Planet orbits quite a bit faster than these tunnels, putting it somewhere ahead of us, in this range. [drags the image of the Planet back and forth]

The Duplicator Hmm, I see. Wait, how are you moving the Planet?

The Duplicator [gasps dramatically] It was you! You stole our planet using that map!

The Runner Beg pardon?

The Duplicator How could you, you fiend! My wife lives there!

The Runner Hah, I wish maps could do that.

The Runner …No I don’t.


Is it funny now? I don’t know, but I do prefer it to the old version:

Duplicator: The Planet is gone!

The Runner So? It’ll come back.

The Duplicator How do you know? *gasp* It was you! It was you! You stole the Planet! That’s how you know!

The Runner

The Runner I think I’ll ignore that entirely. Now help me map out this next area.

[time passes]

The Duplicator So… Where’d you put the Planet? I promise not to tell.

The Runner You still think I “stole” it? Think. Why else might a planet move?

The Runner I’m going to find another tunnel to map out. Catch up once you realize the Planet orbits faster than us.

I know brevity is the soul of wit, but I think that version did a disservice to both characters by not giving either of them a chance to explain themselves.

May 11: Does your MC laugh or cry more in the story?

Original post

Laugh.

I’ve talked before about how I designed the game first and wrote the story around it. And the game is a straightforward (pun intended) platforming challenge. One of the most common questions before I started writing the story was “what are the character running from?” Since I was feeling contrary, I decided the answer was nothing: they were running to something, and weren’t in danger.

This led to a story that… well, it isn’t lighthearted exactly, but it certainly isn’t dark. It’s just a story about a group of aliens exploring a very strange location. No time to be sad*, there are mysteries to solve!

*Not true, there’s almost nothing but time. And while some of the others might be going through some stuff, the Runner is enjoying the break from the hustle and bustle.

May 12: Do you think your story will make your readers cry at some point?

Original posts

There are sad scenes that I could write, yes. Some characters have sad/difficult backstories, for instance. The question is, will the story ever call for exploring those events? I intentionally haven’t decided yet.

I’ve found I enjoy writing most when I’m not 100% sure where it’ll end up. Which might be why it’s so hard for me to finish the plot: I already planned out events that will happen, so I find those scenes less fun to work on.

Besides that, I inadvertently ended up with a lot of children in my audience, and I have to keep them in mind. I’m still going to write what I want, but I also have to be careful how I present that writing.

As an example, if part of the story featured animal abuse, that might not make it into the main game. I might instead release it as a spinoff story with appropriate warnings.

May 13: Share a description of a secondary character.

Original post

Run 3 isn’t exactly a written work, but if it was, this would be the Skater’s first chronological appearance:

The inside of the square blue tunnel was silent by the time the Runner touched down. It appeared everyone had already left.

No, check that. An adolescent sat alone in the corner, his arms full of broken red and white items, and a look of intense concentration on his face. His athletic build and sun-bleached skin suggested a great deal of time spent outdoors.

The boy grumbled in frustration as he attempted to shove a dented wheel into what looked like the bottom of a boot.

“I can hold something for you,” the Runner offered.

“I saa, I gah hiss,” he retorted, trying not to drop the additional pieces held in his mouth. He tossed the wheel into the air, flipped the boot over for a better angle, then deftly re-caught it.

The Runner decided to leave him to it for the moment.

The Skater is next seen doing this:

The Skater bowls the Runner over at the end of Level 10. The Runner hangs in the air, eye closed tightly, as she and her map fall in different directions. The Skater looks towards the Runner with an expression of concern or regret.

But his skates are fixed, at least!

May 14: MC POV: Have you ever broken anyone’s heart?

Original posts

Oh absolutely. Celebrities get lots of attention, but we can’t possibly date everyone who asks, much less commit to long-term relationships. I’ve lost count of the number of people I’ve had to turn down.

You’d think by now I’d know a good strategy to turn someone down, but nope. Everyone takes it differently. If you give a reason, some people take it as a challenge: fix this thing and try again. If you give no reason, some people take that as an invitation to keep trying.

Stepping back from the public eye helped a lot, but it still happens. I mean, consider this. Of the eight people here, three either asked me out or mentioned having feelings for me.

Oh don’t worry, I got used to it long ago. Feelings happen.

For context: [aromantic flag]

The Runner is too polite to name names, so I guess I’ll just have to post this here instead.

Angel: [bisexual flag]​/[pansexual flag]​ (asked out, mentioned feelings)
Gentleman: [gay men flag] (asked out, likely for political reasons)
Student: [lesbian flag] (mentioned feelings)

May 15: Is your MC good at romantic relationships?

Original post

No.

May 16: Is your MC good at relationships in general?

Original post

She’s learning!

She spent a large portion of her life as a celebrity, and it takes time to relearn how to talk to people as equals. Fortunately, the Sailor has plenty of good advice on the topic. Unfortunately, theory is not practice.

I guess it could be telling that she’s in no rush to head back home. If she’d made close friends back home, wouldn’t her priority be to get back to them?

(The answer is no. Friends can wait; there’s advanced technology to discover and a galaxy to explore!)

But it’s also true that she’s made closer friendships out here than she had back home. This is partially a consequence of being stuck with only a few people to talk to, and partially thanks to having more practice.

May 17: Secondary character POV: do you trust the MC? Why / why not?

Original posts

The Skater Why wouldn’t I trust the Runner? She’s cool.

The Skater Stole the- Are you kidding?! She didn’t steal the Planet! That’s the stupidest thing I’ve ever heard, and I say some pretty stupid things, let me tell you.

The Skater Nope, she never tried to take my stuff. Your son did, though. Yes, I can tell.

The Skater …Are we done here? Can I be excused?

The Skater What’s wrong with going where she says? She has the map, I like exploring.

The Skater Why would she set a trap? How would she set a trap?

The Skater Hah! I could get out of that easy. That’s barely even a trap.

The Skater Yeah well, I guess I’m just awesome then.

The Skater Enough, ok? She’s not like that, and I got stuff to do. Bye!

The Skater No I don’t know who she was talking to. Maybe she was thinking out loud. Who even cares?

The Skater Yeah no way, that’s not her style. I’m leaving now, ok? Ok.

The Skater …Do you mind?

The Skater …Uh huh.

The Skater …Oh. I guess?

The Skater Fine, jeez, I’ll ask her about the Government! Bye.

Astute readers will have noticed that the Skater didn’t exactly go into detail. Part of that was the fact that his unnamed conversation partner was being antagonistic, but there’s at least one reason that he’s just not going to admit out loud.

A big reason he trusts the Runner so much is he used to smuggle her maps into exams to cheat. They always provided good answers in an easy-to-understand format, so he came to rely on them. And, dare I say it, trust them. So by extension, he trusts the person behind the maps.

May 18: Do many people find your MC attractive?

Original post

Yep. Especially back in the day, when she was on top of her game as the Cartographer. She had poise, confidence, energy, and a rotation of fancy outfits. This all fueled the hype surrounding her, and the hype in turn drove her to refine her performances. (Like how some real-world video essayists turn each essay into a small theatrical performance, complete with costumes and sets.)

She had a lot of fans, and when you have a lot of fans, some of them are likely to be *~admirers~*.

In her mind, it was all meant to educate and inspire people. Showmanship helped emphasize important points. Printing maps/diagrams on her outfits could pique people’s curiosity in new subjects (at the expense of temporarily staining her skin underneath). Maintaining a large following meant there would be a whole community willing to put effort into learning from her.

May 19: If they knew who you were, would your MC save your life?

Original post

Of course.

But she’d have some choice words for me afterwards, and she’d demand concessions in exchange. She even has a way to hold me to it: she’s keeping secrets that would mess up the plot if revealed.

Fortunately for the plot, this scenario seems unlikely to happen.

May 20: Do illicit substances play any role in your story?

Original post

Illicit substances? No. Characters eating substances not meant for consumption because the warning label was printed in an alien language? Yes.

But it’s fine, he received medical attention and made a full recovery. (Where “full recovery” means “no more deluded than he was before.”)

May 21: MC POV: If you had to give your younger self a piece of advice, what would it be?

Original posts

Hmm. Only one piece of advice?

Well, I could say “find a better work-life balance.” That would’ve saved past me a fair bit of grief.

But you know, I pulled through, and even learned from it. If I only get to send one message, do I really want to spend it on undoing that?

Instead, what if I sent advance warning of a disaster? “On Calming 35, double-check the coupling between cars 26 and 27 of the City Circuit Train.” That single tip would have saved hundreds of person-hours of cleanup.

No, even better: “Tell the Factory workers to scan for ozone pockets before rerouting any rivers.” Or I could sound the alarm about the raid that destroyed the First City.

Sorry, I have a habit of turning personal questions into optimization problems. Given actual time travel, I’d try to save the First City, but if that doesn’t fit the spirit of the game, then I’ll go with avoiding burnout.

This was the longest I’ve spent on any #WritingWonders question, specifically because the Runner needed to mention a date, and I had never spent the time to nail down a calendar system.

Several hours later, I finally have one, complete with number of days/weeks/months in a year and day/month names. The first part was tricky because math, the second was tricky because names.

Based on my already-established naming conventions, each day/month name must be a descriptive noun. With that explained, here are the months:

  1. Northrise
  2. March
  3. Weathering
  4. Southrise
  5. Trek
  6. Calming
  7. Foundation

(Fingers crossed that I did enough research, and I didn’t get Calming and Weathering backwards…)

May 22: Do you prefer writing the first or last chapters?

Original post

I was going to say “first,” but then I realized the answer is probably “neither.”

Most of my plots start in medias res, skipping the scene-setting and exposition. Also, zero out of three major plotlines have an ending written yet.

In other words: what happened before a scene is a secret, and what happens afterwards is also a secret. Or sometimes it’s a puzzle rather than a secret, something you can figure out if you pay attention to the right details. I enjoy writing bite-size mysteries; not much else to it.

May 23: If you switched places with the MC, would you survive the story?

Original post

Depends if I could get a farm going before running out of rations my body is capable of digesting. I’ve previously stated that such rations exist, but I don’t know how many there would realistically be.

Thing is, I barely know anything about farming, and I know even less about hydroponics. There might be a guide somewhere, if I could find it among all the odds and ends. Even then, I’d say my chances are slim.

This despite everyone else being (more or less) friendly towards me. Friendly and not in any danger themselves, but they’d have to watch the first alien they ever met slowly dehydrate and/or starve. (“Care and feeding of alien beings” was sadly not part of their school curriculum.)

Wait a minute, I know all the spoilers. There is a way I could cheat the system and almost assuredly survive, though it wouldn’t be particularly fun.

May 24: Secondary character POV: What was the best thing the MC has done for you?

Original post

She helped me realize it’s ok not to be good at school, you can still be good at other things.

Everything got so much better when I realized I didn’t have to spend the rest of my life sitting still and crunching numbers and writing essays.

May 25: Does your antagonist have a favorite food or drink?

Original post

Yes he does, but beyond that I think I have to pass on this one, sorry. I haven’t done enough research and I haven’t nailed down enough details about their metabolism.

Like, I could say he’s a fan of spicy foods, but I couldn’t tell you what chemicals or substances make food taste spicy to them. (Because it isn’t necessarily capsaicin.)

May 26: Does the media or public opinion play a role in your story?

Original posts

Not at the moment. At least, not directly. The cast is on their own, far away from the media and the public.

That said, several of the characters are driven by hopes of becoming popular, changing public opinion, or righting wrongs. (Almost typed “writing wrongs.” No, that’s my job.) So though society is far away, it indirectly plays a role.

Here are the characters most concerned with the media and/or public opinion:

The Runner’s biggest question in this story is “what lies beyond the Wormhole?” Her second-biggest question is “should the public find out about it?” After all, someone went to some lengths to keep it secret.

The Skater wants to be a famous athlete. He sees this as a chance to catch up to the competition, by practicing on obstacle courses they don’t have access to yet.

The Duplicator wants to shine a light on the Government’s wrongdoings. He sees this as the biggest Government cover-up ever.

The Gentleman is almost completely driven by the idea that success is the best revenge. He sees this as a unique opportunity to accrue wealth, influence, and incredible technology.

May 27: What genre of music best fits your current WIP?

Original post

The game’s soundtrack, written about a decade ago, is full of high-intensity techno*. It fits the action gameplay quite nicely, but the story has started to go a different direction. Nowadays I write more about learning, searching, interpersonal conflicts, secret-keeping, self-doubt… All kinds of things besides action.

*Or something electronic, at least. I can’t always tell the difference between these genres.

If I were to expand the soundtrack, I’d still want electronic music for consistency, but I’d want a wider range of moods.

May 28: Secondary character POV: What was the worst thing the MC has done to you?

Original post

Trying to parent us. Like, what’s even the point of running away from home if all the adults are gonna keep telling us to eat healthy, go to sleep, and play nice?

What if I DO want to be like the kids in the stories? What then? Besides, fairy tales aren’t scientific. There’s no such thing as “turning into strong wild garbage” or “going to the Never Land where you can never grow up.” They made it all up to scare you, and I’m not fooled.

…So yeah, that’s about the worst thing I can think of to say about the Runner. That she and the other adults look out for me.

The Runner isn’t even the one who brings up the stories. She brings up actual scientific reasons to do things. Stuff like the Circadian Cycle. I just wish she wouldn’t try so hard to explain it to me. She should know I’m just going to zone out!

(I had to toss in a Clickhole reference, sorry.)

May 29: What type of government exists in your story?

Original posts

The Government is an institution that records, interprets, and enforces the definitions of words. These codified definitions are known as “laws.” The Government’s various branches each have an official definition of what they are and what they do.

For instance, the Courts are defined as enacting justice, and since that’s in its definition, that’s its mandate. Additionally, the definition of “court” spells out how a court case is handled, mentioning the judge, jury, procedures, etc. And each of those things have their own definitions, and so on.

Similarly, the definition of each public office includes a mechanism for electing or appointing officials, and the definition of “law” includes mechanisms for creating and updating laws.

Since the Government is defined as enforcing definitions, it does so. The end result is something like a representative democracy, except with a bunch of extra steps.

Law enforcement is defined as keeping track of the official definitions of words and resolving definitional disputes. Law enforcement, naturally, tends to attract people passionate about language and linguistics. This passion gives them a reputation for being willing to fight you (and each other (especially each other)) over definitions. Academics, am I right?

Additionally, they issue regular reports on the changing use of language. The Government uses these reports to help determine if and when legal definitions are out of sync with common usage, and therefore ought to be updated.

Note that most crimes count as “definitional disputes.” For instance, if you own a piece of property, that item’s official name reflects the fact that it belongs to you. Thus, theft counts as disputing the item’s definition, so by default law enforcement will give it back to you, unless the thief can disprove the definition.

Oh, speaking of which, names are definitions too. That’s why the Government is so picky about names, and why you have to prove to them that your name accurately describes you. Your name is effectively a small law unto itself, and they take that seriously.

Fortunately, it (supposedly) isn’t that hard to change names. Just complete the name change form and the subsequent verification/renaming process. Unfortunately for anyone in a hurry, it isn’t just a name change form. A better term might be identity change form, since it’s full of questions about gender, profession, identifying item(s), and more. All of this data then becomes the Government’s official definition of you, and people you meet will be able to intuit that it’s accurate.

(For those keeping count, stealing someone’s identifying item is two crimes. One for violating the definition of the item as belonging to the person, and another for violating the definition of the person as possessing the item.)

May 30: What colors do you associate with your WIP? Why?

Original post

Mostly, none. The story frequently shifts between locations, and each location has a different color.

I do associate colors with characters, though. Brown for the Runner’s map, red for the Student’s backpack, green for the Child’s balloon, yellow for the Angel’s halo ring, etc.

I do associate red with the plot thread involving the Student, because the location is red and so is her backpack.

The Student stands in a red-and-gray area, looking confused.

May 31: Does anyone that your antagonist trusts try to rein them in?

Original post

Does anyone try to rein the Angel in? Absolutely.

Are there people the Angel trusts? Certainly.

Does anyone the Angel trusts try to rein him in? Rarely, if ever.

His friends all egg each other on, challenging each other to accomplish ever-greater pranks and feats of engineering (in the vein of MIT “hacks”).

Meanwhile his mentor is very busy, and would only step in in extreme cases. I guess the Angel reins himself in to avoid his mentor’s ire, but that’s mostly “stay on the media’s good side and don’t break anything important,” not “treat people with kindness and respect.”

Appreciation post: Space Station Weird

Lately, I’ve been thinking: I find a lot of cool media online, and almost never share or promote it even though I have a platform. So let’s fix that!

Each post in this series will start with a quick summary to convince you to go look at the thing for yourself, and then I’ll dive in and overanalyze it. (Don’t worry, spoilers will be marked ahead of time.)

Space Station Weird

Space Station Weird is Luke Humphris’s big project over the past year. Set at some nebulous point in the future, it stars a “fridge dude” tasked with maintaining a seemingly-empty space station.

At the time of this writing, there are three episodes, totaling about seven minutes long. Check them out!

Space Station Weird doesn’t have intense action or laugh-out-loud punchlines. Instead, it thrives on mystery, worldbuilding, and understatement. And there’s a quiet sort of humor in the way the narrator sounds so bored as he describes such weird concepts.

I also like the fact that it’s done by one single person. It feels like something I theoretically could make, if I spent enough time practicing that art style, not to mention sound design, animating, and voice acting. The fact that it’s (almost) within the realm of my abilities makes it more impressive to me, since I can tell how it was done and how much effort that would take.

Whereas in the case of [insert big-budget movie/game], everything is polished so well that I can’t tell how they did it or how much effort it took per person, which makes it feel less impressive. Is that just me?

Analysis

Now that you’ve had a chance to see it for yourself, here are some of my ideas about the story. Light spoilers ahead, but I’ll avoid specifics.

Horror?

With only a few small tweaks, this could be a horror story. Being stranded alone in deep space is the perfect opportunity for horror. (We know this because it’s been done so many times.)

But in this case, we get this totally blasé fridge dude, who spends more time commenting on the station’s architecture than he does on genuinely worrying information. That isn’t to say he’s ignorant of danger, he’s just very calm about it. (With one exception so far.) And since the narrator is calm, the viewer gets to be calm too. Considering the title, that’s clearly intended: we aren’t watching “Space Station Existential Dread,” we aren’t watching “Space Station Eldritch Abominations from Beyond the Cosmos,” we’re just watching “Space Station Weird.”

It certainly could become a full-on horror story in the future, but I doubt it. I think it’s good at being a sci-fi slice of life, and will continue in that vein.

Speaking of which, sci-fi isn’t specifically about the future, or outer space; it’s about possibility. Sci-fi asks, what could happen if X was true? X can be “it’s the future in space with fancy gadgets,” but that isn’t necessary. And if that’s the only change, you get a relatively boring story.

On the other hand, Frankenstein is often described as a foundational work of science fiction despite being set on Earth in the modern day. It only had one major difference: scientists could (re)create life. (And promptly neglected and shunned said newly-created life.) Exploring this possibility is what makes it sci-fi, at least according to this one definition.

And I think Space Station Weird will fall under this definition too. It won’t be a story about Fridge Dude escaping alive (horror), or killing aliens in self-defense (action), but rather learning to coexist with whatever lurks here. You know, before it kills him. That’s totally different.

Luke Humphris’s prior works

Space Station Weird is still quite short, but it follows up on several concepts from Luke’s prior work, so let’s take a moment to look at that and see what we can learn. Or skip ahead to the next section, I can’t stop you.

My first exposure to Luke Humphris was while he was writing Dumb Drunk Australian, a long-form autobiographical webcomic. I stumbled upon it towards the end, but upon re-reading, it gets really dark at times. You’ve been warned.

Stories talking about my experiences with Australian drinking culture and toxic masculinity as well as feelings of not belonging.

TW: alcohol, suicide

Description from the comic’s Gumroad page

I think the comic caught my attention because I liked seeing him explain things with silly faces.
14 year old Luke's guide to getting the ladies / 1. Be super nice and treat them like a friend / "Hi Suzy, I got you some bronto-steaks" / 2. Wait / 3. Wait / 4. Don't be an asshole / 5. Wait / 6. wAiT / 7. Wait / 8. Wait / 9. Die alone and full of regrets

The things being explained weren’t always fun…

CW: alcohol

Sometimes have a beer and read at a different pub / Or drink with other people in town / Or with other people at the staff house / Or alone in my room playing the DS

I genuinely believe that if you do not learn from your mistakes, they will come back worse and worse / Not in a karma type of way, but in a logical way / [Luke holds an near-empty glass; to his left are two already-finished glasses; to his right are two still-full glasses marked "poison"] / "Why does this keep happening?" / Being ignorant to what changes you need will only make a hard life

…But Luke knows how to turn bad times into good anecdotes. Sometimes all it takes is a calm reaction to a bad situation, or a completely blank-faced reaction to a weird situation.

CW: alcohol

One night I began to stagger home in the early hours of the morning / I had a new bank account, so I kept the PIN written on a small piece of paper in my wallet / Which was stupid / I was too drunk to read the small print / So naturally I asked for help from the person behind me / The were even nice enough to drive me home / I am lucky nothing bad happened

The place I lived in was near Windsor Castle / When I came home one night there were 2 guards drinking in the kitchen with some housemates / It was a fairly loud house / [Luke, appearing half asleep, stares blankly at the guards] / [Luke continues blankly staring as he begins to walk away] / [Luke continues blankly staring as he walks out of sight]

So yeah, his writing style and sense of humor were there all along.

Dumb Drunk Australian ends rather suddenly. Maybe some pages got lost, but it was going to have to end at some point. The comic was supposed to be about looking back with the benefit of hindsight, not chronicling his life in real time.

Instead, he made a new comic to chronicle his life in real time. My Body is a Bad Robot tells us about his daily life as he struggles to quit drinking. It’s… more positive on average than DDA, but still has some very dark moments.

On the other hand, it includes this extremely normal picture of him questioning his sexuality.

I don't have to classify anything / I'm just very confused / Dropping expectations / Feeling like myself / Looking internally / [Luke is lying down with his mouth wide open; his eyes bulge out of his head and bend 180° down to look inside]

I said earlier I don’t think Space Station Weird is going to turn into straight-up horror. A big part of that is because Luke spent so much time developing his offbeat, understated brand of humor, and I’m hoping he keeps playing to this strength.

Space and time

In Bad Robot, Luke occasionally mentions that he’s spent a lot of time thinking about space.
I have thought about why I want to go to space / I think I want to be on the outside not involved in the world's problems / But that is just me running away again / "Later idiots"

It’s rude to try to psychoanalyze an author you’ve never met, but what if they psychoanalyze themself and you quote them? Is it ok then? (I sure hope so!)

Running from problems is a recurring theme in Luke’s comics. In DDA, Luke literally runs away from a faceless character labeled “problems.” More than once. And in one of the two (I forget which), he briefly fantasizes about going into a coma until everything is better.

Is this what Space Station Weird is about? Is Fridge Dude running from problems, spending all his time asleep so he can skip ahead to some future utopia? Is he traveling to space just to be away from the world?

I don’t think so. It isn’t foreshadowed, nor does he mention it. Fridge Dude isn’t running from anything, he’s just already disconnected from reality. The world isn’t progressing towards utopia as far as he can tell (though he mentions that other fridge dudes think so). He didn’t even choose to go to space; a powerful corporation assigned him there.

If Luke’s writing process is anything like mine, I’d guess that these are all recycled ideas. For instance, after spending some time thinking about the coma concept, you might realize the problems with it. Whether or not the world got better, you’d be too alienated to tell. Not something worth putting yourself through, but potentially something worth putting a fictional character through.

Exploring the possibility further, you might start to flesh out the character’s personality: aloof, skilled but maybe not up-to-date, with hobbies that allow for long breaks. If you’re simultaneously developing an interest in space, it wouldn’t take long to realize ways that it complements the character: physical distance as a metaphor for emotional distance, plenty of machinery that needs to be repaired by someone who knows how old tech works.

A lot can go wrong on a space station (don’t tap the glass!), and you might write those down to use as plot points. Then perhaps you spend a day sketching fun zero-G room designs, and in the process you happen to think of a reason a space station might have been built with all these different styles. Suddenly you have a backstory for the station! By following ideas and seeing what their consequences might be, you can turn vague daydreams into a detailed setting and plot outline. (Now just add small details to finish up.)

I don’t know if this is really how it went down, but it could have been. At the very least, think of these last four paragraphs as a glimpse into my own writing process.

The weird messages

Unmarked spoiler warning from here on out. This is your last chance to watch the videos for yourself.

Episode 0 is mainly scene setting, which might be why it’s number 0. Episode 1 is the first time we get a good look at the station and start getting hints about the plot.

– The station is built in a ton of mismatched styles. At least one of these areas still uses an ancient Windows computer.
– Fridge Dude was woken up to work on life support systems, systems that wouldn’t be needed if he wasn’t awake.
– There’s no timeline for when he gets to go home.
– Fridge Dude’s bunk has a frankly ridiculous number of scary messages scrawled above it, plus a bloody handprint and what might be a claw mark. Or as Fridge Dude describes them, “a few weird messages.”

What do these clues point towards? Is it all one big mystery or multiple smaller ones?

I think the most straightforward explanation would be the horror one. There’s a monster on board, one that requires oxygen and occasionally eats people. Each previous group built a different section of the station in a different style, then scratched a message or two into the bunk before the monster killed them, leaving them unable to update their computer software. Fridge Dude is merely the latest sacrifice, and isn’t meant to make it home. Why? Who knows.

Possibly the weirdest explanation I can think of is that all the machines gained sentience somehow. Phil didn’t ask Fridge Dude to repair the oxygen machine for the oxygen machine’s benefit, not Fridge Dude’s. That could also explain what Phil means by “family” and “we love each other.” And why a spacesuit is walking around on its own. This doesn’t explain the weird messages, or why Fridge Dude was penalized for losing the suit, but perhaps there are multiple separate mysteries afoot.

Speaking of which, I suspect the weird messages might not be literal. As Monty Python points out, if someone was dying they wouldn’t bother to carve “aaarrgghh,” they’d just say it. Would a dozen different murder-victims-to-be all take the time to carve their last words into the same spot? Seems unlikely if you think about it. I’ve thought of one reasonable alternative so far: maybe someone carved the messages there as a trick. For whatever reason they wanted to make Fridge Dude think people died, and went overboard with the number of messages.

Conclusion and episode 3

A lot of mysteries remain in the weird world of Space Station Weird. Which is good, because IMO that’s one of its biggest appeals. That and the humor.

Exciting news: as of March 31, episode 3 has arrived on Patreon! Consider supporting Luke to see it now, though I’m sure it’ll come out on YouTube soon enough.

Episode 3 is what prompted me to finish writing this post, which I made sure to do before watching it, so I wouldn’t spoil anything that isn’t already public. But I can at least make predictions! If trends continue, this episode will contain one new non-human character, a small number of answers, a large number of questions, and at least one moment where Fridge Dude has a calm reaction to something ridiculous and/or terrifying. Oh, and Luke mentioned that he pushed his limits on some scenes, so expect cool visuals too.

When it comes out, I’ll try to resist the urge to come back here and delete/replace all my wrong guesses. But no promises!

Status update: summer 2022-winter 2023

It’s been a while since I last posted, but don’t worry, I’ve been keeping busy.

Last time, I talked about spending the first half of 2022 working on Lime. Since then, most of my changes were categorized and merged. Small bug fixes are to be released in v8.0.1, while everything that counts as a new feature will come out later in v8.1.0. Except for the two huge changes that I spent 4+ months on; those have to wait for v8.2.0 because we want more time to test them.

I spent most of the second half of 2022 on my own projects, gradually working my way up through the layers Run is built on.

Echoes

Starting in late June, I turned my focus to Echoes, which I’ve previously described as my favorite entity-component-system framework available in Haxe.

Over the summer, I rewrote and documented practically the whole codebase. I fixed bugs, cleaned up the unit tests, added new features like better timekeeping, wrote lots of documentation, and generally improved maintainability. But I never lost sight of my goal, which was to get back to work on Runaway. I only added features that I knew I’d need immediately, rather than trying to plan ahead too far.

Runaway

Between August and October, I split my time between Echoes and Runaway, gradually shifting towards the latter but going back to Echoes whenever I found a new bug or needed a new feature. And the first part of Runaway I rewrote was the math. Math is fundamental to games, and it’s worth doing right.

Take this code from my “What is Runaway?” post:

class MotionSystem extends System {
    @:update private function accelerate(acceleration:Acceleration, velocity:Velocity, time:Float):Void {
        velocity.x += acceleration.x * time;
        velocity.y += acceleration.y * time;
        velocity.z += acceleration.z * time;
    }
    
    @:update private function move(velocity:Velocity, position:Position, time:Float):Void {
        position.x += velocity.x * time;
        position.y += velocity.y * time;
        position.z += velocity.z * time;
    }
}

Here, MotionSystem performs two basic operations. First it uses Acceleration to update Velocity, then it uses Velocity to update Position. Because this code gets run every frame, it produces the appearance of 3D motion.

To keep things simple for that post, I handled x, y, and z on separate lines. However, you really shouldn’t have to do that. Any decent math library will provide a more concise option. Before August, that was this:

class MotionSystem extends System {
    @:update private function accelerate(acceleration:Acceleration, velocity:Velocity, time:Float):Void {
        Point3Utils.addProductOf(velocity, acceleration, time, velocity);
    }
    
    @:update private function move(velocity:Velocity, position:Position, time:Float):Void {
        Point3Utils.addProductOf(position, velocity, time, position);
    }
}

Convenient! I know typing out Point3Utils.addProductOf() every time could get annoying, but it still beats OpenFL’s Vector3D class:

var scaledAcceleration:Vector3D = acceleration.clone();
scaledAcceleration.scaleBy(time);
velocity.incrementBy(scaledAcceleration);

Not only is Vector3D less convenient, you have to make a temporary object. For one object that’s fine, but video games can do hundreds if not thousands of these operations per second, and all of those temporary objects have to be garbage collected.

It’s at least possible to make Vector3D more convenient. Just add a “scaledBy()” function that calls clone() and then scaleBy(), combining lines 1 and 2 above. It still has the garbage collection problem, but at least the code looks neater:

velocity.incrementBy(acceleration.scaledBy(time));

But wait! It turns out you can avoid the garbage collection problem too, using inline constructors.

If we inline Vector3D’s constructor and functions, then the Haxe compiler will copy all that code into our function call. After moving the code, the compiler will be able tell that the clone is temporary, and doesn’t actually need to be allocated. Instead, the compiler can allocate local x, y, and z variables, which incur no garbage collection at all. Sadly, pretty much none of Vector3D is inlined (either for backwards compatibility reasons or because this isn’t a priority for OpenFL).

So I looked elsewhere. As luck would have it, the hxmath library provides a fully-inline Vector3 type. Not only does this solve the garbage collection issue, hxmath’s Vector3 offers a lot more features than OpenFL’s Vector3D. (IMO, hxmath is the best math library on Haxelib. Not that there’s much competition.) In particular, it allows you to use operators wherever such operators make sense:

class MotionSystem extends System {
    @:update private function accelerate(acceleration:Acceleration, velocity:Velocity, time:Float):Void {
        velocity += acceleration * time;
    }
    
    @:update private function move(velocity:Velocity, position:Position, time:Float):Void {
        position += velocity * time;
    }
}

(As of this writing, the += operator isn’t available on Haxelib. But it should come out in the next release.)

I initially planned to join forces, submitting improvements to hxmath rather than creating yet another math library. I submitted fixes for the bugs I found, but as I looked closer I started to find more and more things I’d do differently, as well as a couple inconsistencies. Anyone already using hxmath wouldn’t like sudden changes to function names or the order of a matrix’s elements. Breaking changes are always a hard sell, and I’d need to make a lot of them before I could use hxmath for Runaway.

So I began working on my own math library (working name “RunawayMath”), using the knowledge of macros I gained from working on Echoes. Specifically, I wrote macros to handle the repetitive parts. For instance, my Vector3 class can do in 13 lines what hxmath does in ~55.

@:elementWiseOp(A + B) @:inPlace(add)
private inline function sum(other:Vector3):Vector3;

@:elementWiseOp(A - B) @:inPlace(subtract)
private inline function difference(other:Vector3):Vector3;

@:elementWiseOp(A * B) @:commutative @:inPlace(multiply)
private inline function product(other:Vector3):Vector3;

@:elementWiseOp(A * B) @:commutative @:inPlace(multiplyF)
private inline function productF(scalar:Float):Vector3;

@:elementWiseOp(A / B) @:inPlace(divide)
private inline function quotient(other:Vector3):Vector3;

@:elementWiseOp(A / B) @:inPlace(divideF)
private inline function quotientF(scalar:Float):Vector3;

@:elementWiseOp(-A) private inline function inverse():Vector3;

@:elementWiseOp(A + B) tells the macro that when you type vectorA + vectorB, it should make a new vector with the sum of their elements (vectorA.x + vectorB.x, vectorA.y + vectorB.y, and vectorA.z + vectorB.z). @:inPlace tells the macro to enable the corresponding assign operator (+= and so on).

With macros in place, I was able to quickly churn out data types. First I re-implemented most of hxmath’s types, like vectors, lines, and quaternions (but skipped matrices because I don’t need them yet). Then, I added brand new ones, including spheres, planes, orthonormal bases, bivectors, and Bézier curves. At some nebulous point in the future, I plan to implement matrices and rotors.

I made sure to write unit tests as I went, something I’d never bothered with before. As a result, I’m confident that these new types will be less buggy than my old code. Notably, Run 1’s beta lacks shadows due to some sort of error in either my raycast code or my quaternion code. I never managed to find the issue, but I bet my new code will fix it without even trying.

I could go on about all the new features, but I really should save that for some other post.

Practical applications

By November, I was ready to start making use of all this mathematical awesomeness, by returning to the fundamentals of any platformer: running and jumping.

Well, running specifically. It’s the name of the game, after all!

Side-to-side movement

It isn’t really a secret that Run 1’s HTML5 beta looks and feels different from the Flash version, and part of that is the way the Runner moves side-to-side.

The old formula for acceleration (in Flash) used the slow Math.pow() function to produce a specific feeling. When I used a totally different function in HTML5, it produced a different feeling. I did that because I figured that if Run 3 didn’t need Math.pow(), Run 1 didn’t either. Problem was, I wasn’t very careful writing the new formula.

Now back to November 2022. This time, I wanted to try a suggestion I read in an article or social media comment. The idea is to graph speed vs. time, letting you see (and edit) how fast the player character accelerates.

The simplest case is constant acceleration. You gain the same speed every frame (until you reach the maximum), producing a straight line on the graph.

But that’s restrictive, so what about curved acceleration? Specifically, how about using those Bézier curves I mentioned above? It’s easy to get acceleration from a curve that represents velocity, and it lets me play around with a lot of options to see how they all feel.

And that’s without even changing the top speed or acceleration duration. I meant it when I said a lot of options. (Click to view.)

Out of all those options, I think the ones with high initial acceleration (convex shapes) feel the most natural.

The initial burst makes the controls feel more responsive, and then they gently approach the top speed so there’s no sudden jerk. Not only that, it’s true to life: as someone picks up speed in real life, air resistance gets stronger and they accelerate slower.

That said, the curves I showed above are extreme examples. Usually I aim for a bit less acceleration at first and leave a bit of jerk at the end. That way, it’s more obvious that you’re continuing to gain speed over time.

I don’t know which of these most closely matches the original Run 1, but I plan to find out.

Other basic components

By the way, my colorful-square demo app is more than just a rolling simulator. It’s actually a small platformer demo, with collisions and jumping and all that. Just… without any actual platforms, yet.

The “Mobility” button brings up the acceleration graph I’ve already shown. The “Jump” button lets you control how high each type of jump goes (or to disable specific types such as midair jumps). “Air” lets you tweak air resistance and wind speed.

But that’s far from all the components that went into this app. I also took the chance to rewrite and test each of the following:

  • Collisions, both for the invisible walls around the edges and for the boost pads in the middle. Solid collisions are automatically classified as “ground,” “wall,” or “ceiling,” making it easy to write ground-specific or wall-specific behavior. Like the walljump seen above.
  • Friction, including friction with moving platforms. A moving platform will pull you along with it (unless it’s icy), and you’ll keep that momentum if you jump. (I expect to use this for conveyors in Run 3.)
  • A brand new Orientation class that replaces the coordinate space approach I used to use. Turns out I was making things unnecessarily hard for myself, rotating the character’s coordinates rather than rotating the effects of the left/right/jump buttons.

How long can player_03 keep stalling?

As of this moment, I’m running dangerously low on things to do before I get back to work on Run 1. I’ve been putting that off working hard on underlying projects since… summer 2021? It’s certainly taken a while, but soon I’ll be done stalling preparing.

  • When Infinite Mode didn’t work, I could have looked for an easy workaround. Instead, I rewrote all loading code.
  • When I realized loading is best done using threads, I could have just used the tools that existed. Instead, I spent several months revising Lime’s threading classes and adding HTML5 support.
  • I made a decent-size demo project just to show off the new thread code. That took at least a month on its own, and I have no immediate plans for it, but at least it made pretty patterns.
  • I spent months cleaning up Lime’s submodule projects instead of leaving well enough alone. Now, I know more about this section of the code than almost anyone else, so I’m the one responsible for any new bugs that come up. Fortunately there haven’t been too many.
  • I spent over a month (hard to measure because it’s spread out) updating Echoes, even though it demonstrably already worked. It had a couple hiccups, sure, but I could have fixed those without rewriting all the documentation and unit tests. But hey, now we have more documentation and unit tests!
  • When I discovered constructor inlining, I dropped everything to rewrite Runaway’s math classes. They already worked fine, but nooo, the code was a little bit verbose, we couldn’t have that!
  • With new math classes available and a vague sense that I was overcomplicating space, I stumbled across the term orthonormal basis. And that inspired me to rethink the concepts of space, collisions, moving, and jumping all at once. I’m going to need to write a whole new “how space works in Run” post…
  • To take advantage of all this new code, I wrote the demo app shown above. It took about as long as the libnoise demo, but at least this time I plan to keep using it.
  • When first making the demo, I enabled one system at a time, so that I’d only have to debug one section of code at a time. Eventually, I decided just to enable all the systems from the Run 1 beta, even knowing I’d immediately get dozens if not hundreds of errors. I figured it’d let me keep stalling for a while longer locate lots of bugs all at once.

I estimate that fixing that last batch of bugs will take at least… oh. Uh, they’re already done. Took less than a week. I guess there was a benefit to all that feature creep careful planning and solid mathematical foundations.

Except, that is, for one single system. This system was especially complicated, and since I wanted to post this post, I left it commented out. Once I get this system working, I think that will be everything: I’ll be unable to keep stalling ready to work on Run 1.

I’ll be ready to polish the wonky character animations, make the motion feel more like it used to, and, of course, update the level load system to handle Infinite Mode levels, all those things I’ve been saying I want to do. But first, I need to fix up this one complicated system.

So which system is it? It is, in fact…

…the level loading system. Also, the only reason it’s so broken is because I made those other changes to how loading works. Fixing the errors will automatically also make the system compatible with Infinite Mode.

Huh. I guess that means I’m already done stalling. No more redoing old features, I’m officially now working on the thing that started this entire multi-year rabbit hole. Or maybe I could go implement rotors first… One more detour…

Status update: spring 2022

Last time, I said I was done with rabbit holes. I was wrong. Turns out, after escaping one rabbit hole, I promptly fell down the next.

First it was threads. I spent all of March working on an app to demonstrate threads in action. Sure I probably spent more time on it than strictly necessary, but I got a cool demo out of it. Plus, working on a practical example helps catch bugs and iron out rough edges.

With that done, I turned my focus to updating the 21 C++ libraries Lime uses. (Well, 19 out of the 21, anyway.) I forget exactly why I decided it needed to be done then and there, but I think a big part was the fact that people were raising concerns about large unresolved problems in OpenFL/Lime, and this was one I could solve. Technically I started last year, and this was me rolling up my sleeves and finishing it.

A couple stats for comparison: I spent ~4 months working on threads, and produced 53 commits. I spent ~2.5 months working on the C++ libraries, and produced 102 commits. And that’s why you can’t measure effort purely based on “amount of code written.” The thread classes took far more effort to figure out, but a lot of that effort was directed into figuring out which approach to take, and I only committed the version I settled on. The alternative options silently vanished into the ether.

All the while, I spent bits of time here and there working on issue reports. Tracking down odd bugs, adding small features, stuff like that. There are several changes in the pipeline, and if you want the technical details, check out my other post. (Scroll to the bottom for more on the big 19-library update.)

Big changes coming Lime’s way

Lime hasn’t received a new release version in over a year, even though development has kept going the whole time. As we gear up for the next big release, I wanted to take a look at what’s on the way.

Merged changes

These changes have already been accepted into the develop branch, meaning they’ll be available in the very next release.

Let’s cut right to the chase. We have a bunch of new features coming out:

  • 80f83f6 by joshtynjala ensures that every change submitted to Lime gets tested against Haxe 3. That means everything else in this list is backwards-compatible.
  • #1456 by m0rkeulv, #1465 by me, and openfl#2481 by m0rkeulv enable “streaming” music from a file. That is to say, it’ll only load a fraction of the file into memory at a time, just enough to keep the song going. Great for devices with limited memory!
    • Usage: simply open the sound file with Assets.getMusic() instead of Assets.getSound().
  • #1519 by Apprentice-Alchemist and #1536 by me update Android’s minimum-sdk-version to 21. (SDK 21 equates to Android 5, which is still very old. Only about 1% of devices still use anything older than that.) We’re trying to strike a balance between “supporting every device that ever existed” and “getting the benefit of new features.”
    • Tip: to go back, set <config:android minimum-sdk-version="16" />.
  • #1510 by ninjamuffin99 and Cheemsandfriends adds support for changing audio pitch. Apparently this feature has been missing since OpenFL 4, but now it’s back!
    • Usage: since AS3 never supported pitch, OpenFL probably won’t either. Use Lime’s AudioSource class directly.
  • 81d682d by joshtynjala adds Window.setTextInputRect(), meaning that your text field will remain visible when an onscreen keyboard opens and covers half the app.
  • #1552 by me adds a brand-new JNISafety interface, which helps ensure that when you call Haxe from Java (using JNI), the code runs on the correct thread.
    • Personal story time: back when I started developing for Android, I couldn’t figure out why my apps kept crashing when Java code called Haxe code. In the end I gave up and structured everything to avoid doing that, even at the cost of efficiency. For instance, I wrote my android6permissions library without any callbacks (because that would involve Java calling Haxe). Instead of being able to set an event listener and receive a notification later, you had to actively call hasPermission() over and over (because at least Haxe calling Java didn’t crash). Now thanks to JNISafety, the library finally has the callbacks it always should have had.
  • 2e31ae9 by joshtynjala stores and sends cookies with HTTP requests. Now you can connect to a server or website and have it remember your session.
  • #1509 by me makes Lime better at choosing which icon to use for an app. Previously, if you had both an SVG and an exact-size PNG, there was no way to use the PNG. Nor was there any way for HXP projects to override icons from a Haxelib.
    • Developers: if you’re making a library, set a negative priority to make it easy to override your icon. (Between -1 and -10 should be fine.) If you aren’t making a library and are using HXP, set a positive priority to make your icon override others.

…But that isn’t to say we neglected the bug fixes either:

Pull request #1517 by Apprentice-Alchemist is an enormous change that gets its own section.

For context, HashLink is the new(er) virtual machine for Haxe, intended to replace Neko as the “fast compilation for fast testing” target. While I find compilation to be a little bit slower than Neko, it performs a lot better once compiled.

Prior to now, Lime compiled to HashLink 1.10, which was three years old. Pull request #1517 covers two versions and three years’ worth of updates. From the release notes, we can look forward to:

  • new HashLink CPU Profiler for JIT
  • improved 64 bit support – windows release now 64 bit by default
  • new GC architecture and improvements / bugs fixes
  • support for hot reload
  • better stack primitives for faster haxe throw
  • captured stack for closures (when debugger connected)
  • and more!

(As someone who uses HashLink for rapid testing, I’m always happy to see debugging/profiling improvements.)

Apprentice-Alchemist put a lot of effort into this one, and it shows. Months of testing, responding to CI errors, and making changes in response to feedback.

Perhaps most importantly in the long run, they designed this update to make future updates easier. That paid off on April 28, when HashLink 1.12 came out at 3:21 PM GMT, and Apprentice-Alchemist had the pull request up to date by 5:45!

Pending changes

Lime has several pull requests that are – as of this writing – still in the “request” phase. I expect most of these to get merged, but not necessarily in time for the next release.

Again, let’s start with new features:

…And then the bug fixes:

  • #1529 by arm32x fixes static debug builds on Windows.
  • #1538 by me fixes errors that came up if you didn’t cd to an app’s directory before running it. Now you can run it from wherever you like.
  • #1500 by me modernizes the Android build process. This should be the end of that “deprecated Gradle features were used” warning.

Submodule update

#1531 by me is another enormous change that gets its own section. By far the second biggest pull request I’ve submitted this year.

Lime needs to be able to perform all kinds of tasks, from rendering image files to drawing vector shapes to playing sound to decompressing zip files. It’s a lot to do, but fortunately, great open-source libraries already exist for each task. We can use Cairo for shape drawing, libpng to open PNGs, OpenAL to play sound, and so on.

In the past, Lime has relied on a copy of each of these libraries. For example, we would open up the public SDL repository, download the src/ and include/ folders, and paste their contents into our own repo. Then we’d tell Lime to use our copy as a submodule.

This was… not ideal. Every time we wanted to update, we had to manually download and upload all the files. And if we wanted to try a couple different versions (maybe 1.5 didn’t work and we wanted to see if 1.4 was any better), that’s a whole new download-upload cycle. Plus we sometimes made little customizations unique to our copy repos. Whenever we downloaded fresh copies of the files, we’d have to patch our customizations back in. It’s no wonder no one ever wanted to update the submodules!

If only there was a tool to make this easier. Something that can download files from GitHub and then upload them again. Preferably a tool capable of choosing the correct revision to download, and merging our changes into the downloaded files. I don’t know, some kind of software for controlling versions…

(It’s Git. The answer is Git.)

There was honestly no reason for us to be maintaining our own copy of each repo, much less updating our copies by hand. 20 out of Lime’s 21 submodules have an official public Git repo available, and so I set out to make use of them. It took a lot of refactoring, copying, and debugging. A couple times I had to submit bug reports to the project in question, while other times I was able to work around an issue by setting a specific flag. But I’m pleased to announce that Lime’s submodules now all point to the official repositories rather than some never-updated knockoffs. If we want to use an update, all we have to do is point Lime to the newer commit, and Git will download all the right files.

But I wasn’t quite done. Since it was now so easy to update these libraries, I did so. A couple of them are still a few versions out of date due to factors outside of Lime’s control, but the vast majority are fully up-to-date.

Other improvements that happened along the way:

  • More documentation, so that if anyone else stumbles across the C++ backend, they won’t be as lost as I was.
  • Support for newer Android NDKs. Well, NDK 21 specifically. Pixman’s assembly code prevents us from updating further. (Still better than being stuck with NDK 15 specifically!)
  • Updating zlib fixes a significant security issue (CVE-2018-25032).

Looking forward

OpenFL and Lime are in a state of transition at the moment. We have an expanded leadership team, we’ll be aiming for more frequent releases, and there’s plenty of active discussion of where to go next. Is Lime getting so big that we ought to split it into smaller projects? Maybe! Is it time to stop letting AS3 dictate what OpenFL can and can’t do? Some users say so!

Stay tuned, and we’ll find out soon enough. Or better yet, join the forums or Discord and weigh in!

Haxelib review: libnoise

Libnoise

This is a haxe port of libnoise, the coherent noise library. The port is almost complete, only the gradient and noise2D utilities are missing.

Every now and then, I try out a new library and find something cool. libnoise is one such library, so it’s time for a Haxelib review!

In this post, I’ll explain what libnoise offers, how good it is at its job, and where it has room for improvement. Plus whatever else I think of along the way.

As its description states, libnoise is a tool for generating coherent noise. To keep things simple, I’m going to pretend “coherent noise” means “grayscale images.” But if you want to dig deeper, Rick Sidwell has an excellent rundown.

Let’s begin with some coherent noise.

This cloud-like image is what’s called Perlin noise, and it’s just one of many patterns libnoise can make.

Generators

Each of libnoise’s generators creates a distinct pattern. Four of these generators are dead simple.

  • Const a solid color. It can be any shade, though this demo keeps it simple.
  • Cylinder and Sphere create repeating gradients centered on the top-left corner. Because the demo is 2D, they barely resemble their namesakes, so you just have to imagine a bunch of 3D spheres nested inside one another, or a bunch of nested cylinders centered on the left edge.
  • Checker creates a checkerboard pattern on the pixel level. One pixel is white, the next is black, the next is white, and so on.

The other four involve randomness.

  • Perlin generates (what else?) Perlin noise. Not going to go into depth on how it works; see Rick Sidwell’s post for that. This example uses 8 octaves (fractal layers), meaning there’s extra detail but it takes longer to draw.
  • Billow creates what I’d describe as wandering lines, but under the hood it’s very similar to Perlin. If you compare the source code side-by-side, you’ll notice the algorithm is identical except for a single Math.abs() call on line 35.
  • RidgedMultifractal also creates wandering lines. I guess you could also call them “ridges”? That’s probably how it got that name. If you look at the source code, you’ll see that it’s also pretty similar to Perlin, though with a few more differences. In the end, it comes out looking like the inverse of Billow.
  • Voronoi creates a Voronoi diagram. This is a way of partitioning space based on a set of “seed” points, where every pixel is assigned to the closest seed… You know what? I can’t fit a full explanation here, so go read the article for the details.

3D patterns

I hinted at this already, but libnoise is built to generate 3D patterns. You can set the Z coordinate to 0 – as I did – and just make 2D images, but each of these 2D images is a cross-section of the full pattern. That’s why libnoise calls the classes Cylinder and Sphere rather than Line and Circle: the full 3D pattern really is cylindrical/spherical.

Voronoi, Perlin, Billow, and RidgedMultifractal all subtly change because of this. Each pixel in these patterns is influenced by points outside the 2D plane, meaning they appear different than if we were using a 2D algorithm to generate them. (Because the 2D algorithm would only calculate points in the 2D plane.) But it’s very hard to tell just by looking at a single image of them.

Here’s a way to see the difference. If we took different cross-sections of a cylinder, we could see its lines start to curve.

If we kept going until 90°, it would look fully circular, just like Sphere.

Operators

libnoise’s generators make the basic images, but its operators are where things get interesting. They modify those base images in all kinds of ways. Inverting, combining, you name it.

To get a good sense of how an operator works, it helps to see the input and output side-by-side. The downside is, it requires splitting up the canvas and cropping each input and output. To view more of a pattern, hover over it and click the “expand” button in the bottom corner.

Unary operators

The simplest operators are those that only take a single image as input, like the Rotate operator shown above. Most unary operators either perform simple arithmetic or move the pattern somehow.

The above three perform arithmetic on the brightness of each pixel. libnoise represents brightness using a value between -1 (black) and 1 (white).

  • Abs takes the absolute value of the brightness, so anything below 0 (gray) becomes positive. After this, no part of the pattern will be darker than gray.
  • Clamp sets a minimum and maximum value. This demo uses -0.5 (dark gray) as the minimum and 0.5 (light gray) as the maximum, but other programs could adjust further. This cuts out the shadows and highlights but leaves the midtones untouched.
  • Invert does exactly what it says on the tin.


Rotate, Scale, and Translate do exactly what their names imply. Though they can rotate, scale, and translate in any direction, this demo only includes two options each.

Turbulence moves pixels in random directions. “Low” and “high” refer to the maximum distance pixels are allowed to move. However, there’s a bit more to it than that, if we look closely. Fortunately, libnoise allows applying an operator to an operator, so we can Scale the Turbulence.

While it normally looks grainy, if we zoom in enough times, the turbulence starts to look surprisingly smooth. It turns out that it doesn’t move pixels completely at random. There’s an underlying pattern, and that pattern is called… Perlin noise.

Yes indeed, the Turbulence class generates Perlin noise to figure out where it should move each pixel. Perlin noise is a type of gradient noise, and gradient noise always looks smooth when you zoom in. The trick is that the Turbulence class does not zoom in by default, creating the illusion that it isn’t smooth.

Binary operators

libnoise’s two-input operators all perform arithmetic. As a reminder, libnoise represents brightness using a value between -1 (black) and 1 (white).

  • Add takes the sum of two patterns. If both inputs are bright, the output will be even brighter. If both are dark, the output will be even darker.
  • Subtract takes the difference. It’s the same as if you inverted the second pattern before adding.
  • Multiply takes the product of the numbers. Since we’re multiplying numbers less than 1, they always tend towards 0, and we end up with a lot of gray.
  • Average looks like Add but grayer, because that’s exactly what it is: (a + b) / 2. Strangely libnoise doesn’t include this operator, so I implemented it myself for demo purposes.
  • Min takes the darker value at each point.
  • Max takes the lighter value at each point.

Ternary operators

Finally, libnoise has two three-input operators, and they’re both very similar. Their output is always a combination of pattern 1 and pattern 2, and they use pattern 3 to decide what combination.

  • Select is all-or-nothing. If pattern 3 is darker than a certain value, it selects pattern 1 and shows that. If pattern 3 is above that threshold, it selects pattern 2.
  • Blend interpolates between the first two patterns, using pattern 3 to decide how much of each to blend. If pattern 3 is dark, it’ll use more of pattern 1. If pattern 3 is light, it’ll use more of pattern 2.
  • Select also has a fallOff value that makes it do a little of both. If a number is close to the threshold value, it blends just like Blend. If the number is farther away, it selects like normal.

Hopefully now you have a good idea of what each generator and operator does, so it’s time for some more interesting combinations. But first, it’s time to take the training wheels off:

The UI is simple to explain, if unwieldy. Hover over a section of canvas to see a dropdown. Pick an option from the dropdown to fill that part of the canvas. Scroll all the way to the bottom of the dropdown if you want to split the canvas up (or put it back together).

With that out of the way, here are some interesting patterns I’ve come across. Try them out, and be sure to try switching up the generators. You can always revert your changes by clicking the button again.

What patterns can you come up with? If you make something you want to share, you can copy it with ctrl+C, and others can paste it in with ctrl+V. Feel free to post it in the comments, but make sure to insert four spaces in front of your code. If you don’t, WordPress could mess up your formatting.

For those who want to dig even deeper, check out the demo’s source code or libnoise’s source code.

Oh right, the review

I was supposed to be reviewing this library, not just showing off a bunch of cool patterns. …Though on the other hand, showing the cool patterns gives you an idea of what the library is good at. Isn’t that half the point of a review?

Well, perhaps I should do a traditional review too. I’d describe libnoise as functional and well-designed, but lacking in documentation and not very beginner-friendly.

Let’s look at some code. Here’s just about the simplest way you could implement the “minimum of two gradients” sample. (As a reminder, that’s Min, Billow, and Cylinder.)

//Step 1: define the pattern.
var billow:Billow = new Billow(0.01, 2, 0.5, seed, HIGH);
var cylinder:Cylinder = new Cylinder(0.01);
var min:Min = new Min(billow, cylinder);

//Step 2: create something to draw onto.
var bitmapData:BitmapData = new BitmapData(512, 512, false);

//Step 3: iterate through every pixel.
for(x in 0...bitmapData.width) {
    for(y in 0...bitmapData.height) {
        //Step 3a: Get the pixel value, a number between -1 and 1. Use a z coordinate of 0.
        var value:Float = min.getValue(x, y, 0);
        
        //Step 3b: Convert to the range [0, 255].
        var brightness:Int = Std.int(128 + value * 128);
        if(brightness < 0) {
            brightness = 0;
        } else if(brightness >= 256) {
            brightness = 255;
        }
        
        // Step 3c: Convert to a color.
        var color:Int = brightness << 16 | brightness << 8 | brightness;
        
        //Step 3d: Save the pixel color.
        bitmapData.setPixel32(x, y, color);
    }
}

//Step 4: display and/or save the bitmap.
addChild(new Bitmap(bitmapData));

libnoise makes steps 1 and 3a easy enough, but you have to fill in all the other steps yourself. That’s fine – maybe even desirable – for advanced users. However, a new user who just wants to try it out isn’t going to appreciate the extra work. The new user would like to be able to do something like this:

//Step 1: define the pattern.
var billow:Billow = new Billow(0.01, 2, 0.5, seed, HIGH);
var cylinder:Cylinder = new Cylinder(0.01);
var min:Min = new Min(billow, cylinder);

//Step 2: make the drawing.
var noise2D:Noise2D = new Noise2D(512, 512, min);
var bitmapData:BitmapData = noise2D.getBitmapData(GradientPresets.grayscale);

//Step 3: display and/or save the bitmap.
addChild(new Bitmap(bitmapData));

Room for improvement

Here’s what I’d work on if I ever began using libnoise seriously.

  • Finish the port. The GradientPresets and Noise2D classes I mentioned would make it much easier to add color and export images. They existed in the original version(s) of libnoise, but didn’t survive the port.
    • The author explained that they didn’t want to tie libnoise to outside libraries (like OpenFL), but I’d say it’s more than worth it. Besides, conditional compilation makes it easy to support OpenFL without depending on it.
  • More generators. If you ignore the fluff, libnoise only implements two noise algorithms: Perlin and Voronoi. There are plenty more algorithms out there, including Simplex, an improvement on Perlin noise whose patent recently expired, and Worley noise, resembling a textured Voronoi diagram.
    • libnoise already implements value noise under the hood, but doesn’t make it available to the user. It’d be very easy to add.
    • And there’s no need to limit the library to coherent noise. Why not pure static?
  • More options for existing operators. Turbulence, for instance, can’t be scaled without applying an Scale operator to the whole image. But what if you want the zoomed-in turbulence effect without zooming in on the underlying pattern?
  • More operators. I made a custom Average operator for the demo, but that should be in the base library. Besides that, I’d like to see operators to blur, lighten, or darken the image.
  • Better performance, if possible.
  • Better documentation and code style. libnoise is a port of a port, and each time it was ported, most of the comments were lost or replaced. (Nor were all of the surviving comments accurate, due to overzealous copy-pasting.)

That’s all that comes to mind, which is probably a good sign. libnoise is entirely usable as-is, even if there’s work left to do.

Comparing libnoise to other libraries

I’m aware of four different noise libraries in Haxe. libnoise, MAN-Haxe, noisehx, and hxNoise. (Interesting coincidence: all four are from 2014-2016 and haven’t been updated since.)

Let’s look at what each library brings to the table.

  • noisehx offers Perlin noise, and that’s it. It’s also the only library to offer both 2D and 3D Perlin noise, meaning you can save processing time if you only need a 2D image.
  • hxNoise offers diamond-square noise as well, which basically functions as a faster but lower-quality version of Perlin noise. Its Perlin noise is 3D but its diamond-square noise is 2D.
  • MAN-Haxe doesn’t offer Perlin noise at all, though it offers two other types of noise that resemble it. It also offers Worley noise and a couple maze-generation algorithms. That last one is why it’s called “Mazes And Noises.” Oh, and all of this is 2D.

I’d call MAN-Haxe the most grounded of the libraries. It’s built for one specific purpose: to generate maps for HaxeFlixel games. It just incidentally happens to do images too. If your goal is to generate 2D rectangular maps for a 2D game and you happen to be using HaxeFlixel, then MAN-Haxe is right for you.

None of these alternative libraries offer operators, which means libnoise can generate a greater variety of images. That said, it isn’t like libnoise’s operators are particularly complicated. If you wanted to invert hxNoise’s diamond-square noise, you could do that yourself.

…I do believe that wraps it up. For convenience, here’s a shortcut back up to the demo. Now go forth and make some noise!

Status update: winter 2022

In my last status update, I said I’d (1) update Android libraries and then (2) add Infinite Mode to the HTML5 version. As I mentioned in an edit, that first step took a single day. Then the second took… three months. So far.

It all unfolded something like this:

  1. I wanted to add Infinite Mode.
  2. The loading code I used for Explore Mode isn’t up to the task of handling procedurally-generated levels. (Plus I have other plans that require seamless loading.) So I put step 1 on pause in order to write better loading code.
  3. I realized/decided that good loading code needs to support threads. You don’t have to use them, but they should be an option. The good news is, Lime already supports threads. The bad news: not in HTML5. So I put step 2 on hold as I worked to update Lime.
  4. I realized HTML5 threads just aren’t compatible with Lime’s classes, so I took them out and made virtual threads, which are also pretty good.
  5. I stumbled across another way to do HTML5 threads, and realized this new way actually could work in Lime.
  6. I took a bit of a detour, trying to emulate a JavaScript keyword.

Every time I thought I’d reached the bottom of this rabbit hole, I found a way to keep digging. But I’m happy to announce that instead of proceeding with step 6, I’ve turned around and begun the climb back out:

  1. As of today, I finished combining the ideas from steps 3-5 above, which in theory fixes threads in Lime once and for all. I’m sure the community will suggest changes, but I can hopefully shift most of my focus.
  2. Use threads/virtual threads to achieve my vision of flexible and performant loading.
  3. Use this new flexibility to load Infinite Mode’s levels.

I’m also well aware that Run Mobile doesn’t work on Android 12. I’ve been too busy with threads to take a look, but now that I’m (very nearly) done, I really ought to do that next. Update: I did! The fixed version is available on Google Play.

A problem not worth solving

I spent most of Thursday working on a problem. After about ten hours of work, I reverted my changes. Here’s why.

The problem

This past week or so, I’ve been working on a new HTML5Thread” class. One of Lime’s goals is “write once, deploy anywhere,” so I modeled the new class after the existing Thread class. So if you write code based on the Thread class, I want that same code to work with HTML5Thread.

For instance, the Thread class provides the readMessage() function. This can be used to put threads on pause until they’re needed (a useful feature at times), and so I set out to copy the function. In JavaScript, if you want to suspend execution until a message arrives, you use the appropriately-named await keyword.

Just one little problem. The await keyword only works inside an async function, and there’s just no easy way to make those in Haxe. You’d have to modify the JavaScript output file directly, and that’s multiple megabytes of code to sift through.

My solution

At some point I realized that while class functions are difficult, you can make async local functions without too much trouble.

With a workaround in mind, I thought about how best to add it to Lime. As mentioned above, Lime is designed around the assumption that everything should be cross-platform. So if I was going to support async functions in JavaScript, I wanted to support them on all targets.

It would be easy to use: call Async.async() to make an async function, then call Async.await() to wait for something. These functions would then use some combination of JavaScript keywords, Lime Futures, and macro magic to make it happen. You wouldn’t have to worry about the details.

To make a long story short, it almost worked.

The problems with my solution

  1. My solution was too complicated.
  2. My solution wasn’t complicated enough.

My code contained a bunch of little “gotchas”; there were all kinds of things you weren’t allowed to do, and the error message was vague. “Invalid location for a call to Async.await().” Then if you wanted to use the await keyword (which is kind of the point), you had to write JavaScript-specific code (which is against Lime’s design philosophy). Oh, and there were a couple other differences between platforms.

From the user’s point of view, it was too complicated. But the only way to make it simpler would be to add a lot more code under the hood, so in a way, it also wasn’t complicated enough.

Doing this right would take a whole library’s worth of code and documentation, and as it turns out, that library already exists.

Perspective

In my last post, I quoted Elon Musk saying “First, make your requirements less dumb.” But as I said, I don’t think that’s the right order of operations. I mean, yeah, start out with the best plan you can make. But don’t expect to plan correctly first try. You need perspective before you can get the requirements right, and the easiest way to get perspective is to forge ahead and build the thing.

For instance, when I started this detour, I had no idea there might be valid and invalid locations to call Async.await(). That only occurred to me after I wrote the code, tried experiments like if(Math.random() < 0.5) Async.await(), and saw it all come crashing down. (Actually it wasn’t quite that dramatic. It just ran things out of order.)

Ten hours of work later, I have a lot more perspective. I can see how big this project is, how hard the next steps will be, and how useful users might find it. Putting all that together, I can conclude that this falls firmly in the category of feature creep.

Lime doesn’t need these features at the moment. Not even for my upcoming HTML5Thread class. All that class needs is the async keyword in one specific spot, which only takes eight lines of code. Compare that to my original solution’s 317 lines of code that still weren’t enough!

Basically what I’m saying is, I’m glad I spent the time on this, because I learned worthwhile lessons. I’m also glad I didn’t spend more than a day on it.

On starting over

(Don’t panic, I’m not personally starting over.)

How many of you have heard of the band Wintergatan?

Martin Molin, the face of Wintergatan, is a musician/engineer who rose to Internet fame after building the “Marble Machine” you see above. I assume you can tell why – that thing is really cool.

Sadly, from an engineering standpoint, the machine was a nightmare. Only capable of playing songs very similar to the one it was built for, and held together with elbow grease and wishful thinking. Marbles could collide if it tried to play adjacent notes, it was hard to time the notes properly, and the marbles kept clogging up or spilling out.

So he started over.

Marble Machine X

On January 1 2017, Martin announced the Marble Machine X, an entirely new device that would fix all the flaws in the original. Over the next four and a half years, he posted regular development updates. Even if – like me – you only watched some of the videos, you’d still learn a lot about both the MMX and mechanical engineering.

Martin went all out this time around, using CAD and CNC to build parts with millimeter precision, prototyping five different versions of a single piece so he could test them side-by-side, taking hours of video footage and building enormous spreadsheets with the data, measuring exactly how many milliseconds early or late the notes were, and taking design suggestions from a worldwide community of engineers. Most of all, he was unafraid to remove parts that he’d spent weeks or months on, if they weren’t quite working right.

It’s not really awesome to spend one and a half weeks building something that you have to redo, but I’m really used to that, and I’m actually good at starting over… I’m not so interested in this machine if it doesn’t play good music.

-Part of Martin’s heartfelt speech. (Make sure to watch the video for the rest.)

He sure did start over. Often enough that his angle grinder and “pain is temporary” catchphrase became a community meme, and then ended up on merchandise.

Was it worth it? Oh yeah. Looking at his last edited video before he switched to raw streams, the MMX ended up as an engineering marvel. Not only does it look great, it can drop thousands of marbles without error. When there is an error (5:26), he can instantly diagnose the problem and swap out the parts needed to fix it, no angle grinder necessary. Immediately after fixing it, he tried again and dropped thirty thousand in a row with zero errors. Four years well spent, I’d say!

So, just this January, it occurred to me that I hadn’t heard from Martin since that last video. The one posted all the way back in June. I didn’t mean to forget about him. In fact, I’m subscribed. Sure I skipped all the streams, but why did he stop posting edited videos?

A Lesson in Dumb Design

Wintergatan’s latest video, posted last September, has the answers. It’s titled “Marble Machine X – A Lesson in Dumb Design,” and in it, Martin discusses “dumb requirements” in the MMX.

First, make your requirements less dumb. Your requirements are definitely dumb. It does not matter who gave them to you; it’s particularly dangerous if a smart person gave you the requirements, because you might not question them enough. […] It’s very common; possibly the most common error of a smart engineer is to optimize the thing that should not exist.

-Elon Musk

Leaving aside Elon Musk himself, this seems like good advice. Martin gives an example of how it applies to the MMX at 5:49: he’s built the machine based off the fundamental assumption that marbles should always follow constrained single-file pathways. All the situations he’s encountered over the years where marbles would clog up, or apply pressure to a piece of tubing and burst out, or clog up, or jump over dividers, or clog up – all of these situations resulted from trying to constrain the marbles more than necessary.

Most were fixable, of course. He’s got well over a hundred videos’ worth of solved problems. But as he graduated from testing a few marbles per second to playing entire songs, he discovered more and more things wrong. Eventually, he concluded that the MMX, despite all the work put into it, wasn’t fixable. Now, he’s planning to produce one complete song with it, and then – once again – start over.

Judging by the YouTube comments, the community did not take this news well.

Drummers lose drum sticks. Violinists break bows. Guitarists lose picks. The marble machine can drop a marble.

-Thomas R

The MMX is literally almost complete and could be complete if only you allowed for a margin of error and stopped reading into all these awful awful self-help books.

-rydude998

“Make requirements less dumb” is a fantastic approach, but please don’t forget that “looks cool” is not a dumb requirement for your project.

-David H (referring to when Martin talked about form vs. function)

The perfect marble machine isn’t going to happen unless you seriously water down the beautiful artistic aspects that made the MMX so special to begin with. If that’s what it takes, then what’s the point? You’ll have a soulless husk of what was previously a wonderful and inspiring piece of art.

-BobFoley1337

This is the story of an artist who became an engineer to build his art and in so doing forgot the meaning of art.

-Nick Uhland

Note: If I quoted you and you’d rather I didn’t, let me know and I’ll take it down.

All good points. I’m not necessarily on board with the tone some of them took, but can you blame them? The project seemed so close, and so many people were excited to see the culmination of all this work, and then Martin pulls the rug out from under everyone.

But before we judge, let’s hear Martin’s side of the story:

I got many ideas for how to design a simpler, functional Machine, and I can’t stop thinking about it. I have heard that when you build your third house, you get it right. I think the same goes for Marble Machines.

[…]

I do know the generous response and objections that most crowdfunders have when I describe the Marble Machine X as a failure, and you are all correct. Its not a complete failure. What I have learned in the MMX process is the necessary foundation for the success of MMX-T.

[…]

If it’s hard to understand this decision let me provide some context: The MMX looks like it is almost working, but it isn’t. The over-complex flawed design makes the whole machine practically unusable. I now have the choice between keeping patching up flaws, or turn a page and design a machine which can be 10X improved in every aspect.

This may not be surprising coming from the guy who built four entire game engines between the three Run games, but I’m sympathetic to Martin. I know all too well that a thing that looks almost perfect from an outsider’s perspective can be a mess inside.

The Codeless Code: Case 105: Navigation

The Codeless Code is a collection of vaguely-Zen-like stories/lessons about programming. The premise is odd at first, but just go with it.

A young nun approached master Banzen and said:

“When first presented with requirements I created a rough design document, as is our way.

“When the rough design was approved I began a detailed design document, as is our way. In so doing I realized that my rough design was ill-considered, and thus I discarded it.

“When the detailed design was approved I began coding, as is our way. In so doing I realized that my detailed design was ill-considered, and thus I discarded it.

“My question is this:

“Since we must refactor according to need, and since all needs are known only when implementation is underway, can we not simply write code and nothing else? Why must we waste time creating design documents?”

Banzen considered this. Finally he nodded, saying:

“There is no more virtue in the documents than in a handful of leaves: you may safely forgo producing either one. Before master Mugen crossed the Uncompiled Wasteland he made eight fine maps of the route he planned to take. Yet when he arrived at the temple gates he burned them on the spot.”

The nun took her leave in high spirits, but as she reached the threshold Banzen barked: “Nun!”

When the nun turned around, Banzen said:

“Mugen was only able to burn the maps because he had arrived.”

I hope the analogy here is clear. When Martin built the original Marble Machine, he produced a single song and retired it. He then built the Marble Machine X, and plans to produce a single song before retiring it too. Now he’s working on the Marble Machine X-T, and he’s hoping that “when you build your third house, you get it right” applies here too.

He could never have made it this far if not for the first two machines. If he hadn’t built the original, he wouldn’t have known where to start on the second. If not for spending years on the MMX fixing all kinds of issues and making it (seemingly) almost work, he wouldn’t know where to start designing the third. Years of building the machine gave him a clearer picture than any amount of planning, and that picture is the only reason he can perform the “first” step of making his requirements less dumb.

I don’t think Martin could have gotten the requirements right on his first or second try, but it’s good that he tried. That was the other point of the “Navigation” parable. Mugen was only able to burn the maps because he had arrived. If Martin hadn’t started by making a solid plan, the MMX could not have been as good as it ended up being. If the MMX hadn’t reached the point of “almost working,” its greatest flaws wouldn’t have been exposed.

The Codeless Code: Case 91: The Soul of Wit

And now we arrive at how this relates to my own work. As I said at the beginning, I’m not starting anything over. However, I recently realized I needed to pivot a little.

I had built my code one feature at a time. Like Martin testing 30,000 marbles, I tested simple cases, over and over, and they worked. Then, like Martin livestreaming actual music, I devised a real-world example. It was basic, but it was something someone might actually want to do.

And that led to a cascade of problems. Things I hadn’t thought of while planning but which were obvious in retrospect. Problems with easy solutions. Problems with hard solutions. All kinds of stuff.

I was capable of fixing these problems. In fact, I had a couple different avenues to explore; at least one would certainly have worked. How could I be so certain I was on the wrong track?

Wangohan […] emailed his predicament to the telecommuting nun.

“I know nothing of this framework,” the nun wrote back. “Yet send me your code anyway.”

Wangohan did as he was asked. In less than a minute his phone rang.

“Your framework is not right,” said Zjing. “Or else, your code is not right.”

This embarrassed and angered the monk. “How can you be so certain?” he demanded.

“I will tell you,” said the nun.

Zjing began the story of how she had been born in a distant province, the second youngest of six dutiful daughters. Her father, she said, was a lowly abacus-maker, poor but shrewd and calculating; her mother had a stall in the marketplace where she sold random numbers. In vivid detail Zjing described her earliest days in school, right down to the smooth texture of the well worn teak floors and the acrid yet not unpleasant scent of the stray black dog that followed her home in the rain one day.

“Enough!” shouted the exasperated Wangohan when a full hour had passed, for the nun’s narrative showed no sign of drawing to a close. “That is no way to answer a simple question!”

“How can you be so certain?” asked Zjing.

I was writing a tutorial as I went, and that’s what tipped me off.

Each time I came up with a workaround, I had to imagine explaining it in the tutorial: “If you’re using web workers and passing a class instance from your main class to your web worker, you’ll need to add that class to the worker’s header, and then call restoreInstanceMethods() after the worker receives it. This is enough if that’s the only class you’re using but fails if you’re using subclasses that override any of the instance methods, so in that case you also need to these five other steps…”

Which is a terrible tutorial! Way too complicated. My framework was not right, or else, my code was not right. It was time to step back and rethink my requirements.

A mistaken assumption

When this all began, I had one goal: fulfill Lime issue #1081: Add asynchronous worker support for the HTML5 target. This core goal led to all my other requirements:

  1. Use web workers.
  2. Maintain backwards compatibility.
  3. Match Lime’s coding style.
  4. Write easy-to-use code.

Clearly, I was already violating requirement #4. And a couple weeks ago, I’d also realized that #1 and #2 were incompatible. Web workers were always going to break existing code, which is why I’d made them opt-in. You can’t use them by accident; you have to turn them on by hand. I was arguably also violating #3: web-worker-specific code was now taking up the majority of two files that weren’t supposed to be web-worker-specific. (Which could be ok in another context, but it’s not how Lime likes to handle these situations.)

No other feature in Lime requires reading so much documentation just to get started. Nothing else in Lime has this many platform-specific “gotchas.” Very few other things in Lime require opting in the way this does. This new code…

This new code never belonged in Lime.

That was my faulty assumption. I’d assumed that because the feature was on Lime’s wishlist, it belonged in Lime. But Lime is about making code work the same on all platforms, and web workers are just too different, no matter how much I try to cover them up with syntax sugar. In reality, the feature doesn’t belong in Lime or on Lime’s wishlist, a fact that became clear only after months of work.

Once again, I’m not starting over here. For a time, I thought I had to, but in fact my code is pretty much fine. My mistake was trying to put that code where it didn’t belong. The correct place would be a standalone library, which is the new plan. (As for Lime issue #1081, I’ve come up with a promising single-threaded option. Not quite the same, but still good.)

I’m confident I’m making the right decision here. The pieces finally fit together and the finish line is in sight.

Hopefully, Martin is making the right decision too. His finish line is farther off, but he’s made a good map to guide him there. Whether he burns that map upon arrival remains to be seen.

Guide to threads in Lime

Disclaimer: this guide focuses on upcoming features, currently only available via Git.

Concurrent computing

Concurrent computing is a form of computing in which several computations are executed concurrently—during overlapping time periods—instead of sequentially—with one completing before the next starts.

This is a property of a system—whether a program, computer, or a network—where there is a separate execution point or “thread of control” for each process. A concurrent system is one where a computation can advance without waiting for all other computations to complete.

Concurrent computing is a form of modular programming. In its paradigm an overall computation is factored into subcomputations that may be executed concurrently. Pioneers in the field of concurrent computing include Edsger Dijkstra, Per Brinch Hansen, and C.A.R. Hoare.

[Source: English Wikipedia.]

In simpler terms, concurrent execution means two things happen at once. This is great, but how do you do it in OpenFL/Lime?

Choosing the right tool for the job

This guide covers three classes. Lime’s two concurrency classes, and Thread, the standard class they’re based on.

Class Thread Future ThreadPool
Source Haxe Lime Lime
Ease of use ★★★★★ ★★★★☆ ★★★☆☆
Thread safety ★☆☆☆☆ ★★★★☆ ★★★★☆
HTML5 support No Yes Yes

But before you pick a class, first consider whether you should use threads at all.

  • Can you detect any slowdown? If not, threads won’t help, and may even slow things down.
  • How often do your threads interact with the outside world? The more often they transfer information, the slower and less safe they’ll be.

If you have a slow and self-contained task, that’s when you consider using threads.

Demo project

I think a specific example will make this guide easier to follow. Suppose I’m using libnoise to generate textures. I’ve created a feature-complete app, and the core of the code looks something like this:

private function generatePattern(workArea:Rectangle):Void {
    //Allocate four bytes per pixel.
    var bytes:ByteArray = new ByteArray(
        Std.int(workArea.width) * Std.int(workArea.height));
    
    //Run getValue() for every pixel.
    for(y in Std.int(workArea.top)...Std.int(workArea.bottom)) {
        for(x in Std.int(workArea.left)...Std.int(workArea.right)) {
            //getValue() returns a value in the range [-1, 1], and we need
            //to convert to [0, 255].
            var value:Int = Std.int(128 + 128 * module.getValue(x, y, 0));
            
            if(value > 255) {
                value = 255;
            } else if(value < 0) {
                value = 0;
            }
            
            //Store it as a color.
            bytes.writeInt(value << 16 | value << 8 | value);
        }
    }
    
    //Draw the pixels to the canvas.
    bytes.position = 0;
    canvas.setPixels(workArea, bytes);
    bytes.clear();
}

The problem is, this code makes the app lock up. Sometimes for a fraction of a second, sometimes for seconds on end. It all depends on which pattern it’s working on.

(If you have a beefy computer and this looks fine to you, try fullscreen.)

A good user interface responds instantly when the user clicks, rather than locking up. Clearly this app needs improvement, and since the bulk of the work is self-contained, I decide I’ll solve this problem using threads. Now I have two problems.

Using Thread

The easiest option is to use Haxe’s Thread class. Since I know a single function is responsible for the freezing, all I need to do is change how I call that function.

-generatePattern(new Rectangle(0, 0, canvas.width, canvas.height));
+Thread.create(generatePattern.bind(new Rectangle(0, 0, canvas.width, canvas.height)));

View full changes

Thread.create() requires a zero-argument function, so I use bind() to supply the rectangle argument. With that done, create() makes a new thread, and the app no longer freezes.

I’d love to show this in action, but it doesn’t work in HTML5. Sorry.

The downside is, the app now prints a bunch of “null pointer” messages. It turns out I’ve added a race condition.

Thread safety basics

The problem with Haxe’s threads is the fact that they’re just so convenient. You can access any variable from any thread, which is great if you don’t mind all the subtle errors.

My generatePattern() function has two problem variables:

  • module is a class variable, and the main thread updates it with every click. However, generatePattern() assumes module will stay the same the whole time. Worse, module briefly becomes null each time it changes, and that can cause the “null pointer” race condition I mentioned above.
  • canvas is also a class variable, which is modified during generatePattern(). If multiple threads are going at once, it’s possible to modify canvas from two threads simultaneously. canvas is a BitmapData, so I suspect it will merely produce a garbled image. If you do the same to other object types, it could permanently break that object.

Before I go into too much detail, let’s try a simple solution.

-Thread.create(generatePattern.bind(new Rectangle(0, 0, canvas.width, canvas.height)));
+lastCreatedThread = Thread.create(module, generatePattern.bind(new Rectangle(0, 0, canvas.width, canvas.height)));
-private function generatePattern(workArea:Rectangle):Void {
+private function generatePattern(module:ModuleBase, workArea:Rectangle):Void {
    //Allocate four bytes per pixel.
    var bytes:ByteArray = new ByteArray(
        Std.int(workArea.width) * Std.int(workArea.height));
    
    //Run getValue() for every pixel.
    for(y in Std.int(workArea.top)...Std.int(workArea.bottom)) {
        for(x in Std.int(workArea.left)...Std.int(workArea.right)) {
            //getValue() returns a value in the range [-1, 1], and we need
            //to convert to [0, 255].
            var value:Int = Std.int(128 + 128 * module.getValue(x, y, 0));
            
            if(value > 255) {
                value = 255;
            } else if(value < 0) {
                value = 0;
            }
            
            //Store it as a color.
            bytes.writeInt(value << 16 | value << 8 | value);
        }
    }
    
+   //If another thread was created after this one, don't draw anything.
+   if(Thread.current() != lastCreatedThread) {
+       return;
+   }
+   
    //Draw the pixels to the canvas.
    bytes.position = 0;
    canvas.setPixels(workArea, bytes);
    bytes.clear();
}

View full changes

Step one, pass module as an argument. That way, the function won’t be affected when the class variable changes. Step two, enforce a rule that only the last-created thread can modify canvas.

Even then, there’s still at least one theoretical race condition in the above block of code. Can you spot it?

Whether or not you find it isn’t the point I’m trying to make. My point is that thread safety is hard, and you shouldn’t try to achieve it alone. I can spot several types of race condition, and I still don’t trust myself to write perfect code. No, if you want thread safety, you need some guardrails. Tools and design patterns that can take the guesswork out.

My favorite rule of thumb is that every object belongs to one thread, and only that thread may modify that value. And if possible, only that thread should access the value, though that’s less important. Oftentimes, this means making a copy of a value before passing it, so that the receiving thread can own the copy. This rule of thumb means generatePattern() can’t call canvas.setPixels() as shown above, since the main thread owns canvas. Instead, it should send a thread-safe message back and allow the main thread to set the pixels.

And guess what? Lime’s Future and ThreadPool classes provide just the tools you need to do that. In fact, they’re designed as a blueprint for thread-safe code. If you follow the blueprint they offer, and you remember to copy your values when needed, your risk will be vastly reduced.

Using Future

Lime’s Future class is based on the general concept of futures and promises, wherein a “future” represents a value that doesn’t exist yet, but will exist in the future (hence the name).

For instance, BitmapData.loadFromFile() returns a Future<BitmapData>, representing the image that will eventually exist. It’s still loading for now, but if you add an onComplete listener, you’ll get the image as soon as it’s ready.

I want to do pretty much the exact same thing in my sample app, creating a Future<BitmapData> that will wait for the value returned by generatePattern(). For this to work, I need to rewrite generatePattern() so that it actually does return a value.

As discussed under thread safety basics, I want to take both module and workArea as arguments. However, Future limits me to one argument, so I combine my two values into one anonymous structure named state.

-private function generatePattern(workArea:Rectangle):Void {
+private static function generatePattern(state: { module:ModuleBase, workArea:Rectangle }):ByteArray {
    //Allocate four bytes per pixel.
    var bytes:ByteArray = new ByteArray(
        Std.int(workArea.width) * Std.int(workArea.height));
    
    //Run getValue() for every pixel.
    for(y in Std.int(workArea.top)...Std.int(workArea.bottom)) {
        for(x in Std.int(workArea.left)...Std.int(workArea.right)) {
            //getValue() returns a value in the range [-1, 1], and we need
            //to convert to [0, 255].
            var value:Int = Std.int(128 + 128 * module.getValue(x, y, 0));
            
            if(value > 255) {
                value = 255;
            } else if(value < 0) {
                value = 0;
            }
            
            //Store it as a color.
            bytes.writeInt(value << 16 | value << 8 | value);
        }
    }
    
-    //Draw the pixels to the canvas.
-    bytes.position = 0;
-    canvas.setPixels(workArea, bytes);
-    bytes.clear();
+    return bytes;
}

Now I call the function, listen for the return value, and draw the pixels.

-generatePattern(new Rectangle(0, 0, canvas.width, canvas.height));
+future = Future.withEventualValue(generatePattern, { module: module, workArea: new Rectangle(0, 0, canvas.width, canvas.height) }, MULTI_THREADED);
+
+//Store a copy of future at this point in time.
+var expectedFuture:Future<ByteArray> = future;
+
+//Add a listener for later.
+future.onComplete(function(bytes:ByteArray):Void {
+   //If another thread was created after this one, don't draw anything.
+   if(future != expectedFuture) {
+       return;
+   }
+   
+   //Draw the pixels to the canvas.
+   bytes.position = 0;
+   canvas.setPixels(new Rectangle(0, 0, canvas.width, canvas.height), bytes);
+   bytes.clear();
+});

View full changes

This event listener always runs on the main thread, meaning only the main thread ever updates canvas, which is super helpful for thread safety. I still check whether another thread was created, but that’s only to make sure I’m drawing the right image, not because there’s a risk of two being drawn at once.

And this time, I can show you an HTML5 demo! Thanks to the use of threads, the app responds instantly after every click.

I should probably also mention that I set Future.FutureWork.maxThreads = 2. This means you can have two threads running at once, but any more will have to wait. Click enough times in a row, and even fast patterns will become slow. Not because they themselves slowed down, but because they’re at the back of the line. The app has to finish calculating all the previous patterns first.

(If the problem isn’t obvious from the small demo, try fullscreen.)

This seems pretty impractical. Why would the app spend all this time calculating the old patterns when it knows it won’t display them? Well, the reason is that you can’t cancel a Future once started. For that, and for other advanced features, you want to use ThreadPool directly instead of indirectly.

Oh yeah, did I mention that Future is built on top of ThreadPool? Hang on while I go check. …Apparently I never mentioned it. Well, Future is built on top of ThreadPool. It tries to provide the same features in a more convenient way, but doesn’t provide all the features. Canceling jobs, sending progress updates,

Using ThreadPool

Thread pools are a common way to make threads more efficient. It takes time to start up and shut down a thread, so why not reuse it instead? Lime’s ThreadPool class follows this basic pattern, though it prioritizes cross-platform compatibility, thread safety, and ease of use over performance.

When using ThreadPool, you’ll also need to be aware of its parent class, WorkOutput, as that’s your ticket to thread-safe message transfer. You’ll receive a WorkOutput instance as an argument (with the benefit that it can’t become null unexpectedly), and it has all the methods you need for communication.

sendComplete() and sendError() convey that your job succeeded/failed. When you call one of them, ThreadPool dispatches onComplete or onError as appropriate, and then initiates the thread recycling process. Don’t call them if you aren’t done!

sendProgress() works differently: you can call it as much as you like, with whatever type of data you like. It has no special meaning other than what you come up with. Unsurprisingly, sendProgress() corresponds to onProgress.

generatePattern() only needs sendComplete(), at least for now.

-private function generatePattern(workArea:Rectangle):Void {
+private static function generatePattern(state: { module:ModuleBase, workArea:Rectangle }, output:WorkOutput):Void {
    //Allocate four bytes per pixel.
    var bytes:ByteArray = new ByteArray(
        Std.int(workArea.width) * Std.int(workArea.height));
    
    //Run getValue() for every pixel.
    for(y in Std.int(workArea.top)...Std.int(workArea.bottom)) {
        for(x in Std.int(workArea.left)...Std.int(workArea.right)) {
            //getValue() returns a value in the range [-1, 1], and we need
            //to convert to [0, 255].
            var value:Int = Std.int(128 + 128 * module.getValue(x, y, 0));
            
            if(value > 255) {
                value = 255;
            } else if(value < 0) {
                value = 0;
            }
            
            //Store it as a color.
            bytes.writeInt(value << 16 | value << 8 | value);
        }
    }
    
-   //Draw the pixels to the canvas.
-   bytes.position = 0;
-   canvas.setPixels(workArea, bytes);
-   bytes.clear();
+   output.sendComplete(bytes, [bytes]);
}

Hmm, what’s up with “sendComplete(bytes, [bytes])“? Looks kind of redundant.

Well, each of the “send” functions takes an optional array argument that improves performance in HTML5. It’s great for transferring ByteArrays and similar packed data containers, but be aware that these containers will become totally unusable. That’s no problem at the end of the function, but be careful if using this with sendProgress().

With generatePattern() updated, the next step to initialize my ThreadPool.

//minThreads = 1, maxThreads = 1.
threadPool = new ThreadPool(1, 1, MULTI_THREADED);
threadPool.onComplete.add(function(bytes:ByteArray):Void {
    //Draw the pixels to the canvas.
    bytes.position = 0;
    canvas.setPixels(new Rectangle(0, 0, canvas.width, canvas.height), bytes);
    bytes.clear();
});

This time, I didn’t include a “latest thread” check. Instead, I plan to cancel old jobs, ensuring that they never dispatch an onComplete event at all.

-generatePattern(new Rectangle(0, 0, canvas.width, canvas.height));
+threadPool.cancelJob(jobID);
+jobID = threadPool.run(generatePattern, { module: module, workArea: new Rectangle(0, 0, canvas.width, canvas.height) });

This works well enough in the simplest case, but the full app actually isn’t this simple. The full app actually has several classes listening for events, and they all receive each other’s events. To solve this, they each have to filter.

Allow me to direct your attention to ThreadPool.activeJob. This variable is made available specifically during onComplete, onError, or onProgress events, and it tells you where the event came from.

threadPool.onComplete.add(function(bytes:ByteArray):Void {
+   if(threadPool.activeJob.id != jobID) {
+       return;
+   }
+   
    //Draw the pixels to the canvas.
    bytes.position = 0;
    canvas.setPixels(new Rectangle(0, 0, canvas.width, canvas.height), bytes);
    bytes.clear();
});

View full changes

Now, let’s see how the demo looks.

It turns out, setting maxThreads = 1 was a bad idea. Even calling cancelJob() isn’t enough: the app still waits to finish the current job before starting the next. (As before, viewing in fullscreen may make the problem more obvious.)

When a function has already started, cancelJob() does two things: (1) it bans the function call from dispatching events, and (2) it politely encourages the function to exit. There’s no way to force it to stop, so polite requests are all we get. If only generatePattern() was more cooperative.

Green/virtual threads

Green threads are what happens when you want thread-like behavior in a single-threaded environment. (“Virtual threads” can mean the same thing, but Java seems to be claiming the term for something else.)

As it happens, it was JavaScript’s definition of “async” that gave me the idea for this feature. JavaScript’s async keyword runs a function right on the main thread, but sometimes puts that function on pause to let other functions run. Only one thing ever runs at once, but since they take turns, it still makes sense to call them “asynchronous” or “concurrent.”

Most platforms don’t support anything like the async keyword, but we can imitate the behavior by exiting the function and starting it again later. Doesn’t sound very convenient, but unlike some things I tried, it’s simple, it’s reliable, and it works on every platform.

Exiting and restarting forms the basis for Lime’s green threads: instead of running a function on a background thread, run a small bit of that function each frame. The function is responsible for returning after a brief period, because if it takes too long the app won’t be able to draw the next frame in time. Then ThreadPool or FutureWork is responsible for scheduling it again, so it can continue. This behavior is also known as “cooperative multitasking” – multitasking made possible by functions voluntarily passing control to one another.

Here’s an outline for a cooperative function.

  1. The first time the function is called, it performs initialization and does a little work.
  2. By the end of the call, it stores its progress for later.
  3. When the function is called again, it checks for stored progress and determines that this isn’t the first call. Using this stored data, it continues from where it left off, doing a little more work. Then it stores the new data and exits again.
  4. Step 3 repeats until the function detects an end point. Then it calls sendComplete() or (if using Future) returns a non-null value.
  5. ThreadPool or FutureWork stops calling the function, and dispatches the onComplete event.

This leaves the question of where you should store that data. In single-threaded mode, you can put it wherever you like. However, this type of cooperation is also useful in multi-threaded mode so that functions can be canceled, and storing data in class variables isn’t always thread safe. Instead, I recommend using the state argument. Which is, incidentally, why I like to call it “state.” It provides the initial input and stores progress.

Typically, state will have some mandatory values (supplied by the caller) and some optional ones (initialized and updated by the function itself). If the optional ones are missing, that indicates it’s the first iteration.

-private static function generatePattern(state: { module:ModuleBase, workArea:Rectangle }, output:WorkOutput):Void {
+private static function generatePattern(state: { module:ModuleBase, workArea:Rectangle, ?y:Int, ?bytes:ByteArray }, output:WorkOutput):Void {
-   //Allocate four bytes per pixel.
-   var bytes:ByteArray = new ByteArray(
-       Std.int(workArea.width) * Std.int(workArea.height));
+   var bytes:ByteArray = state.bytes;
+   
+   //If it's the first iteration, initialize the optional values.
+   if(bytes == null) {
+       //Allocate four bytes per pixel.
+       state.bytes = bytes = new ByteArray(
+           Std.int(workArea.width) * Std.int(workArea.height));
+       
+       state.y = Std.int(workArea.top);
+   }
+   
+   //Each iteration, determine how much work to do.
+   var endY:Int = state.y + (output.mode == MULTI_THREADED ? 50 : 5);
+   if(endY > Std.int(workArea.bottom)) {
+       endY = Std.int(workArea.bottom);
+   }
    
    //Run getValue() for every pixel.
-    for(y in Std.int(workArea.top)...Std.int(workArea.bottom)) {
+   for(y in state.y...endY) {
        for(x in Std.int(workArea.left)...Std.int(workArea.right)) {
            //getValue() returns a value in the range [-1, 1], and we need
            //to convert to [0, 255].
            var value:Int = Std.int(128 + 128 * module.getValue(x, y, 0));
            
            if(value > 255) {
                value = 255;
            } else if(value < 0) {
                value = 0;
            }
            
            //Store it as a color.
            bytes.writeInt(value << 16 | value << 8 | value);
        }
    }
    
+   //Save progress.
+   state.y = endY;
+   
+   //Don't call sendComplete() until actually done.
+   if(state.y >= Std.int(workArea.bottom)) {
        output.sendComplete(bytes, [bytes]);
+   }
}

Note that I do more work per iteration in multi-threaded mode. There’s no need to return too often; just often enough to exit if the job’s been canceled. It also incurs overhead in HTML5, so it’s best not to overdo it.

Single-threaded mode is the polar opposite. There’s minimal overhead, and you get better timing if the function is very short. Ideally, short enough to run 5+ times a frame with time left over. On a slow computer, it’ll automatically reduce the number of times per frame to prevent lag.

Next, I tell ThreadPool to use single-threaded mode, and I specify a workLoad of 3/4. This value indicates what fraction of the main thread’s processing power should be spent on this ThreadPool. I’ve elected to take up 75% of it, leaving 25% for other tasks. Since I know those other tasks aren’t very intense, this is plenty.

-threadPool = new ThreadPool(1, 1, MULTI_THREADED);
+threadPool = new ThreadPool(1, 1, SINGLE_THREADED, 3/4);

View full changes

Caution: reduce this number if creating multiple single-threaded ThreadPools. If two pools each have a workLoad of 3/4, then they’ll take up 150% of the allocated time per frame, and your app will slow down by (at least) 50%. Instead, try to keep the combined workLoad under 1.

In any case, it’s time for another copy of the demo. Since we’re nearing the end, I also went ahead and implemented progress events. Now you can watch the progress in (closer to) real time.

These changes also benefit multi-threaded mode, so I created another multi-threaded version for comparison. With progress events, you can now see the slight pause when it spins up a new web worker (which isn’t that often, since it keeps two of them running).

(For comparison, here they both are in fullscreen: green threads, web workers.)

I don’t know, I like them both. Green threads have the benefit of being lighter weight, while web workers have the benefit of being real threads, meaning you could run eight in parallel without slowing the main thread.

My advice? Write code that works both ways, as shown in this guide. Keep your options open, since the configuration that works best for a small app may not be what works best for a big one. Good luck out there!