most of my writing Zola 2025-12-25T00:00:00+00:00 https://elijah.run/blog/atom.xml 2025-12-25T00:00:00+00:00 2025-12-25T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/media-monotasking/ <p>I, like many people in 2025, have a looooong list of games, movies, tv, and books that I want to enjoy sometime between now and death.</p> <p>Some tools I have created to deal with this problem include a <a href="/backlog/">backlog</a>, and a <a href="/whats-good/">monthly reflection on what I enjoyed</a>. These both help me focus; I either finish a piece of media or I give up entirely. None of this "Oh I started five books and two shows and a dozen games this month" no: either finish what you started, or basnish it to the shadow realm; there is too much good stuff to drag a bunch of half finish <em>stuff</em> around too.</p> <p>Between work, taking care of the kids, walking the dog, keeping the house tidy, eating, sleeping, keeping up with friends, and exercising I have ~1-2 hours of "me time" per day to spend on <em>whatever</em>. In the past I woud mindlessly scroll for <em>hours</em> after work and on the weekends, but I had to <a href="/whats-good/2024/10/#tech-screenzen">lock that shit down</a> pretty quick after having a kid; life's too short to waste perfectly good free time.</p> <p>Ok here's the thing: between What's Good and The Backlog I feel <em>ok</em> about my media focus, but I think I can do better. I still find myself starting games I never finish, starting a book on the 1st of the month that's barely half way done by the end, and mindlessly scrolling the one app I didn't lock down yet-- am I really scrolling Libby?</p> <h1 id="the-proposal">The Proposal</h1> <blockquote> <p>Do One Thing at a Time.</p> </blockquote> <p>If you start reading a book, watching a show, listening to a podcast, <em>whatever</em>, that's the only media you consume until you either complete it or abandon that forever.</p> <p>Not one <strong>book</strong> at a time, or one <strong>movie</strong> at a time, but one <em>thing</em> at a time, across all media.</p> <ul> <li>Start a movie? You cannot stop half way through and listen to a podcast while you do dishes.</li> <li>Start a videogame? Git gud or you're gonna get really bored.</li> <li>Start reading a book? Get cozy af because you're locked in.</li> </ul> <p>Of course you can go about your life eating and sleeping and working, but the only <em>media</em> you can consume is <em>the thing</em> until it is done or you call it quits.</p> <p>Done is subjective, but you get the idea.</p> <ul> <li>Roll credits on a movie.</li> <li>Finish the last chapter in a book.</li> <li>Finish a season of television.</li> </ul> <h2 id="it-s-not-quite-that-simple">It's not quite that simple</h2> <p>I mean you totally could just do that, but you'd spend a lot of time driving in silence because you started playing Elden Ring and it's good but you're <em>really bad</em> at souls-likes.</p> <p>And honestly truly doing one thing at a time is like the <a href="https://en.wikipedia.org/wiki/Jain_vegetarianism">Jain vegetarianism</a> of media diets, it's the most intense version.</p> <p>I propose, and will personally strive, for a more moderate version which is going to use the following categories to guide our media decisions:</p> <ul> <li>Active Media: (Videogames) media that requires your participation. <ul> <li>Solving puzzles, shooting zombies, stuff like that.</li> </ul> </li> <li>Active Visual Media: (Books) media that requires your attention. <ul> <li>You don't need to solve puzzles, but you do need to flip pages.</li> </ul> </li> <li>Passive Visual Media: (Movies/TV/Videos) benefits from attention. <ul> <li>The media will move on without you, but you should be keeping up.</li> </ul> </li> <li>Passive Media: (Music/Podcasts/Audiobooks) you can put it on in the background. <ul> <li>This is "Doing the Dishes" stuff.</li> </ul> </li> </ul> <p><img src="https://elijah.run/blog/media-monotasking/flow-diagram.jpg" alt="A diagram showing the decision tree of what media to consume when" /></p> <blockquote> <p>I'm going to call these "Media Moods" because "Media Situations" feels clunky. You totally <em>can</em> do these based on mood, it's just simpler to describe situations that limit what you <em>can</em> do rather than the more fuzzy <em>what you want to do</em>.</p> </blockquote> <p>For each of these moods I would pick a single piece of media to engage with. Any time you are in that mood, you are reading/watching/listening to that thing, until that thing is done.</p> <p>For example:</p> <ul> <li>While I am rowing I finish a movie (passive visual) I started yesterday.</li> <li>Later I am cleaning up the house, so I finish a podcast (passive) I stared that morning.</li> <li>In the evening I play a puzzle game (active) I have been working on for a few days.</li> <li>Until I decide my brain is tired and I want to do something else, so I read this month's book (active visual).</li> </ul> <blockquote> <p>Yes, I am drawing an arbitrary line between "Active" and "Active Visual" for convenience; you could totally merge those to simplify things even further.</p> </blockquote> <p>This means that you have <em>some</em> choice of media based on your circumstances, but there is still a prescriptive framework around what you <em>should</em> be doing media-wise to avoid getting easily distracted.</p> <h2 id="this-is-not-for-everybody">This is not for everybody</h2> <p>It goes without saying, this is a framework for burning down a backlog. Even more so, it is for people who have very little time to burn that backlog, and they need to squeeze every minute out of their days to make any progress on that project.</p> <p>If you don't have a backlog, or you already have a healthy way of handling your backlog, this is probably not for you.</p> <p>It's also not compatible with like... roommates. You should definitely make an exception to whatever your media backlog dictates if it means having a good time with friends and/or family. Don't be weird about it please.</p> <h2 id="this-is-a-thing-i-am-trying">This is a thing I am trying</h2> <p>It would be awesome if I could say this is something I've been doing for a year and it's worked flawlessly -- but no. This is just a thing I am starting and trying in hopes that it will improve my backlog burn rate.</p> <p>And if it doesn't we'll learn something too. Either way, I'll keep you posted.</p> 2025-07-17T00:00:00+00:00 2025-07-17T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/how-i-use-typst/ <p>This week I re-wrote <a href="/Elijah%20Voigt.pdf">my resume</a> in <a href="https://typst.app/">typst</a>.</p> <p><a href="https://elijah.run/blog/how-i-use-typst/old.pdf">Old</a>: <img src="https://elijah.run/blog/how-i-use-typst/old.png" alt="old resume" /></p> <p><a href="https://elijah.run/blog/how-i-use-typst/new.pdf">New</a>: <img src="https://elijah.run/blog/how-i-use-typst/new.png" alt="new resume" /></p> <p><em>Can you tell the difference? No? Excellent!</em></p> <p>Since graduating college I don't write a lot of LaTeX, but so far I prefer typst!</p> <h1 id="anguished-typst-is-not-just-a-better-latex">😧 typst is not (just) a better LaTeX</h1> <p>I posted some first impressions on <a href="https://bsky.app/profile/elijah.run">Bluesky</a>:</p> <blockquote class="bluesky-embed" data-bluesky-uri="at://did:plc:srziwhupeikb2bcnyndug2ga/app.bsky.feed.post/3ltx7vsqt3s2m" data-bluesky-cid="bafyreid2ol6s3bji2jkyiaxnwdgylql7qja72wojpnno5y2eohxahggswu" data-bluesky-embed-color-mode="system"><p lang="en">LaTeX being compatible with TeX makes it feel like the C++ of typesetting; a rich ecosystem with lots of bloat. <p>typst not being constrained in that way makes it feel like the Zig of typesetting; smaller ecosystem but a much smoother writing experience.</p>— Elijah Voigt (<a href="https://bsky.app/profile/did:plc:srziwhupeikb2bcnyndug2ga?ref_src=embed">@elijah.run</a>) <a href="https://bsky.app/profile/did:plc:srziwhupeikb2bcnyndug2ga/post/3ltx7vsqt3s2m?ref_src=embed">July 14, 2025 at 1:25 PM</a></blockquote><script async src="https://embed.bsky.app/static/embed.js" charset="utf-8"></script></p> <p>To elaborate: typst doesn't feel like an <em>improvement</em> on LaTeX, it feels like a from-the-ground-up typsestter that learns from the past uhh -- wait <a href="https://en.wikipedia.org/wiki/LaTeX">LaTeX</a> came out in 1984? and <a href="https://en.wikipedia.org/wiki/TeX">TeX</a> came out in 1978?? So yeah it learns from the past 41+ years.</p> <h1 id="adult-computer-typst-is-just-programming-for-me">🧑‍💻 typst is just programming (for me)</h1> <p>To be fair, so is LaTeX. If you give me a programming language I'm gonna program.</p> <p>The typst docs make it seem like it's <code>markdown</code> with functions, which is somewhat true:</p> <pre data-lang="typ" style="background-color:#191919;color:#f8f8f2;" class="language-typ "><code class="language-typ" data-lang="typ"><span>== this is a title </span><span> </span><span>this is a paragraph. </span><span> </span><span>- this </span><span>- is </span><span>- a </span><span>- list </span></code></pre> <p>But my resume is more... function heavy than advertised. For example, this:</p> <p><img src="https://elijah.run/blog/how-i-use-typst/example.png" alt="example styled section" /></p> <p>Came from this function:</p> <pre data-lang="typ" style="background-color:#191919;color:#f8f8f2;" class="language-typ "><code class="language-typ" data-lang="typ"><span>// Function for printing a role </span><span>#let role(org: none, location: none, title: none, date: none, description: ntypone) = { </span><span>[ </span><span> #set text(weight: &quot;bold&quot;) </span><span> #org </span><span> #set text(weight: &quot;regular&quot;) </span><span> ( </span><span> #text(13pt, location) </span><span> ) </span><span> #h(1fr) </span><span> #set text(weight: &quot;bold&quot;) </span><span> #title </span><span> #set text(weight: &quot;regular&quot;) </span><span> #date </span><span>]; </span><span>text(12pt, description); </span><span>}; </span></code></pre> <p>Invoked like this:</p> <pre data-lang="typ" style="background-color:#191919;color:#f8f8f2;" class="language-typ "><code class="language-typ" data-lang="typ"><span>#role( </span><span> org: [CoreOS], </span><span> location: [San Francisco, CA/Remote], </span><span> title: [Documentarian], </span><span> date: [09/2016 - 05/2017], </span><span> description: [ </span><span> - #xp([Migration Docs.], [Reviewed, tested, and contributed to public facing Kubernetes developer docs.]) </span><span> ] </span><span>) </span></code></pre> <p>The result looks good, but it certainly doesn't "feel like markdown"... that's OK! I know how to program and the functions are simple!</p> <pre data-lang="typ" style="background-color:#191919;color:#f8f8f2;" class="language-typ "><code class="language-typ" data-lang="typ"><span>// function definition: </span><span>#myfn(a, b, c) = { </span><span> [ </span><span> #set foo(bar=&quot;thing&quot;) </span><span> #a </span><span> #set foo(other=&quot;stuff&quot;) </span><span> #b -- #c </span><span> ] </span><span>} </span><span> </span><span>// called like so: </span><span>#myfn([asdf], [abcd123], [hello world]) </span><span> </span><span>// produces: </span><span>// asdf abcd123 -- hello world </span></code></pre> <h1 id="rabbit2-typst-is-fast">🐇 typst is fast</h1> <p>Which is... not surprising. Not just because it's written in Rust and I'm a voluntary Rust shill -- any new typesetter would probably be an improvement! Any time you do a full rewrite, and ignore legacy usage, you're gonna go sonic-levels of fast. And again... LaTeX is 41 years old; there's a lot of cruft in that ecosystem.</p> <p>LaTeX's output looks <em>gorgeous</em> but it is SLOW and BULKY. To build my old resume I needed:</p> <ul> <li>A 12GB+ docker container.</li> <li>Bunch of style files I copied from a friend 10 years ago.</li> <li>A bunch of build phases I frankly did not understand.</li> </ul> <p>This was my Makefile which was <em>as simple as I could get it</em>:</p> <pre data-lang="make" style="background-color:#191919;color:#f8f8f2;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#e9fdac;">TARGET</span><span style="color:#ff5e5e;">=</span><span style="color:#fbe3bf;">resume </span><span> </span><span style="color:#8cdaff;">default</span><span style="color:#ff5e5e;">: </span><span style="color:#fbe3bf;">pdf </span><span> </span><span style="color:#8cdaff;">dvi</span><span style="color:#ff5e5e;">: </span><span style="font-style:italic;color:#ff5e5e;">${</span><span style="font-style:italic;color:#fc9354;">TARGET</span><span style="font-style:italic;color:#ff5e5e;">}</span><span style="color:#fbe3bf;">.tex </span><span> </span><span style="color:#e9fdac;">latex </span><span style="font-style:italic;color:#ff5e5e;">${</span><span style="font-style:italic;color:#fc9354;">TARGET</span><span style="font-style:italic;color:#ff5e5e;">}</span><span>.tex </span><span> </span><span style="color:#8cdaff;">ps</span><span style="color:#ff5e5e;">: </span><span style="color:#fbe3bf;">dvi </span><span> </span><span style="font-style:italic;color:#ff5e5e;">${</span><span style="font-style:italic;color:#fc9354;">DOCKER</span><span style="font-style:italic;color:#ff5e5e;">} </span><span style="color:#e9fdac;">dvips</span><span style="font-style:italic;color:#fc9354;"> -R -Poutline -t</span><span> letter </span><span style="font-style:italic;color:#ff5e5e;">${</span><span style="font-style:italic;color:#fc9354;">TARGET</span><span style="font-style:italic;color:#ff5e5e;">}</span><span>.dvi</span><span style="font-style:italic;color:#fc9354;"> -o </span><span style="font-style:italic;color:#ff5e5e;">${</span><span style="font-style:italic;color:#fc9354;">TARGET</span><span style="font-style:italic;color:#ff5e5e;">}</span><span>.ps </span><span> </span><span> </span><span style="color:#8cdaff;">pdf</span><span style="color:#ff5e5e;">: </span><span style="color:#fbe3bf;">ps </span><span> </span><span style="color:#e9fdac;">ps2pdf </span><span style="font-style:italic;color:#ff5e5e;">${</span><span style="font-style:italic;color:#fc9354;">TARGET</span><span style="font-style:italic;color:#ff5e5e;">}</span><span>.ps resume.pdf </span><span> </span><span style="color:#e9fdac;">ps2pdf </span><span style="font-style:italic;color:#ff5e5e;">${</span><span style="font-style:italic;color:#fc9354;">TARGET</span><span style="font-style:italic;color:#ff5e5e;">}</span><span>.ps </span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">Elijah Caine McDade Voigt resume.pdf</span><span style="color:#ffffff;">&quot; </span><span> </span><span style="color:#e9fdac;">ps2pdf </span><span style="font-style:italic;color:#ff5e5e;">${</span><span style="font-style:italic;color:#fc9354;">TARGET</span><span style="font-style:italic;color:#ff5e5e;">}</span><span>.ps </span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">Elijah Caine M. Voigt resume.pdf</span><span style="color:#ffffff;">&quot; </span><span> </span><span style="color:#8cdaff;">clean</span><span style="color:#ff5e5e;">: </span><span> </span><span style="color:#e9fdac;">rm </span><span style="color:#ff5e5e;">*</span><span>.aux </span><span> </span><span style="color:#e9fdac;">rm </span><span style="color:#ff5e5e;">*</span><span>.log </span><span> </span><span style="color:#e9fdac;">rm </span><span style="color:#ff5e5e;">*</span><span>.dvi </span><span> </span><span style="color:#e9fdac;">rm </span><span style="color:#ff5e5e;">*</span><span>.ps </span><span> </span><span style="color:#8cdaff;">veryclean</span><span style="color:#ff5e5e;">: </span><span> </span><span style="color:#e9fdac;">rm </span><span style="color:#ff5e5e;">*</span><span>.aux </span><span> </span><span style="color:#e9fdac;">rm </span><span style="color:#ff5e5e;">*</span><span>.log </span><span> </span><span style="color:#e9fdac;">rm </span><span style="color:#ff5e5e;">*</span><span>.dvi </span><span> </span><span style="color:#e9fdac;">rm </span><span style="color:#ff5e5e;">*</span><span>.ps </span><span> </span><span style="color:#e9fdac;">rm </span><span style="color:#ff5e5e;">*</span><span>.pdf </span><span> </span><span style="color:#8cdaff;">shell</span><span style="color:#ff5e5e;">: </span><span> </span><span style="font-style:italic;color:#ff5e5e;">${</span><span style="font-style:italic;color:#fc9354;">DOCKER</span><span style="font-style:italic;color:#ff5e5e;">} </span><span style="color:#e9fdac;">/bin/sh </span></code></pre> <p>What is this <code>dvips</code> program? What's a <code>dvi</code> file? And <code>.aux</code>?? Why do I need to build a <code>.ps</code> and a <code>.pdf</code>??</p> <p>Anyway, LaTeX is heavy and (for obvious reasons) typst is relatively light weight.</p> <p>For comparison here's how I build my current resume:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ typst compile resume.typ </span></code></pre> <p>My dependencies are... <code>typst</code> and the <code>FreeMono</code> font.</p> <h1 id="monocle-face-closing-thoughts">🧐 Closing Thoughts</h1> <p>But like I said, I don't write a lot of LaTeX so why re-write my resume -- my only actively maintained LaTeX doc -- in a new typesetter?</p> <p>Basically: I <em>dread</em> updating my resume <em>because</em> LaTeX is so heavy. Every time I think to add it or tweak it or god forsake make a style change I find about a dozen reasons <em>not</em> to sink <em>hours</em> into that task.</p> <p>I'm still in the honeymoon phase, but I feel a lot more <em>excited</em> to write with typst. Maybe I'll update my resume more than once a year!</p> 2022-05-17T00:00:00+00:00 2022-05-17T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/rust-default-from/ <p>I've been writing Rust off and on since 2014 and consistently since 2019 when I got into Rust Game Development. Once I started writing more Rust code I noticed it wasn't just more lines of code, but each part of the code was more verbose.</p> <p>Coming from Python where ideas <em>tend</em> to be pretty succinct, Rust forced you to spell everything out in intense detail. Of course you <em>got</em> something for that verbosity -- "if it compiles, it probably works" -- but my hands were getting tired. There has to be a better way!</p> <h2 id="verbose-structs">Verbose Structs</h2> <p>There is a better way, but let's clarify what the problem <em>is</em> exactly. Take this example of a real struct from the <a href="https://bevyengine.org/">bevy game engine</a> <code>PbrBundle</code>:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#ff5e5e;">pub </span><span style="font-style:italic;color:#fbdfb5;">type </span><span>PbrBundle </span><span style="color:#ff5e5e;">= </span><span>MaterialMeshBundle&lt;StandardMaterial&gt;; </span></code></pre> <p>It's a type-alias to a <code>MaterialMeshBundle</code>. Let's check that out:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#ff5e5e;">pub </span><span style="font-style:italic;color:#fbdfb5;">struct </span><span>MaterialMeshBundle&lt;M&gt; where </span><span> M: SpecializedMaterial, </span><span>{ </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">mesh</span><span>: Handle&lt;Mesh&gt;, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">material</span><span>: Handle&lt;M&gt;, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">transform</span><span>: Transform, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">global_transform</span><span>: GlobalTransform, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">visibility</span><span>: Visibility, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">computed_visibility</span><span>: ComputedVisibility, </span><span>} </span></code></pre> <p>Wow that's a fair number of struct members.</p> <p>If you wanted to create a <code>PbrBundle</code> by hand it would be a <em>tedious</em> process.</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">let</span><span> my_pbr </span><span style="color:#ff5e5e;">=</span><span> PbrBundle { </span><span> mesh: </span><span style="color:#6699cc;">get_mesh_handle</span><span>(), </span><span> material: </span><span style="color:#6699cc;">get_material_handle</span><span>(), </span><span> transform: Transform { </span><span> translation: Vec3::new(), </span><span> rotation: Quat::new(), </span><span> scale: Vec3::new(), </span><span> }, </span><span> global_transform: GlobalTransform { </span><span> translation: Vec3::new(), </span><span> rotation: Quat::new(), </span><span> scale: Vec3::new(), </span><span> }, </span><span> visibility: Visibility { </span><span> is_visibile: </span><span style="color:#ff8942;">true</span><span>, </span><span> }, </span><span> computed_visibility: ComputedVisibility { </span><span> is_visibile: </span><span style="color:#ff8942;">true</span><span>, </span><span> }, </span><span>} </span></code></pre> <p>Wow my fingers are already tired.</p> <p>Now, assuming you don't know the punchline, you're probably thinking: <em>Just use a constructor!</em></p> <p>That solves the use-case where the author of the code has a constructor for my use-case, something like this:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">let</span><span> my_pbr </span><span style="color:#ff5e5e;">= </span><span>PbrBundle::new(mesh, material); </span><span style="color:#6d6d6d;">// Default Transform and Visibilty </span></code></pre> <p>But what if I want a <em>mostly</em> "default" <code>PbrBundle</code>, with say <code>is_visible = false</code>? Or I want to add a Transform with a custom scale but a default translation and rotation?</p> <p>Basically, what if I am <em>picky</em> and want the <em>flexibility</em> of struct initialization with the <em>convenience</em> of constructor methods?</p> <h2 id="default-fill-in-the-blanks">Default: Fill in the blanks</h2> <p>This is totally supported thanks to <a href="https://doc.rust-lang.org/std/default/trait.Default.html">Rust's "Default" Trait</a>.</p> <p>The usage is something like this from the previous example:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">let</span><span> my_pbr </span><span style="color:#ff5e5e;">=</span><span> PbrBundle { </span><span> mesh: </span><span style="color:#6699cc;">get_mesh_handle</span><span>(), </span><span> material: </span><span style="color:#6699cc;">get_material_bundle</span><span>(), </span><span> transform: Transform { </span><span> scale: Vec3::new(</span><span style="color:#fdb082;">2.0</span><span>, </span><span style="color:#fdb082;">2.0</span><span>, </span><span style="color:#fdb082;">2.0</span><span>), </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span> }, </span><span> visibility: Visibilty { </span><span> is_visible: </span><span style="color:#ff8942;">true</span><span>, </span><span> }, </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span>} </span></code></pre> <p>This example shows creating a <code>PbrBundle</code> with a custom <code>mesh</code>, <code>material</code>, and <code>scale</code>, but everything else is a "Default" value.</p> <p>While this flexibility is totally possible with constructors, it would require some creativity, maybe something like this?</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">let</span><span> pbr_bundle </span><span style="color:#ff5e5e;">= </span><span>PbrBundle::new(mesh, material) </span><span> .</span><span style="color:#6699cc;">with_scale</span><span>(Vec3::new(</span><span style="color:#fdb082;">2.0</span><span>, </span><span style="color:#fdb082;">2.0</span><span>, </span><span style="color:#fdb082;">2.0</span><span>)) </span><span> .</span><span style="color:#6699cc;">with_visibility</span><span>(</span><span style="color:#ff8942;">true</span><span>); </span></code></pre> <p>This is fine, but it is a <strong>lot</strong> of toil for the author. They need to add and maintain a method for each element of their nested struct, document those methods, probably write tests, and all to accomplish the goal of a "Fill in the rest for me" API.</p> <p>One nice part of <code>Default</code> is it can be automagically added to any struct whose members implement it via <code>#[derive(Default)]</code>. This means you get that "Fill in the rest for me" interface for free!</p> <h2 id="close-but-distinct-types">Close but Distinct Types</h2> <p>Another pain-point I found in Rust was converting between similar but distinct types. Unlike my last language Python, which was <em>very</em> forgiving about types (to a fault), Rust requires very precise type expressions.</p> <p>Let's take this example:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#6d6d6d;">// Base Engine Color </span><span>#[</span><span style="color:#e9fdac;">derive</span><span>(Default, Debug, PartialEq)] </span><span style="font-style:italic;color:#fbdfb5;">struct </span><span>Color { </span><span> </span><span style="color:#e9fdac;">red</span><span>: </span><span style="font-style:italic;color:#fbdfb5;">f32</span><span>, </span><span> </span><span style="color:#e9fdac;">green</span><span>: </span><span style="font-style:italic;color:#fbdfb5;">f32</span><span>, </span><span> </span><span style="color:#e9fdac;">blue</span><span>: </span><span style="font-style:italic;color:#fbdfb5;">f32</span><span>, </span><span>} </span><span> </span><span style="color:#6d6d6d;">/// Color for UI elements </span><span>#[</span><span style="color:#e9fdac;">derive</span><span>(Default, Debug, PartialEq)] </span><span style="font-style:italic;color:#fbdfb5;">struct </span><span>UiColor(Color); </span><span> </span><span style="color:#6d6d6d;">/// Just a demo function, not sure if this is useful... </span><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">color_rotate</span><span>(</span><span style="font-style:italic;color:#fc9354;">color</span><span>: Color) -&gt; Color { </span><span> Color { </span><span> red: color.green, </span><span> green: color.blue, </span><span> blue: color.red, </span><span> } </span><span>} </span><span> </span><span style="color:#6d6d6d;">// Does not compile! </span><span style="color:#6d6d6d;">// E0308: mismatched types expected struct `Color`, found struct `UiColor` </span><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">main</span><span>() { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> a </span><span style="color:#ff5e5e;">=</span><span> UiColor(Color { </span><span> red: </span><span style="color:#fdb082;">0.5</span><span>, </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span> }); </span><span> </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> b </span><span style="color:#ff5e5e;">= </span><span style="color:#6699cc;">color_rotate</span><span>(a); </span><span>} </span></code></pre> <p>Here we have a <code>UiColor</code> struct that wraps our base <code>Color</code> struct. We want to use a method made for <code>Color</code> values but we get an error that the compiler is expecting a <code>Color</code> but we gave it a <code>UiColor</code>! Come on Rust, just look inside the box!</p> <p>We can work around this issue like so:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">main</span><span>() { </span><span> </span><span style="color:#6d6d6d;">// ... </span><span> </span><span> </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> b </span><span style="color:#ff5e5e;">=</span><span> UiColor(</span><span style="color:#6699cc;">color_rotate</span><span>(a.</span><span style="color:#fdb082;">0</span><span>)); </span><span>} </span></code></pre> <p>Which passes the inside of <code>a</code> to <code>color_rotate</code> and then wraps the return in a new <code>UiColor</code> struct. This works, but it's hard to read and more importantly it requires a keep our API in our head to write <em>any</em> code.</p> <h2 id="from-and-into-simple-type-coercion">From and <code>into</code>: Simple type coercion</h2> <p>The solution is to use the <a href="https://doc.rust-lang.org/std/convert/trait.From.html">"From"</a> and trait which provides the <code>into()</code> method.</p> <p>Extending the above example, we can implement <code>From Color -&gt; UiColor</code> and <code>From UiColor -&gt; Color</code> like so:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">impl </span><span style="color:#fbe3bf;">From</span><span>&lt;Color&gt; </span><span style="color:#ff5e5e;">for </span><span>UiColor { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">from</span><span>(</span><span style="font-style:italic;color:#fc9354;">input</span><span>: Color) -&gt; UiColor { </span><span> UiColor(input) </span><span> } </span><span>} </span><span> </span><span style="font-style:italic;color:#fbdfb5;">impl </span><span style="color:#fbe3bf;">From</span><span>&lt;UiColor&gt; </span><span style="color:#ff5e5e;">for </span><span>Color { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">from</span><span>(</span><span style="font-style:italic;color:#fc9354;">input</span><span>: UiColor) -&gt; Color { </span><span> input.</span><span style="color:#fdb082;">0 </span><span> } </span><span>} </span></code></pre> <p>Unfortunately we can't do anything like <code>#[derive(From&lt;UiColor&gt;)]</code> (yet?) but implementing these traits is fairly straight forward and <em>very</em> powerful.</p> <p>Here we can see our <code>main</code> function is fixed with passing assertions.</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">main</span><span>() { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> a </span><span style="color:#ff5e5e;">=</span><span> UiColor(Color { </span><span> red: </span><span style="color:#fdb082;">0.5</span><span>, </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span> }); </span><span> assert_eq!(a, UiColor(Color { red: </span><span style="color:#fdb082;">0.5</span><span>, green: </span><span style="color:#fdb082;">0.0</span><span>, blue: </span><span style="color:#fdb082;">0.0 </span><span>})); </span><span> </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> b: UiColor </span><span style="color:#ff5e5e;">= </span><span style="color:#6699cc;">color_rotate</span><span>(a.</span><span style="color:#6699cc;">into</span><span>()).</span><span style="color:#6699cc;">into</span><span>(); </span><span> assert_eq!(b, UiColor(Color { red: </span><span style="color:#fdb082;">0.0</span><span>, green: </span><span style="color:#fdb082;">0.0</span><span>, blue: </span><span style="color:#fdb082;">0.5 </span><span>})); </span><span>} </span></code></pre> <p>Rust was not only able to cast our <code>UiColor</code> to a <code>Color</code> in the call to <code>color_rotate</code> but we were able to coerce the result back to a <code>UiColor</code> by declaring the type of our <code>b</code> variable.</p> <p>Using <code>From</code> and <code>into()</code> is great because it allows you to ignore the specifics of the types you're working with while still getting the benefits of a strong type system. When you use it enough it can feel like parts of your code are "Duckly" typed, like Python and Ruby which have very ergonomic type interactions.</p> <h2 id="from-simple-to-complex">From Simple to Complex</h2> <p>Since learning about <code>From</code> started to abuse it to convert simplified types to more complex ones.</p> <p>Take for example this UI struct in Bevy:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#ff5e5e;">pub </span><span style="font-style:italic;color:#fbdfb5;">struct </span><span>NodeBundle { </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">node</span><span>: Node, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">style</span><span>: Style, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">color</span><span>: UiColor, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">image</span><span>: UiImage, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">focus_policy</span><span>: FocusPolicy, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">transform</span><span>: Transform, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">global_transform</span><span>: GlobalTransform, </span><span> </span><span style="color:#ff5e5e;">pub </span><span style="color:#e9fdac;">visibility</span><span>: Visibility, </span><span>} </span></code></pre> <p>On it's own this isn't bad, but if you write enough UI code it can get tedious. Both <code>Node</code> and <code>Style</code> are nested structs that have a lot of complexity -- <code>Style</code> is a struct with 21 members! -- so using <code>Default</code> won't cut it here.</p> <p>Instead I made a "dumbed down" version like this:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">struct </span><span>SimpleNodeBundle { </span><span> </span><span style="color:#e9fdac;">position</span><span>: SimplePosition, </span><span> </span><span style="color:#e9fdac;">color</span><span>: Color, </span><span> </span><span style="color:#e9fdac;">size</span><span>: Vec2, </span><span>} </span><span> </span><span style="font-style:italic;color:#fbdfb5;">enum </span><span>SimplePosition { </span><span> BottomLeft, </span><span> BottomRight, </span><span> TopLeft, </span><span> TopRight, </span><span>} </span></code></pre> <p>This is maybe <em>too</em> simple, but you can add the complexity you need down the line. The important part is that our <code>Simple</code> struct is you know... less complex than what it's going to map to.</p> <p>Now that we have a simple struct that we can use to quickly write out some UI elements.</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">let</span><span> my_ui_element </span><span style="color:#ff5e5e;">=</span><span> SimpleNodeBundle { </span><span> position: SimplePosition::TopRight, </span><span> size: Vec2 { x: </span><span style="color:#fdb082;">25.0</span><span>, y: </span><span style="color:#fdb082;">100.0 </span><span>}, </span><span> color: Color::</span><span style="color:#fdb082;">RED</span><span>, </span><span>} </span></code></pre> <p>On it's own though this is useless. Bevy doesn't know what a <code>SimpleNodeBundle</code> is, we need to convert this to the "lower level" struct it's replacing. We need to cast it up to a Bevy <code>NodeBundle</code> with an implementation of <code>From&lt;SimpleNodeBundle&gt; for NodeBundle</code>:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">impl </span><span style="color:#fbe3bf;">From</span><span>&lt;SimpleNodeBundle&gt; </span><span style="color:#ff5e5e;">for </span><span>NodeBundle { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">impl </span><span>from(input: SimpleNodeBundle) -&gt; NodeBundle { </span><span> NodeBundle { </span><span> color: UiColor(input.color), </span><span> style: Style { </span><span> position: input.position.</span><span style="color:#6699cc;">into</span><span>(), </span><span> size: input.size.</span><span style="color:#6699cc;">into</span><span>(), </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span> } </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span> } </span><span> } </span><span>} </span><span> </span><span style="color:#6d6d6d;">// NodeBundle&#39;s position is a Rect&lt;Val&gt; so we convert SimplePosition to Rect&lt;Val&gt; </span><span style="font-style:italic;color:#fbdfb5;">impl </span><span style="color:#fbe3bf;">From</span><span>&lt;SimplePosition&gt; </span><span style="color:#ff5e5e;">for </span><span>Rect&lt;Val&gt; { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">impl </span><span>from(input: SimplePosition) -&gt; Rect&lt;Val&gt; { </span><span> </span><span style="color:#ff5e5e;">use </span><span>SimplePosition::</span><span style="color:#ff5e5e;">*</span><span>; </span><span> </span><span style="color:#ff5e5e;">match</span><span> input { </span><span> BottomLeft </span><span style="color:#ff5e5e;">=&gt;</span><span> Rect { </span><span> bottom: </span><span style="color:#6699cc;">percent</span><span>(</span><span style="color:#fdb082;">0.0</span><span>), </span><span> left: </span><span style="color:#6699cc;">percent</span><span>(</span><span style="color:#fdb082;">0.0</span><span>), </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span> }, </span><span> BottomRight </span><span style="color:#ff5e5e;">=&gt;</span><span> Rect { </span><span> bottom: </span><span style="color:#6699cc;">percent</span><span>(</span><span style="color:#fdb082;">0.0</span><span>), </span><span> right: </span><span style="color:#6699cc;">percent</span><span>(</span><span style="color:#fdb082;">0.0</span><span>), </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span> }, </span><span> TopLeft </span><span style="color:#ff5e5e;">=&gt;</span><span> Rect { </span><span> top: </span><span style="color:#6699cc;">percent</span><span>(</span><span style="color:#fdb082;">0.0</span><span>), </span><span> left: </span><span style="color:#6699cc;">percent</span><span>(</span><span style="color:#fdb082;">0.0</span><span>), </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span> }, </span><span> TopRight </span><span style="color:#ff5e5e;">=&gt;</span><span> Rect { </span><span> top: </span><span style="color:#6699cc;">percent</span><span>(</span><span style="color:#fdb082;">0.0</span><span>), </span><span> right: </span><span style="color:#6699cc;">percent</span><span>(</span><span style="color:#fdb082;">0.0</span><span>), </span><span> </span><span style="color:#ff5e5e;">..</span><span style="color:#fbe3bf;">Default</span><span>::default() </span><span> }, </span><span> } </span><span> } </span><span>} </span><span> </span><span style="color:#6d6d6d;">// Similarly size is a Size&lt;Val&gt; but we have a Vec2, so we conver to the right type </span><span style="font-style:italic;color:#fbdfb5;">impl </span><span style="color:#fbe3bf;">From</span><span>&lt;Vec2&gt; </span><span style="color:#ff5e5e;">for </span><span>Size&lt;Val&gt; { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">impl </span><span>from(input: Vec2) -&gt; Size&lt;Val&gt; { </span><span> Size { </span><span> width: Val::Px(input.x), </span><span> height: Val::Px(input.y), </span><span> } </span><span> } </span><span>} </span></code></pre> <p>Putting this all together we get (pseudocode) something like this:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">let</span><span> my_ui_element </span><span style="color:#ff5e5e;">=</span><span> SimpleNodeBundle { </span><span> position: SimplePosition::TopRight, </span><span> size: Vec2 { x: </span><span style="color:#fdb082;">25.0</span><span>, y: </span><span style="color:#fdb082;">100.0 </span><span>}, </span><span> color: Color::</span><span style="color:#fdb082;">RED</span><span>, </span><span>} </span><span> </span><span style="color:#6699cc;">some_bevy_ui_method</span><span>(my_ui_element.</span><span style="color:#6699cc;">into</span><span>()); </span></code></pre> <p>Skeptical readers might be thinking "Wow that is awful. This is so much code just to convert one stuct to another slightly simpler struct". You're right that it's a lot of code, but I promise in practice this is a game changer. Instead of remembering how to express your ideas to your library of choice every single time, you can express a higher level concept and <code>.into()</code> your framework's lower-level structure. Being able to succinctly express yourself while still getting the flexibility of a strong expressive type system is a killer feature of Rust and the use of these traits.</p> <h2 id="conclusion">Conclusion</h2> <p>Many of Rust's "pros" are also "cons". Memory safety results in frustratingly negotiating with the compiler. A strong type system with compile-time complexity results in slow (but improving) compile times.</p> <p>For the purposes of this post it is Rust's bias toward being explicit. Unlike other languages which automagically apply crazy changes to your code, Rust rarely <em>assumes</em> you want magic sprinkled everywhere. You can opt-in to that magic and all of the compile-time and runtime penalties that come with it, but it won't be secretly given to you for you to opt-out of.</p> <p>Rust can sprinkle magic on your code, but you have to explicitly call <code>.magic()</code> -- or in our case <code>.into()</code> and <code>.default()</code>. This is a nice middle ground between tedious code and black magic. When a language has <em>too much</em> magic it can result in wild performance implications from seemingly small code changes. While tedious code is just a pain to write, even if it is transparent about it's runtime performance. Here Rust is able to be transparent, you can audit every use of <code>.into()</code> and assess the runtime penalty, while still feeling magical.</p> 2022-01-18T00:00:00+00:00 2022-01-18T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/buying-a-house-04-is-this-investing/ <p><em>This is part 4 of a 4 part series on buying a home</em>. <em>Check out <a href="https://elijah.run/blog/buying-a-house-01-lets-go-shopping/">part 1</a> for to start from the beginning.</em></p> <p>Before buying a house a major talking point used to convince me of the idea was some variation of "It's a great investment!" or "You're just throwing money away by renting". I try to have a healthy level of skepticism about things, especially when counter-arguments aren't part of most the usual talking points on a topic. So let's figure out "is buying a house a good investment?"</p> <h2 id="houses-are-a-great-investment-origin-story">"Houses are a great investment" origin story</h2> <p>Before de-bunking -- or just bunking -- this myth, let's ask where it came from.</p> <p>Hundreds of years ago, when you had to own property to be considered a voting citizen, investing in property was a great idea! Housing was the most affordable asset you could buy that could passively grow in value.</p> <p>As the years went on this continued to be true. Housing is a market rarely which declines value, with umm... one big exception, and it can easily be passed on to family, creating generational wealth.</p> <p>This notion also pre-dates easier market investing tools like the index fund, Vanguard, and Robinhood. When it wasn't really clear how an average individual could get a diversified portfolio of stocks and bonds, a house was a safe investment that was all but guaranteed to increase in value -- and it would protect you from snow and rain and shit!</p> <p>So the origin story of this mantra isn't <em>false</em>, but it may be outdated. Let's find out.</p> <h2 id="meet-finley">Meet Finley</h2> <p>For the purposes of this post we will invent a hypothetical protagonist named Finley. Our fictional mascot was created in a lab to be incredibly average: the most basic curve-fitting person we could find in the the Portland tri-state area. Finley likes all the Marvel movies, drinks IPAs, and they go on hikes every other weekend. They also make an average salary, live in an average apartment, and intend to buy an average house. How convenient for talking about hypotheticals.</p> <p>Finley lives in Portland, so all of our numbers are for Portland Oregon, USA.</p> <h3 id="first-finley-rents-an-apartment">First Finley rents an apartment</h3> <p>Finley is currently renting an apartment. On average this is $1250 for the Portland area. This is roughly equivalent to what you would pay for a mortgage on a 300k house with a 20% (60k) down payment and a 3.0% interest rate (which is only a little optimistic).</p> <p>Here's what I used to cook up those numbers: * <a href="https://www.mortgagecalculator.org/">https://www.mortgagecalculator.org/</a> * <a href="https://www.realtor.com/mortgage/rates/Portland_OR">https://www.realtor.com/mortgage/rates/Portland_OR</a></p> <h3 id="then-finley-buys-a-house">Then Finley buys a house</h3> <p>In 2020, when I bought a house, the median house prices ranged from 430k to 500k trending upward. In the last 12 months from writing this post median home prices have ranged from 469k to 550k with a trend still going up.</p> <p><a href="https://www.redfin.com/city/30772/OR/Portland/housing-market">https://www.redfin.com/city/30772/OR/Portland/housing-market</a></p> <p>For simplicity let's say Finley got a deal and bought a house for 480k with a 3.0% mortgage rate. Their down-payment is 96k and their monthly mortgage payment is $1,900.</p> <p>Right off the bat Finley is paying 650 more per month on top of the 96k they had to pay up front. Let's assume Finley is still able to cover this cost since they're very frugal and despite cost of living going up 34% they are still spending less than 1/3 of their income on living expenses.</p> <h2 id="how-money-works-for-you">How money works for you</h2> <p>What has Finley <em>financially</em> gained and lost by buying a house?</p> <p>Finley has <em>lost</em> the ability to invest a lump sum of 96k on top of an extra 600/month which could be going toward a range of investments. Assuming Finley would have invested this in a stock market index we expect this to have increased in value by about 10% year over year given historical trends. To put numbers to these claims: 96k alone would have appreciated in value to 1,600,000 over 30 years, the duration of most home loans. If we add the additional 600/month we would have about 2,860,000 after 30 years.</p> <p>(these are very rough calculations using this python code)</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span>a </span><span style="color:#ff5e5e;">= </span><span style="color:#fdb082;">600 </span><span style="color:#ff5e5e;">* </span><span style="color:#fdb082;">12 </span><span style="color:#6d6d6d;"># Annual investable income </span><span>s </span><span style="color:#ff5e5e;">= </span><span style="color:#fdb082;">96000 </span><span style="color:#6d6d6d;"># Starting seed fund </span><span style="color:#ff5e5e;">for </span><span>x </span><span style="color:#ff5e5e;">in </span><span style="color:#6699cc;">range</span><span>(</span><span style="color:#fdb082;">30</span><span>): </span><span style="color:#6d6d6d;"># Collect 30 years </span><span> s </span><span style="color:#ff5e5e;">*= </span><span style="color:#fdb082;">0.1 </span><span style="color:#6d6d6d;"># Grow by 10% for the year </span><span> s </span><span style="color:#ff5e5e;">+= </span><span>a </span><span style="color:#6d6d6d;"># Add money saved by renting </span></code></pre> <p>(this is a very rough approximation of collecting interest over 30 years)</p> <p>Note this doesn't mean he are guaranteed to have 2.9 million at the end of 30 years; some of those gains would be diminished by inflation, rent would go up, and the market could have a down-turn at the 30 year mark. Plus, Finley would probably spend more as they get older (called "lifestyle inflation") resulting in diminished invest-able funds -- but they would also probably get cost of living raises over time, so this should all come out in the wash.</p> <p>But a house would <em>also</em> grow in value right? This is tricky to calculate because we're in a weird housing market right now -- don't call it a bubble -- but let's take the last 10 years at face value. In 2011 the median Portland house was ~260k, and today it is ~550k. This is a 111% return over 10 years, or 7.82% annually.</p> <p>So if Finley sold their house after 10 years they would make a profit of ~290k.</p> <p>I neglected to calculate what Finely's index fund strategy would return after only 10 years, so let's do that now. Using the above code, but limiting it to just 10 years instead of 30, we get a ~363k total which does beat the housing investment by a fair margin. <a href="https://www.calculator.net/roi-calculator.html?beginbalance=259000&amp;endbalance=550000&amp;investmenttime=date&amp;investmentlength=2.5&amp;beginbalanceday=08%2F01%2F2011&amp;endbalanceday=08%2F01%2F2021&amp;ctype=1&amp;x=69&amp;y=36/">citation needed</a></p> <p>Note that I am omitting all of the nickles and dimes that add up on both sides. Capital gains tax makes your index fund strategy less appealing. Home ownership means you are on the hook to fix anything that breaks, pay property tax, and there are fees associated with buying and selling a home -- plus the cost of the loan which we didn't touch on at all! At the same time, all the money Finley was paying in rent really was being flushed away and not appreciating in value, which is a shame.</p> <h2 id="elijah-just-tell-me-if-it-s-good-or-bad">Elijah, just tell me if it's good or bad</h2> <p>A house is not a great investment, but that's not the point.</p> <p>I didn't buy a house because it was a good investment, I bought a house because I wanted more space and it penciled out to about the same monthly cost as my super expensive bougie central-Portland apartment. I am glad that my house will re-coop some cost when I decide to move out, and I do like the stability of home ownership; I don't need to worry about rent going up, or getting evicted because my landlord wants to demolish the building for something else, etc.</p> <p>So many variables go into the "is this a good investment" question. If you need to stretch yourself thin, or you already have debt, or what the monthly cost is -- all of these and more contribute to the viability of buying a house. If your monthly housing costs wouldn't go up, it's probably a solid move.</p> <p>Home ownership is not perfect, it's not a great investment, but it is an OK use of your money if you can afford it. It's financially sound if you can make it work, comfortably, but it's not your only option. Being frugal and renting an affordable apartment allows you to make other "better" investments, if that's all you're min-maxing for.</p> <p>I don't regret buying a house (yet?) but I would be just as happy if I was renting a house, or still living in an apartment.</p> <h2 id="one-more-soap-box">One more soap-box</h2> <p>Side note: we need to build more housing. Portland passed a bill recently allowing more dense housing. If I had more money I would throw money at building triplex housing in Portland.</p> <p>For too many of my 20-something peers the problem is not "If I want to buy a house" but "When can I afford to buy a house?". There is a bit of a market boom right now, but with more housing the market should settle down. In a perfect world anybody would be able to buy property if they want.</p> <hr /> <p>Thank you for reading this fourth and final installment of my housing series. Now go out a buy a house!</p> <p>Lol, just kidding. Go do more research you crazy kid.</p> 2022-01-04T00:00:00+00:00 2022-01-04T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/buying-a-house-03-renovation-station/ <p><em>This is part 3 of a 4 part series on buying a home</em>. <em>Check out <a href="https://elijah.run/blog/buying-a-house-01-lets-go-shopping/">part 1</a> for to start from the beginning.</em></p> <p>Congratulations! You slogged through finding, trying to buy, and then successfully buying a house. Give yourself a pat on the back because that is an impressive feat you've accomplished.</p> <p>This post is more of a personal post than the others covering the renovations we've made to our house in the first year. I hope it helps illustrate some of the pros (and cons) of being able to do whatever the heck you want when you own the place.</p> <h2 id="let-s-fuck-up-this-yard">Let's fuck up this yard</h2> <p>The best part of our house has been the back yard. I'm writing this post from my standing desk on my back patio, our doggo loves playing fetch in the yard, and we even started a garden.</p> <h3 id="patiooooo">Patiooooo</h3> <p>Our first project was to build a stone patio. Not some half-assed "throw some rocks on the ground and call it good" thing, but some real HGTV quality shit.</p> <p>Lucy of course did all of the research, planning, and purchasing of materials. I moved a bunch of rocks.</p> <p>Before we could build the patio we had to dig 7" of dirt out from the ground, which is non-trivial. We invited some friends, bought Thai food, and got to work in late October.</p> <p>Fun fact: 7" of dirt for a 10"x16" hole is a shit-ton of dirt. We eventually had to rent one of those big metal waste disposal bins you usually see in front of houses or other construction sites. Of course then we had to move the dirt a second time from piles in our yard to the bin, but then it was gone for good.</p> <p>Once we dug out the dirt we were able to fill it back in with rocks (pebbles) then smaller rocks (sand) and finally pavers, which is what the industry calls short cube rocks -- not to be confused with bricks.</p> <p>The hardest part was definitely making everything flat. The pavers are big squares, which I though would be forgiving if the underlying sand and gravel wasn't flat, but wow I was wrong. Eventually I found my groove and got very zen about making the sand flat. We used used a 10' 2x4 and dragged it across our layer of sand in a variety of patterns until eventually it was "flat enough". In retrospect it was kinda fun. Very calming once you submit to the process.</p> <p>Finally we put the pavers down, filled the cracks with polymeric sand (which is just "concrete lite"), and sealed that bitch up.</p> <p>The whole project took a few weeks and cost about 1k for materials.</p> <p>We almost gave up at the end when we realized making things flat was way harder than expected. The quote to get the project finished, which was 90% done, was around 5k which gave us heart-burn so we stubbornly finished it ourselves. So it turns out we saved ~5k by doing this project ourselves.</p> <p>After this I'm game to do any home project myself as long as it doesn't involve electricity and plumbing in the walls.</p> <h3 id="baby-you-re-a-fiiiireee-pit">Baby you're a fiiiireee-pit</h3> <p>A comparatively small project was our fire pit. This was just a fire ring with some bricks stacked on the outside. Very safe, very simple.</p> <p>Honestly it was nice to get a quick win.</p> <h3 id="big-ass-raised-bed">Big ass raised bed</h3> <p>We also built a big ass raised bed. It's 4'x12' and it gets a <em>lot</em> of sun. Also a quick win.</p> <p>Everything feels like a quick win after building that fucking patio.</p> <h2 id="new-hvac-who-this">New HVAC, who this?</h2> <p>In December ancient gas furnace ran out of diesel, resulting in not having heat... funny how that works. This is usually trivial to solve: you call a company and they fill your gas tank and charge you a few hundo. With gas in the tank your heat comes back on.</p> <p>Multiple people talked about how great the furnace we had was. It was installed with the house, so it was at least 70 years old and it did work well. The heat that came out of the vents was super hot, but it also ran on gasoline in a tank in our basement.</p> <p>I'm sure the carbon footprint of my home's gas heating is negligible when compared to the carbon footprint of every flight I take, but Lucy and I decided if we <em>could</em> get rid of it we <em>should</em>. We did some research on alternatives and decided an electric Heat-Pump would be a good alternative. It would provide us with heat <em>and</em> cooling, and wouldn't run on diesel; the only downside is it'd be expensive.</p> <p>A few things:</p> <ol> <li>A Heat-Pump is a heating and cooling system that operates on similar physics to a refrigerator (taking warm air and turning it into cool air, and the inverse).</li> <li>The Heat-Pump has an outdoor unit, you've definitely seen them outside some homes. Usually it's a metal box about 3" cubed with a fan.</li> <li>They don't tell you that's just the <em>outdoor</em> part of the Heat-Pump, there is also a huge freaking indoor unit that converts chemical energy into warm/cool air that gets pumped around your house.</li> <li>Heat-pumps are fully electric so no gas!</li> </ol> <p>So that's how a Heat-Pump works. A few stars aligned for this to work out:</p> <ol> <li>Our house already has vents that the oil furnace used to heat the house. This means we can easily heat/cool the entire house with one indoor unit.</li> <li>We had some extra cash we didn't end up needing for our down-payment, so we were able to pay in cash saving us a lot in interest payments for the Heat-Pump (like 10%!).</li> </ol> <p>We could have made it work without those happy coincidences; they make in-wall Heat-Pumps that heat/cool one or two rooms, and we could have financed if we were strapped for cash, or we could have just kept the old gas furnace. But we <em>were</em> able to upgrade our central-air because 6 months later Portland had a 118℉ heat wave and another 100℉+ heat wave the day before our wedding! Both of which we barely worried about because we were fortunate enough to have AC.</p> <h2 id="life-after-renting">Life after renting</h2> <p>I hope this post showed some of the awesome pros of home ownership, like being able to add a patio, with some healthy cons, like having to spend 12k on a new furnace. We were able to get rid of the gas guzzling heating system in favor of a full-electric system. We were able to build a patio, add a raised bed, fuck up the yard, without asking anybody for permission.</p> <p>It's a lot of work, and a lot of money sometimes, but it is empowering to be able to <em>own</em> your house.</p> <hr /> <p>In the next post we'll throw all this soft "american dream" and "empowered to change" stuff out the window and figure out if this is a good investment!</p> 2021-12-21T00:00:00+00:00 2021-12-21T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/buying-a-house-02-show-me-the-money/ <p><em>This is part 2 of a 4 part series on buying a home</em>. <em>Check out <a href="https://elijah.run/blog/buying-a-house-01-lets-go-shopping/">part 1</a> for to start from the beginning.</em></p> <p>In this post we will talk about everything that happens between getting an offer accepted on a house and your move-in day. How exciting! Let's get started.</p> <h2 id="so-you-got-an-offer-accepted">So you got an offer accepted</h2> <p>You slogged through the home-buying process, your heart was broken in 12 pieces when you barely didn't get "the one" and then "the other one" and finally "the real one", but you powered through and now you have an offer accepted.</p> <p>Great, you're half way done! Just kidding, this part is much easier, but it is time consuming.</p> <p>Now in theory you can skip all these steps with a shit-ton of money and taking a shot of unnecessary risk. I mean you can pay in cash for a house <em>and</em> waive home inspections to expedite the home-buying process. I only have one anecdote of this happening, and the guy who told it to me was a prick, so I'm pretty sure it's fake news. The only reason you would pay in cash is to avoid paying mortgage fees and to close the deal faster. The only reason you would waive inspections is to make close on the house faster if you don't care or really trust the current owner. Mortgage rates are bonkers low right now, and most people aren't in <em>that much</em> of a rush to close on their house, so again: fake news. For normal people with a normal amount of debt and a sane income we do things the way I'm about to describe.</p> <p><em>Steps down from soap-box</em>.</p> <p>Now that your offer is accepted you have a 30 day clock. At the end of those 30 days you will hopefully have a house. Very cool.</p> <h2 id="inspections">Inspections</h2> <p>The first step is to schedule home inspections. Your realtor should do this. You will schedule a standard home inspection, but you can also request special inspections if you are concerned. For example we tested for Radon and buried oil tanks, because both of those are shitty things in Portland. These inspections are one of the few fees paid by the buyer because you as the buyer can be as picky as you want about the inspections.</p> <p>Once the inspections happen you'll do a walk-through. Often this is the <em>second time you have seen the house in person</em>, which I just find hilarious. The inspector should walk you through the house and tell you literally everything wrong with the house that they can find. They might also mention any concerns they have based on the listing's disclosures and the year the house was built/renovated. The inspector shouldn't have any horse in this race, they're getting paid weather the deal falls through or not, so they should be pretty matter-of-fact. This means they might tell you some scary shit like "This ceiling is probably asbestos, don't touch it until you get that tested", this is just because you need to know everything that might be a deal breaker before the next step.</p> <p>Once the walk-through is complete your inspector will write up an inspection report and send it to you and your realtor for you to review and keep. This is a very long, very boring, and very important document. Keep a copy.</p> <h2 id="negotiations">Negotiations</h2> <p>Now that you know everything about the house:</p> <ul> <li>Disclosures given in the house listing.</li> <li>Inspection results.</li> <li>You've been inside like... twice.</li> </ul> <p>You can enter the negotiation phase fully informed. Negotiations are not done in a board room -- I wish -- they're done asynchronously by the seller and buyer's realtors.</p> <p>You sit down <strong>with your realtor</strong> and go over the inspection results. If your realtor is good they'll basically do the negotiation for you, reviewing what the plan is before locking it in.</p> <p>For example, our realtor said the following:</p> <ul> <li>The pipes are clogged with baby wipes or something, that can take 7k to fix so let's ask to knock off 7k.</li> <li>The garage door is busted, let's ask to take 5k off for that.</li> <li>Some other stuff got knocked off but I don't remember it a year later.</li> </ul> <p>We figure out how much to ask for by basically looking up how much it would reasonably cost to fix broken shit. If your roof was caving in you would be able to ask to take a lot, if a stair was busted that would probably be less.</p> <p>You can't get away with asking for price reductions for <em>aesthetic</em> differences, like you wish the house had more outlets, but you could ask to get the price knocked if the electrical was unsafe.</p> <p>Your realtor should then write this up and send it to the seller's realtor who will respond with a counter. Sometimes they will choose to fix broken things themselves or they may drop the price for a given request. In our case the owner dropped the price a bit, around 3k, and spent her own money to fix clogged pipes, which cost a few thousand.</p> <p>In the negotiation the seller has an incentive to keep the final price fairly high while the buyer has an incentive to get the price as low as possible. This means sellers might find more expensive quotes and the buyer might find cheaper quotes for repair costs. But both parties have an incentive to complete the sale of the house.</p> <p>If a seller refuses to negotiate on the price of the house, the buyer at this point has the option to pull out of the deal. The buyer will most likely get their Earnest Money back because they found something out about the house after the offer was accepted which the buyer refused to fix or decrease the price for.</p> <h2 id="but-what-does-the-bank-think-it-s-worth">But what does the bank think it's worth?</h2> <p>After negotiating on a price the bank gets involved again. It takes the bank under a week to, given the home inspection plus their piles of data, determine what they think it's worth. This has very little to do with what you thought it was worth and it is what the bank is willing to cover in the form of a home loan.</p> <p>If the bank decides the house is worth <em>more</em> than the final offer you're golden. You get a loan for the negotiated price and it'll be 100% covered by the bank.</p> <p>For example: If the final offer is 100k and the bank decides the house is worth 105k, you need to do the normal 20% down payment, and then you get an 80k loan for the rest.</p> <p>... But if the bank decides the house is worth <em>less</em> than the final offer you're some amount of fucked. You need to pay the difference out of pocket, or pull out of the deal.</p> <p>For example: If the final offer is 100k but the bank decides the house is worth 60k, you need to cover 20k down payment, plus the 20k difference on the remaining cost. 100k - 20k = 80k ideal loan. 80k - 20k = 60k what the bank will actually loan you.</p> <p>Note, I'm using 100k as a round number for math examples. This is not some cruel joke, it's just easier numbers. Most houses we looked at were between 350k and 600k.</p> <hr /> <h3 id="side-note-why-do-we-need-down-payments">Side note, why do we need down payments?</h3> <p>Wait, why do we have down payments? It's complicated, but I <em>think</em> it's a collection of reasons:</p> <ul> <li>The more you pay up front, the less monthly payments you have. 20% up front is a good way to reduce your monthly payments from the get-go.</li> <li>Credit worthiness or some bullshit. Banks want to make sure you have skin in the game, so before you can get approved for a loan you need to prove you have 20%+ of the cost of the house for them to cover the other 80%.</li> </ul> <p>That's my best bet. Ask a realtor or your local economist or banker or something; they'll probably have a better answer.</p> <hr /> <p>Once the bank settles on a price you can close on the house. You'll get a real loan for the final price (minus the difference you've paid in cash). Finally, 30 days (ish) after getting your offer accepted you'll walk into a title company, go over a stack of papers with a professional, sign a bunch of those papers, and walk out the proud owner of a house.</p> <h2 id="what-now">What now?</h2> <p>We had to wait a few weeks after closing to actually move in. The previous owner needed the funds from closing to close on her own house, but that ended up going faster than expected so we only had to stay at my parents house (#privilege #safetynet) for two weeks.</p> <p>Your realtor should give you some keys for the new house. I suggest throwing those out once you're moved in and replace the locks -- even in a nice neighborhood, you have no idea who has the old keys.</p> <p>Rent a moving van and bribe your friends, or hire some movers if you're smart, and enjoy your beautiful house!</p> <hr /> <p>Wait, is this pipe supposed to be leaking? In the next post we'll talk about home renovations!</p> 2021-09-02T00:00:00+00:00 2021-09-02T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/python-dynamic-dispatch/ <p>🦝 Hey want tO see a nEat PYtHoN triCK?</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="font-style:italic;color:#ff5e5e;">def </span><span style="color:#8cdaff;">f</span><span>(</span><span style="font-style:italic;color:#fc9354;">name</span><span>: </span><span style="color:#fbe3bf;">str</span><span>): </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">hiya </span><span style="color:#fdb082;">{}</span><span style="color:#ffffff;">&quot;</span><span>.</span><span style="color:#e9fdac;">format</span><span>(name)) </span><span> </span><span style="color:#6699cc;">locals</span><span>()[</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">f</span><span style="color:#ffffff;">&quot;</span><span>](</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">Spongebob</span><span style="color:#ffffff;">&quot;</span><span>) </span></code></pre> <p>🐮 Oh cool, does that use the string <code>"f"</code> to find--</p> <p>🦝 DoEs It FiNd ThE fUnCtIoN <code>"f"</code> aNd CaLl iT?? yEaH It DoEs.</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>$ python script.py </span><span>hiya Spongebob </span></code></pre> <p>🐮 Oh that's neat. I didn't know you could do that with Python--</p> <p>🦝 whaTEveR i dON't cArE wHaT YOU tHinK.</p> <p>🐮 You asked me --</p> <p>🦝 I sAiD sHuT uP. i'M dOiNg HoT gIrL sHiT.</p> <p>🐮 Are you ok? You're yelling a lot about Python again--</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#6d6d6d;"># shut up </span><span> </span><span style="font-style:italic;color:#ff5e5e;">def </span><span style="color:#8cdaff;">test_patrick</span><span>(</span><span style="font-style:italic;color:#fc9354;">context</span><span>: TestContext): </span><span> </span><span style="color:#6d6d6d;"># do thing </span><span> </span><span style="font-style:italic;color:#ff5e5e;">def </span><span style="color:#8cdaff;">test_crabs</span><span>(</span><span style="font-style:italic;color:#fc9354;">context</span><span>: TestContext): </span><span> </span><span style="color:#6d6d6d;"># do other thing </span><span> </span><span>context </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">TestContext</span><span>() </span><span>tests </span><span style="color:#ff5e5e;">= </span><span>[] </span><span style="color:#ff5e5e;">for </span><span>t </span><span style="color:#ff5e5e;">in </span><span style="color:#6699cc;">locals</span><span>(): </span><span> </span><span style="color:#ff5e5e;">if </span><span>t.</span><span style="color:#e9fdac;">startswith</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">test_</span><span style="color:#ffffff;">&quot;</span><span>): </span><span> tests.</span><span style="color:#e9fdac;">append</span><span>(t) </span><span style="color:#ff5e5e;">for </span><span>test </span><span style="color:#ff5e5e;">in </span><span>tests: </span><span> </span><span style="color:#e9fdac;">test</span><span>(context) </span></code></pre> <p>🐮 Oh so you could...</p> <p>🦝 ...</p> <p>🐮 Are you going to interrupt me?</p> <p>🦝 Not yet.</p> <p>🐮 Ok... you could search your scope and find a bunch of functions with a name and common interface--</p> <p>🦝 AND CALL THEM! DYNAMIC DISPATCH!!</p> <p>🐮 That's nice.</p> <p>🦝 YEAH IT IS COOL!</p> 2021-08-26T00:00:00+00:00 2021-08-26T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/buying-a-house-01-lets-go-shopping/ <p>So you're interested in buying a house eh? Well boy howdy do I have a series of blog posts for you. This four part series will equip you, a first-time home buyer, with a bunch of knowledge I wish I had before starting my journey to buying a home.</p> <p>Here's the road-map:</p> <ol> <li><a href="https://elijah.run/blog/buying-a-house-01-lets-go-shopping/">Part 1: 🛒 Let's Go Shopping!</a> in which we start looking for houses.</li> <li><a href="https://elijah.run/blog/buying-a-house-02-show-me-the-money/">Part 2: 💸 Show Me the Money!</a> in which we buy a freaking house.</li> <li><a href="https://elijah.run/blog/buying-a-house-03-renovation-station/">Part 3: 👷 Renovation Station.</a> in which we live in the freaking house.</li> <li><a href="https://elijah.run/blog/buying-a-house-04-is-this-investing/">Part 4: 📈 Is This Investing?</a> where we figure out if we made a huge financial mistake (or achievement!)</li> </ol> <hr /> <h2 id="hey-quick-disclaimer">Hey, quick disclaimer</h2> <p>Some things I <strong>can</strong> claim to be an expert on: Docker, Linux, why my dog is making that face, etc. But with home-buying I am at best a novice, so please take everything you read here with a grain of salt.</p> <p>I am just one lowly home owner in his mid-20s who bought a house during a pandemic. I got here through a bunch of privilege, support, luck, and the modest amount amount of hard work <em>required</em> to live a wonderful mortgage-filled life.</p> <p>I am not a financial advisor, realtor, none of that stuff. I encourage you to do independent research, check my work, etc. Don't make any decisions based purely on this series. Please use it as one of many sources in your research before buying a house.</p> <p>Okay, <strong>now</strong> we can get started.</p> <hr /> <h2 id="so-um-why-do-you-want-to-buy-a-house">So um, why do you want to buy a house?</h2> <p>Before diving into the home buying process, let's ask an annoyingly simple (but annoyingly effective) question: "why?"</p> <p>There's a million reasons to want to buy a house. A few that come to mind are</p> <ol> <li>You want more space or a yard.</li> <li>You want an investment.</li> <li>You want to be in a residential neighborhood.</li> <li>You want to fulfill the American Dream™.</li> </ol> <p>Now none of these <em>require</em> you to buy a house. You could rent a house to get more space, invest in high yield assets to make passive income, etc. but a house tends to check <strong>all</strong> these boxes.</p> <p>My fiance (now wife) and I we were living in a small apartment before buying a house. This apartment was perfect pre-COVID. It was a 10-20 minutes walking commute to work, centrally located, and big enough to sleep, cook meals, and rest while being young 20-somethings living our best lives.</p> <p>When COVID hit we though we could muddle through in the apartment, but around the 2 month mark we thought:</p> <blockquote> <p>Hmm, this isn't gonna be the "Come on Morty. Just two weeks, we'll be in and out. Just a little pandemic to spice things up." thing that we thought this would be...</p> </blockquote> <p>We looked for house rentals, but while there were hundreds of houses for sale there were almost no houses rentals available in the Portland area. I was always resistant to buying a house, but after realizing it was a reasonable financial decision we got started.</p> <p>This is where our home buying journey begins.</p> <h2 id="let-s-triage">Let's triage</h2> <p>Before buying a house, or even visiting a house, you should go to sites like Zillow and Redfin to get a lay of the land. They both have basically the same home listings, so choose whichever site has a better interface for you. This goes without saying though; you found this obscure blog post so you're probably deeply familiar with all the different home buying sites. I promise this series will start saying less "yeah duh" stuff in the next section.</p> <p>We knew roughly what neighborhoods we wanted to live in: East Portland with a bias toward the Tabor neighborhoods. This made it quick to narrow our search and skip the existential question "where in the world do I want to live?" I'm sure some people confront.</p> <h2 id="money-talk">Money talk</h2> <p>Once you have a feel for the housing market in your target locations, check in with your finances.</p> <h3 id="a-big-wad-of-cash">A big wad of cash</h3> <p>Buying your soon-to-be house will require a down-payment. This means you'll need a very big pile of money <em>very soon</em>. Figure out what your liquid assets (savings, non-retirement investment accounts, dubious favors you can call in) talley up to. This is the max you could use for a down-payment.</p> <p>Down payments tend to be 20% of the purchase price of the house, and houses are hundreds of thousands of dollars, so... yeah that's a lot.</p> <p>For example:</p> <ul> <li>If you have 20k in usable assets, you can afford a down payment on a 100k house.</li> <li>84k usable assets? 420k house.</li> <li>13.8k usable assets? 69k house.</li> </ul> <p>Nice.</p> <p>This sets an upper-limit on which houses you can afford. Hopefully this upper-limit lines up with median home prices in your target neighborhoods.</p> <h3 id="a-smaller-monthly-wad-of-cash">A smaller monthly wad of cash</h3> <p>You also need to consider the monthly cost of your potential home. Ideally you only spend up to 30% of your monthly income on living expenses. This means your house (with utilities) costs no more than 30% of what you take home. Some of you may be laughing at this because it is in fact an outdated metric that has not kept up with inflating home prices. Don't worry, we'll cover how terrible home prices are in a later installment.</p> <p>The 30% rule says:</p> <ul> <li>If you take home 2k/month (roughly a 30k/year salary after taxes) your housing should cost no more than 600/month.</li> <li>5833/month (100k salary) take-home? 1750 housing expenses.</li> <li>11452/month (200k salary) take-home? 3435 housing expenses.</li> </ul> <p>But what does that translate to for home prices? You don't pay for the whole house up front (the bank does that). Instead you pay for it in monthly installments with interest -- which is hard for me to intuit. Skipping some prelude:</p> <ul> <li>If your house is 100k, you put up a 20k down payment, and you acquire a 30 year loan with 4% interest, your monthly payment will be ~587.</li> <li>200k house, 40k down, 4% interest 30yr loan? ~1070.</li> </ul> <p>With these numbers, as the price of the house goes up the monthly payments go up sub-linearly. This means so a 400k house is a little less than 2000/month, ec.</p> <p>I have no idea how to calculate these numbers by hand, I always use a calculator... <a href="https://www.zillow.com/mortgage-calculator/">https://www.zillow.com/mortgage-calculator/</a>.</p> <p>Once you have established you can afford both the down-payment and monthly payments now we can start shopping!</p> <p>If prices in your target neighborhood are out of reach, explore other candidates; neighborhoods near your targets, or even the 'burbs. Also if you just like shopping for homes for the kick of it, because shopping is fun, power to you.</p> <h2 id="get-yourself-a-realtor">Get yourself a realtor</h2> <p>A realtor is like a lawyer. You never <em>want</em> one, but they are incredible necessary for some of life's biggest shit.</p> <p>We were #blessed to have a family friend that was an active realtor; her name is Mary.</p> <p>The relationship we had with our realtor was like this:</p> <ol> <li>Every week around Tuesday we would send her a list of houses we wanted to see. Between 5-10 houses every week.</li> <li>Occasionally she would also find a house she thought we would be interested in.</li> <li>The first week she picked about half the houses to get a feel for what we were interested in, and we brought some we found on our own.</li> <li>She would drive us around to these houses, because we refuse to buy a car and she is <em>very</em> nice.</li> <li>Mary would schedule a time for us to visit the house, around 30 minutes per house, during this time we were the only ones in the homes.</li> </ol> <p>As a card-carrying realtor Mary had special privileges not given to us normies, and information not found on public sites. For example:</p> <ul> <li>Mary had a list of things disclosed about the house like "It's very full of asbestos" or "It's full of radon" or "The roof caved in last year, just fixed". Info you would want to know before putting an offer on a house.</li> <li>She also had access to some cloak-and-dagger shit that let her into all of the houses on the market. Ever seen a lock-box on a for-sale house? She could open that lock-box and get the key out with an app on her phone.</li> <li>She also knew how to fill out the copious forms needed to put an offer on a house, but we'll get to that in the next post.</li> </ul> <h2 id="so-many-houses">So. Many. Houses.</h2> <p>In total we were house shopping for about 5 weeks. We averaged visiting 10 houses per week, doing 1 to 2 trips per week and visiting 6 houses per trip.</p> <p>We saw a lot of houses. Some great houses in safe boring neighborhoods far from where we wanted to live. Some terrible houses that people kept calling "fixer-uppers" like they were in the pocket of big shitty house.</p> <p>A lot of houses that were great except for like... one really important thing. Some houses had a great location but were too small. Or the house was the perfect size but too far from our target location. Or the price and size were good but the neighbors were burning a literal pile of trash and they were like... very good at it, implying this wasn't a one-off thing.</p> <p>My point is, there were a lot of houses. A lot that were good, many more that were non-starters.</p> <p>The first week or two was basically a mulligan. So many houses are so bad you start to think <em>all</em> houses are bad and you trick yourself into settling. DO NOT SETTLE. This is a huge freaking purchase, grind it out and you will find a house you like. It might be at the upper end of your budget, or it might be a <em>little bit of a fixer upper</em>, but you'll know the perfect house when you find it. And you will <em>definitely</em> know when it's <em>not</em> the right house.</p> <h2 id="getting-on-your-knees-and-asking-the-bank-for-a-pre-approval">Getting on your knees and asking the bank for a pre-approval</h2> <p>At this point in the story we haven't bought any houses yet, but before you are allowed to <em>think</em> about buying a house you need to get a bank to say "Yeah they're good for it". This requires asking a bank pretty please, giving them like 3 years of financial, employment, and credit records. If you're buying this with a partner they need to do the same thing.</p> <p>In our case we tried to get a super cheap loan but we were told our offer might get rejected if we didn't have a "local loan" meaning a loan provided by a local Oregon company -- what year is it 1980? In the end all of our loan options were super cheap, even the local ones, so we shrugged and told ourselves the cheapest loan probably had a bunch of closing fees that would basically cancel out the higher mortgage rate.</p> <p>A <em>fun fact</em> about the loan process is that you get pre-approved for a loan up to some amount, which is used to put an offer on a house Once the offer is accepted you get the <em>real</em> loan for whatever the final loan amount ends up being. But we're getting ahead of ourselves.</p> <h2 id="the-offer-process">The offer process</h2> <p>At some point you'll make an offer on a house. This might <em>sound</em> like we're buying a house, but trust me there are a lot more offers on houses than there are sales on houses.</p> <p>When you put in an offer on a house you basically roll for initiative and if you get a Nat-20 it gets accepted. You should still treat every offer as if it were to be accepted, because you can roll a Nat-20 on a persuasion check with a rat just as often as slaying a dragon. Offers are accepted and rejected basically at random, but each one is important and should be treated as "the one".</p> <p>When putting an offer on a house you fill out a bunch of complicated legal-ese forms with your realtor. Taking your pre-approval you decide on an offer up to your pre-approval limit. In Portland right now we need to offer <em>way more</em> than the asking price which is bullshit and I hate it.</p> <p>You also fill out a bunch of details like when you want the sale to be complete (30 days), weather you're planning on doing inspections (yes), and how much money you can put up <em>before your down payment</em>. That pre-down-payment is called Earnest Money and it's often between 2k and 10k, it counts toward your down payment, and it can be refunded if the deal falls through.</p> <h2 id="it-s-a-marathon-pace-yourself">It's a marathon, pace yourself</h2> <p>Lucy went into this expecting that buying a house was like buying anything else. You find what you like, offer a fair (or listed) price, and boom you get it.</p> <p>Unfortunately housing is not like buying a phone, or basically anything else most people spend money on. In Portland at least supply does not match demand, so prices are not only higher than they ought to be, but the prices are also <em>full of lies</em>. The asking price is often 10% below the final price -- if you're lucky! Plus your offer on a house may be rejected for mysterious reasons. More than once we were asked to re-submit an offer and to give our "highest and best" offer whatever the fuck that means.</p> <p>Be prepared to visit at <strong>dozens</strong> of houses. Be prepared to make an offer on multiple houses. You will have your heart broken when you find the perfect house, just to have it swiped out of your hands by a bank, somebody paying all in cash, somebody waiving inspections, or just some jerk that got an offer in half an hour before you did -- that bastard.</p> <p>Take your time. This is a big purchase. It will pay off. It will get better. Your patience will be rewarded.</p> <p>Good luck. See you in the <a href="https://elijah.run/blog/buying-a-house-02-show-me-the-money/">part 2</a> where we get to buy a freaking house!</p> 2021-07-07T00:00:00+00:00 2021-07-07T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/git-main-branch/ <p>So I was on my laptop running an ancient release of Linux called Fedora 30 -- so old COVID wasn't even a twinkle in a bat's eye when it was released.</p> <p>On this ancient OS I'm have <code>git 2.21.3</code> which is like 13 versions behind. I ask yum -- err <em>dnf</em> -- if there is an upgrade and it's like "nope". Nothing makes me feel older than software.</p> <p>I've gotten into the habit of naming my default branches <code>main</code> instead of <code>master</code> because every time I say "master" I need to take a shower. I'm pretty sure the latest git names the default branch <code>main</code> out of the box, but git from April 2019 was not aware of problematic language choices.</p> <p>So as one does I'm starting a new repo and I decide before updating <code>git</code>, which is gonna require like 3 Fedora upgrades, I'm just going to rename the branch. I know I'll get woke-git if I upgrade Fedora, but I want to write this post first so we're gonna find a workaround.</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>[branch:master]$ git branch -m main </span><span>error: refname refs/heads/master not found </span><span>fatal: Branch rename failed </span></code></pre> <p>Hmm. So on a bare repo you can't rename the starting branch because there's no objects in the repo... or something like that <em>waves hands</em>.</p> <p>Umm... I guess we'll go spelunking into the <code>.git</code> directory to see if we can manually force our <code>main</code> branch naming.</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>[branch:master]$ cd .git </span><span>[.git]$ tree -F </span><span>. </span><span>├── branches/ </span><span>├── config </span><span>├── description </span><span>├── HEAD </span><span>├── hooks/ </span><span>│   ├── A bunch of sample scripts </span><span>├── info/ </span><span>│   └── exclude </span><span>├── objects/ </span><span>│   ├── info/ </span><span>│   └── pack/ </span><span>└── refs/ </span><span> ├── heads/ </span><span> └── tags/ </span></code></pre> <p>Well unsurprisingly we have a bunch of empty directories and some sample scripts.</p> <p>The only files that <em>might</em> be worth looking at are <code>config</code>, <code>description</code>, and <code>HEAD</code>.</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>// config </span><span>[core] </span><span> repositoryformatversion = 0 </span><span> filemode = true </span><span> bare = false </span><span> logallrefupdates = true </span></code></pre> <p>This seems to be config options for this repo, none of which mention branch naming so it's a skip for me.</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>// description </span><span>Unnamed repository; edit this file &#39;description&#39; to name the repository. </span></code></pre> <p>I had no idea this feature existed. I have never seen it be used -- but fun facts!</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>// HEAD </span><span>ref: refs/heads/master </span></code></pre> <p>And the money shot. Let's change that bad boy to <code>refs/heads/main</code> and see if my magical git prompt picks it up:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>[.git]$ cd ../ </span><span>[branch:main]$ </span></code></pre> <p>Heyo! There you have it. How to manually change the name of your main branch in a fresh git repo.</p> <p>Sanity check, this won't fuck up if we commit right?</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>$ cargo init . </span><span>$ git add . &amp;&amp; git commit -m &quot;initial commit&quot; </span><span>[main (root-commit) ebecbee] initial commit </span><span> 3 files changed, 12 insertions(+) </span><span> create mode 100644 .gitignore </span><span> create mode 100644 Cargo.toml </span><span> create mode 100644 src/main.rs </span></code></pre> <h2 id="note">Note...</h2> <p>After anxiously upgrading to Fedora 34 I confirmed that git fixed a lot of this:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>[pop@lappy foo]$ git init </span><span>hint: Using &#39;master&#39; as the name for the initial branch. This default branch name </span><span>hint: is subject to change. To configure the initial branch name to use in all </span><span>hint: of your new repositories, which will suppress this warning, call: </span><span>hint: </span><span>hint: git config --global init.defaultBranch &lt;name&gt; </span><span>hint: </span><span>hint: Names commonly chosen instead of &#39;master&#39; are &#39;main&#39;, &#39;trunk&#39; and </span><span>hint: &#39;development&#39;. The just-created branch can be renamed via this command: </span><span>hint: </span><span>hint: git branch -m &lt;name&gt; </span></code></pre> <p>And yes, <code>git branch -m &lt;name&gt;</code> does work on an empty repo.</p> 2021-01-31T00:00:00+00:00 2021-01-31T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/circular-imports/ <p>Don't you hate when you're writing a python program and you get hit with one of these?</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>Traceback (most recent call last): </span><span> File &quot;a.py&quot;, line 1, in &lt;module&gt; </span><span> from b import g </span><span> File &quot;/home/pop/Projects/src/localhost/circular/b.py&quot;, line 1, in &lt;module&gt; </span><span> from a import f </span><span> File &quot;/home/pop/Projects/src/localhost/circular/a.py&quot;, line 1, in &lt;module&gt; </span><span> from b import g </span><span>ImportError: cannot import name &#39;g&#39; from partially initialized module &#39;b&#39; (most likely due to a circular import) (/home/pop/Projects/src/localhost/circular/b.py) </span></code></pre> <p>What a drag.</p> <p>Circular imports are of course the literal devel in programming, but in a Godel Escher Bach sort of way they are all around us.</p> <h2 id="welcome-to-my-ted-talk">welcome to my ted talk</h2> <p>There are a lot of problems in the world that everybody agrees are problems<sup class="footnote-reference"><a href="#1">1</a></sup>, but we can sometimes disagree about what the solution is.</p> <p>Many of these problems stem can be thought of as emergent properties of other phenomena.</p> <p>Take for example, plastic.</p> <h2 id="fuck-plastic">fuck plastic</h2> <p>Remember the good old days in America when women tended to the house while PTSD ridden white men worked a 40 minute drive away and the ethnically, racially, sexually, abally discriminated minorities were repressed?</p> <p>Well something that actually was pretty sweet was milk delivery.</p> <p>Milk, delivered regularly in reusable glass bottles had incredibly ecological benefits compared with how we get milk<sup class="footnote-reference"><a href="#2">2</a></sup> today.</p> <p>Deliveries are an incredibly efficient means of distriuting goods<sup class="footnote-reference"><a href="#3">3</a></sup> and re-using glass bottles requires far less energy than say, melting a bottle down or worse creating a plastic single-use bottle.</p> <p>I really want to focus on the glass bottles because they are <em>so</em> efficient, why did we ever stop?</p> <p>Lobbying.</p> <h2 id="from-plastics-import-lobbying">from plastics import lobbying</h2> <p>That's right: lobbying, not the free market, gave us the fucking horrific amount of single-use plastic we have today.</p> <p>For sure there are a million things we need to cut the fuck out if we want this planet to be anything close to what we were born into in 100 years, but plastics are worth spending a few hundred words.</p> <p>The plastics industry spent millions<sup class="footnote-reference"><a href="#4">4</a></sup> lobbying to allow the use of single use plastics instead of reusable containers.</p> <p>Instead of identifying the inevitable crisis of too much fucking plastic and shutting that shit down, governments across the world wer just like "yeah this is cool, pour up my dude".</p> <p>Congress, given no external input, very well could have legislated against the use of single-use plastics, but because they were convinced... with money... otherwise here we are in a world full of single use plastics, and just like a virus it's spread so far that it feels impossible to slow it's spread.</p> <p>So clearly we need to stop lobbying. We need to think of some clever way to prevent corruptable government officials from being bought off to vote against the interests of their constituents, their country, and future generations...</p> <p>Well that's clearly impossible.</p> <p>Let's dig deeper. What causes lobbying?</p> <p>Profit.</p> <h2 id="from-lobbying-import-profit">from lobbying import profit</h2> <p>It is clear that profit is the reason lobbying exists.</p> <p>Some lobbying is good. If you own a farm and you hear about legislation that will like... just straight up dump a bunch of nuclear waste on your property, you're probably going to knock on some doors and grease some palms to prevent that from happening.</p> <p>Arithmetically it makes sense that lobbying is tied to profit. If a piece of legislation is projected to decrease profits by 30 million USD, you're probably willing to spend 2 million USD to prevent it from going through.</p> <p>Lobbying, like swordfishing<sup class="footnote-reference"><a href="#5">5</a></sup>, is a targeted, and therefore monetarily effective, way to influence change.</p> <p>Maximizing profit has a lot of other problems<sup class="footnote-reference"><a href="#6">6</a></sup> like supressing wages despite productivity going up, and cock-blocking public alternatives to private services like healthcare... Where was I?</p> <p>Oh right, so this begs the question, where does profit come from?</p> <h2 id="from-profit-import">from profit import ???</h2> <h2 id="soapbox-intensifies">soapbox intensifies</h2> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p>All <em>good faith actors</em> actors agree are a problem.</p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p>Yes, also mylks.</p> </div> <div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup> <p>Citation needed?</p> </div> <div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup> <p>... <em>sigh</em> citation needed.</p> </div> <div class="footnote-definition" id="5"><sup class="footnote-definition-label">5</sup> <p>The internet phishing scam not the oceanic activity.</p> </div> <div class="footnote-definition" id="6"><sup class="footnote-definition-label">6</sup> <p>... citation... needed...?</p> </div> 2020-11-13T00:00:00+00:00 2020-11-13T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/seagl-2020-lets-build-a-game-with-rust/ <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>This is the outline for a talk I gave at the Seattle GNU Linux conference (SeaGL) 2020. Once the video is posted I will link it here.</p> </span> </span> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>The code for this post can be found at https://github.com/pop/lets-make-games-with-rust.</p> </span> </span> <p>Like many of you, I really like games. I enjoy playing games, talking about games, and a few times I've even tried making games. I enjoy making games for a few reasons:</p> <ul> <li>Games are an interesting technical challenge.</li> <li>They are a flexible artistic outlet.</li> <li>I don't make games in my day-job (yay hobbies).</li> </ul> <p>I'm also interested in this programming language called Rust! You might have heard of it. Rust is a maturing systems programming language which aims to be performant, reliable, and productive.</p> <ul> <li>Performant in that it often goes head to head with C and C++ in benchmarks.</li> <li>Reliable in that it refuses to compile memory unsafe code.</li> <li>Productive because it includes "zero cost abstractions" (link <code>filter</code> and <code>map</code>) and awesome tooling.</li> </ul> <p>The community is pretty good too!</p> <p>This post is about bringing those two interests together. Let's learn how to build a videogame with the Rust programming language.</p> <div class="warning"> <span class="warning-title"> Warning </span> <span class="warning-body"> <p>This post is for folks who have a passing familiarity with Rust. If Rust is new to you, you get a little confused.</p> <p>You're a smart cookie though, I'm sure you'll keep up.</p> </span> </div> <h2 id="making-games-entities-components-systems-gear">Making Games: Entities, Components, Systems ⚙️</h2> <p>Games are an incredibly fun and flexible type of project to work on.</p> <p>At it's core, all games need a "game loop":</p> <ol> <li>Process input</li> <li>Transforms state</li> <li>Display new state</li> <li>Repeat</li> </ol> <p>Outside of that, the possibilities are endless. But while the <em>possibilities</em> are endless, there are a few <em>patterns</em> that lots of folks seem to gravitate toward.</p> <p>You could write a whole book on game programming patterns (and somebody has, links at the bottom). Today we're going to talk about one popular pattern: <strong>ECS: Entity Component System</strong>. Here's what that looks like:</p> <h3 id="components">Components</h3> <p>Pieces of data mixed, matched, and queried on.</p> <p>Example: Some components needed to simulate physics might be "Mass", "Movable", and "Friction"</p> <h3 id="entities">Entities</h3> <blockquote> <p>A Unique ID associated with a collection of Components.</p> </blockquote> <p>Example: Potted plant you can break have the following components:</p> <ul> <li><code>Sprite("/path/to/plant.png")</code></li> <li><code>Mass(6.8)</code></li> <li><code>Movable(True)</code></li> <li><code>Location((2, 5, 2))</code></li> </ul> <p>Each of these components are reusable. Instead of creating a "PottedPlant" class in code, we can define each entity in a config file like this:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>potted-plant.txt: </span><span>Sprite /path/to/plant.png </span><span>Mass 6.8 </span><span>Movable True </span><span>Location 2 5 2 </span></code></pre> <p>This makes our engine much more reusable and separates our content from our logic.</p> <p>But how do we use these components?</p> <h3 id="systems">Systems</h3> <p>Functions that operate on entities with specific components.</p> <p>A System queries for all entities with a specific subset of components and does some transformation on it.</p> <p>Example: a system that applies fire damage to an entity might look like this:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>entities_on_fire = entities.query(on_fire=True, health &gt; 0) </span><span>for entity in entities_on_fire: </span><span> entity.health -= 5 </span></code></pre> <p>This is nice compared with a classes-based approach where we would need to manage inheritance to manually make sure everything interacted correctly. Here we define systems based on what features an entity has. The resulting systems and components interact with eachother in interesting and potentially unexpected ways.</p> <p>ECS is a popular pattern for creating interactive games and simulations. Engines like Unity have some ECS patterns built in, and almost every big game engine uses ECS in some way.</p> <p>Of course ECS isn't a silver bullet, but for this blog post it's good enough.</p> <h2 id="tools-of-the-trade-c-and-c-hammer-pick">Tools of the Trade: C and C++ ⚒️</h2> <p>C and C++ are very popular languages in games programming. They are defacto in the industry and many large engines, both internal and licensed engines, are written in C++.</p> <p>I'm not here to bash on C and C++, but it can be useful to point out why you would bother using Rust if the norm is C++.</p> <p>The usual arguments in favor of Rust go a little something like this:</p> <ul> <li>Rust is memory safe; in Rust it is very difficult to reference freed memory, mutate memory in two threads, and dereference a null pointer.</li> <li>Rust is expressive; a lot of functional-programming language features exist in Rust without the usual run-time cost of those languages.</li> <li>Rust doesn't have any of the C/C++ legacy baggage but <em>can</em> inter-operate with C/C++ codebases.</li> <li>Rust has a kick-ass community and an ecosystem of battle-tested and safe code.</li> </ul> <p>So why does this all matter for games? I think of it it like this: Any project when it gets sufficiently complex benefits from Rust. Rust, by preventing a whole class of memory bugs, makes it easier to maintain a complex codebase over time. It might not be life or death, or as mission critical as security software, but completely avoiding null-pointer bugs, at essentially no performance cost, sounds like a huge weight off my shoulders.</p> <p>Games are by their nature huge and sprawling codebases. Many bugs in games are caught by a compiler, but even more errors would be caught by the Rust borrow-checker. And being able to use some of the nice functional-programming features would be nice too.</p> <p>Of course Rust is a relatively new language so your mileage may vary. If I ran a big game studio I don't think I would throw out my C++ code and start fresh with Rust, but I would definitely put some research and development into it for new projects (said the Rust fanboy).</p> <h2 id="rusty-games-hello-amethyst-gem">Rusty Games: Hello Amethyst 💎</h2> <p>Writing games in a safe, expressive, not C/C++ language sounds great; where do I start?</p> <p>You could write a game from scratch, but there are engines written in Rust you can use today! Some of these focus on ease of use, some are for 2D games, some focus on compiling for the web. Most of these engines require you to write Rust, as opposed to using a GUI, but even that is changing.</p> <p>For a comprehensive list check out <a href="https://www.arewegameyet.rs">https://www.arewegameyet.rs</a></p> <p>You could write this talk for almost any Rust Game Engine, but my personal favorite is Amethyst, so we'll use that. Amethyst has a solid API, very active community, and is a good mix of flexible, convenient, and powerful.</p> <p>Amethyst checks off a few other boxes:</p> <ul> <li>Implements an ECS runtime. Register Components, create Entities, and run Systems in Amethyst.</li> <li>Data driven design. Almost all data in Amethyst can be read in from a Config file.</li> <li>Apache + MIT licensed. Free as in speech is always nice.</li> </ul> <h3 id="step-0-join-the-cargo-cult">Step 0: Join the Cargo Cult</h3> <p>In this step we're going to get Rust setup and create a "hello world" Rust project.</p> <p>If you haven't already, setup your Rust toolchain and start a Rust project.</p> <ol> <li>Install <code>rustup</code>, the Rust toolchain manager.</li> <li>Run <code>rustup toolchain install stable</code> to install the latest stable Rust.</li> <li>Run <code>cargo new seagl-game</code> to create a "hello world" Rust application.</li> <li>Navigate to the new <code>seagl-game</code> folder. Add this to the end of our project's metadata file, <code>Cargo.toml</code>:</li> </ol> <pre style="background-color:#191919;color:#f8f8f2;"><code><span># Cargo.toml </span><span>[dependencies.amethyst] </span><span>version = &quot;0.15.1&quot; </span><span>features = [&quot;vulkan&quot;] # &quot;metal&quot; on MacOS </span></code></pre> <ol start="5"> <li>Run <code>cargo build</code> to build and cache our dependencies. You should see a <strong>bunch</strong> of output like this:</li> </ol> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ cargo build </span><span>... </span><span>Compiling either v1.6.1 </span><span>Compiling gimli v0.23.0 </span><span>Compiling adler v0.2.3 </span><span>Compiling object v0.22.0 </span><span>... </span></code></pre> <p>Now we have a "hello world" Rust project we can start building on.</p> <h3 id="step-1-draw-a-window-triangular-ruler">Step 1: Draw a Window 📐</h3> <p>Before we run, we need to walk. And before we walk we crawl. And before we crawl we draw a window. This is, of course, a little harder than just asking your computer "Please draw me a window".</p> <p>First we need to add a bunch of imports to our project:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#ff5e5e;">use </span><span>amethyst::{ </span><span> assets::{AssetStorage, Loader}, </span><span> core::{ </span><span> timing::Time, </span><span> transform::{Transform, TransformBundle}, </span><span> }, </span><span> derive::SystemDesc, </span><span> ecs::{ </span><span> Component, DenseVecStorage, Entities, Join, Read, ReadStorage, System, SystemData, </span><span> WriteStorage, </span><span> }, </span><span> input::{InputBundle, InputHandler, StringBindings}, </span><span> prelude::</span><span style="color:#ff5e5e;">*</span><span>, </span><span> renderer::{ </span><span> plugins::{RenderFlat2D, RenderToWindow}, </span><span> types::DefaultBackend, </span><span> Camera, ImageFormat, RenderingBundle, SpriteRender, SpriteSheet, SpriteSheetFormat, </span><span> Texture, </span><span> }, </span><span> utils::application_root_dir, </span><span>}; </span></code></pre> <p>This is every dependency we will need for the entire project, so if <code>cargo build|run</code> complains about unused dependencies, don't worry... we'll get there.</p> <p>Here we are including a few useful</p> <p>Then we need to add some boiler-plate to our <code>main</code> function:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#6d6d6d;">// This is necessary to make Rust&#39;s type-checker happy </span><span style="color:#6d6d6d;">// Our main function technically returns an Amethyst Result </span><span style="color:#6d6d6d;">// It can either return an Amethyst error or a unit value </span><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">main</span><span>() -&gt; amethyst::</span><span style="color:#fbe3bf;">Result</span><span>&lt;()&gt; { </span><span> </span><span style="color:#6d6d6d;">// Not required, but a logger very useful </span><span> amethyst::start_logger(</span><span style="color:#fbe3bf;">Default</span><span>::default()); </span><span> </span><span> </span><span style="color:#6d6d6d;">// Declare some useful variables used to tell Amethyst where our asset files and config files live </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> app_root </span><span style="color:#ff5e5e;">= </span><span style="color:#6699cc;">application_root_dir</span><span>()</span><span style="color:#ff5e5e;">?</span><span>; </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> assets_dir </span><span style="color:#ff5e5e;">=</span><span> app_root.</span><span style="color:#6699cc;">join</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">assets</span><span style="color:#ffffff;">&quot;</span><span>); </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> display_config_path </span><span style="color:#ff5e5e;">=</span><span> app_root.</span><span style="color:#6699cc;">join</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">config</span><span style="color:#ffffff;">&quot;</span><span>).</span><span style="color:#6699cc;">join</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">display.ron</span><span style="color:#ffffff;">&quot;</span><span>); </span><span> </span><span> </span><span style="color:#6d6d6d;">// Declare a renderer bundle </span><span> </span><span style="color:#6d6d6d;">// Amethyst adds this collection of 2D Render systems to our game&#39;s runtime </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> renderer </span><span style="color:#ff5e5e;">= </span><span>RenderingBundle::&lt;DefaultBackend&gt;::new() </span><span> .</span><span style="color:#6699cc;">with_plugin</span><span>( </span><span> RenderToWindow::from_config_path(display_config_path)</span><span style="color:#ff5e5e;">? </span><span> .</span><span style="color:#6699cc;">with_clear</span><span>([</span><span style="color:#fdb082;">1.00</span><span>, </span><span style="color:#fdb082;">0.33</span><span>, </span><span style="color:#fdb082;">0.00</span><span>, </span><span style="color:#fdb082;">1.0</span><span>]), </span><span> ).</span><span style="color:#6699cc;">with_plugin</span><span>(RenderFlat2D::default()); </span><span> </span><span> </span><span style="color:#6d6d6d;">// Build the game&#39;s systems </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> game_data </span><span style="color:#ff5e5e;">= </span><span>GameDataBuilder::default() </span><span> .</span><span style="color:#6699cc;">with_bundle</span><span>(renderer)</span><span style="color:#ff5e5e;">?</span><span>; </span><span> </span><span> </span><span style="color:#6d6d6d;">// Build our application, which includes our game data, where our assets live, and our starting state </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let </span><span style="color:#ff5e5e;">mut</span><span> game </span><span style="color:#ff5e5e;">= </span><span>Application::new(assets_dir, SeaglState, game_data)</span><span style="color:#ff5e5e;">?</span><span>; </span><span> </span><span> </span><span style="color:#6d6d6d;">// Run the game! </span><span> game.</span><span style="color:#6699cc;">run</span><span>(); </span><span> </span><span> </span><span style="color:#6d6d6d;">// Nothing bad happened, so return `()` </span><span> </span><span style="color:#fbe3bf;">Ok</span><span>(()) </span><span>} </span></code></pre> <p>That won't compile because we haven't defined our <code>SeaglState</code>.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ cargo run </span><span>... </span><span>error[E0425]: cannot find value `SeaglState` in this scope </span><span> --&gt; src/main.rs:17:49 </span><span> | </span><span>30 | let mut game = Application::new(assets_dir, SeaglState, game_data)?; </span><span> | ^^^^^^^^^^ not found in this scope </span></code></pre> <p>Let's add <code>SeaglState</code></p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#6d6d6d;">// States can store values, but for now we can use a unit-struct </span><span style="font-style:italic;color:#fbdfb5;">struct </span><span>SeaglState; </span><span> </span><span style="color:#6d6d6d;">// We get a working state for free by rubber-stamping the &quot;SimpleState&quot; struct onto our SeaglState </span><span style="color:#6d6d6d;">// We will implement our own logic for handling state start-up in the next step </span><span style="font-style:italic;color:#fbdfb5;">impl </span><span>SimpleState </span><span style="color:#ff5e5e;">for </span><span>SeaglState { } </span></code></pre> <p>We will add some methods to <code>SeaglState</code> later, but for now this makes Rust and Amethyst happy enough to compile.</p> <p>But if we run our code we get a wonderfully cryptic error message:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>Compiling seagl-talk v0.1.0 (/home/pop/seagl-talk) </span><span> Finished dev [unoptimized + debuginfo] target(s) in 24.81s </span><span> Running `target/debug/seagl-talk` </span><span>Error: Error { inner: Inner { source: None, backtrace: None, error: File(Os { code: 2, kind: NotFound, message: &quot;No such file or directory&quot; }) } } </span></code></pre> <p>We get errors like this when we have an unhandled exception in our code. In our <code>main</code> function, that is any place where we call a function with a <code>?</code>, e.g., <code>foo(...)?;</code>.</p> <p>TLDT (Too Long Didn't Troubleshoot) this is because we haven't created our display config file!</p> <p>Add a new file <code>display.ron</code> in a new folder called <code>config/</code>:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#6d6d6d;">// config/display.ron </span><span>( </span><span> title: </span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">SeaGL!</span><span style="color:#ffffff;">&quot;</span><span>, </span><span> dimensions: </span><span style="color:#fbe3bf;">Some</span><span>((</span><span style="color:#fdb082;">500</span><span>, </span><span style="color:#fdb082;">500</span><span>)), </span><span>) </span></code></pre> <p>Now when we <code>cargo run</code> we should get a wonderful orange window:</p> <p><img src="/images/seagl-2020/blank-window.png" alt="It worked! We drew a window." /></p> <h3 id="step-2-draw-a-seagl-dove">Step 2: Draw a SeaGL 🕊️</h3> <p>Alas, we have a window but no game! Let's draw our first character to the screen.</p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>Did you know that SeaGL's mascot is named Patch? https://seagl.org/news/2020/09/10/naming-contest.html</p> </span> </span> <p>First we'll create a Component for our Seagl.</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#e9fdac;">derive</span><span>(Default)] </span><span style="color:#ff5e5e;">pub </span><span style="font-style:italic;color:#fbdfb5;">struct </span><span>Seagl; </span><span> </span><span style="font-style:italic;color:#fbdfb5;">impl </span><span>Component </span><span style="color:#ff5e5e;">for </span><span>Seagl { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">type </span><span>Storage </span><span style="color:#ff5e5e;">= </span><span>DenseVecStorage&lt;</span><span style="font-style:italic;color:#fbdfb5;">Self</span><span>&gt;; </span><span>} </span></code></pre> <p>Next we'll create a Seagl entity.</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">impl </span><span>SimpleState </span><span style="color:#ff5e5e;">for </span><span>SeaglState { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">on_start</span><span>(</span><span style="color:#ff5e5e;">&amp;mut </span><span style="font-style:italic;color:#fc9354;">self</span><span>, </span><span style="font-style:italic;color:#fc9354;">data</span><span>: StateData&lt;GameData&gt;) { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let </span><span style="color:#ff5e5e;">mut</span><span> transform </span><span style="color:#ff5e5e;">= </span><span>Transform::default(); </span><span> transform.</span><span style="color:#6699cc;">set_translation_xyz</span><span>(</span><span style="color:#fdb082;">50.0</span><span>, </span><span style="color:#fdb082;">50.0</span><span>, </span><span style="color:#fdb082;">0.0</span><span>); </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> seagl </span><span style="color:#ff5e5e;">= </span><span>Seagl::default(); </span><span> data.world </span><span> .</span><span style="color:#6699cc;">create_entity</span><span>() </span><span> .</span><span style="color:#6699cc;">with</span><span>(seagl) </span><span> .</span><span style="color:#6699cc;">with</span><span>(transform) </span><span> .</span><span style="color:#6699cc;">build</span><span>(); </span><span> } </span><span>} </span></code></pre> <p>This is a good start, but our Seagl is a spriteless ghost!</p> <div class="warning"> <span class="warning-title"> Warning </span> <span class="warning-body"> <p>Seagull ghosts are terrifying. Add a sprite!</p> </span> </div> <p>First we need to load the spritesheet into memory. Add this in our <em>on_start</em> function above where we added the seagl:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">let</span><span> sprite_sheet_handle </span><span style="color:#ff5e5e;">= </span><span>{ </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> loader </span><span style="color:#ff5e5e;">=</span><span> data.world.read_resource::&lt;Loader&gt;(); </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> texture_storage </span><span style="color:#ff5e5e;">=</span><span> data.world.read_resource::&lt;AssetStorage&lt;Texture&gt;&gt;(); </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> texture_handle </span><span style="color:#ff5e5e;">=</span><span> loader.</span><span style="color:#6699cc;">load</span><span>( </span><span> </span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">texture/spritesheet.png</span><span style="color:#ffffff;">&quot;</span><span>, </span><span> ImageFormat::default(), </span><span> (), </span><span> </span><span style="color:#ff5e5e;">&amp;</span><span>texture_storage, </span><span> ); </span><span> </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> sprite_sheet_store </span><span style="color:#ff5e5e;">=</span><span> data.world.read_resource::&lt;AssetStorage&lt;SpriteSheet&gt;&gt;(); </span><span> loader.</span><span style="color:#6699cc;">load</span><span>( </span><span> </span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">texture/spritesheet.ron</span><span style="color:#ffffff;">&quot;</span><span>, </span><span> SpriteSheetFormat(texture_handle), </span><span> (), </span><span> </span><span style="color:#ff5e5e;">&amp;</span><span>sprite_sheet_store, </span><span> ) </span><span>}; </span></code></pre> <p>Then modify our Seagl entity like so:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span style="color:#a6e22e;">++ main.rs </span><span>@@ impl SimpleState for SeaglState </span><span>@@ fn on_start(...) </span><span> let mut transform = Transform::default(); </span><span> transform.set_translation_xyz(50.0, 50.0, 0.0); </span><span style="color:#a6e22e;">+ let sprite = SpriteRender::new(sprite_sheet_handle.clone(), 0); </span><span> let seagl = Seagl::default(); </span><span> data.world </span><span> .create_entity() </span><span> .with(seagl) </span><span style="color:#a6e22e;">+ .with(sprite) </span><span> .with(transform) </span><span> .build(); </span></code></pre> <p>Let's see. We created a Seagl entity. Let's try running this thing:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ cargo run </span><span>... </span><span>thread &#39;main&#39; panicked at &#39;Tried to fetch resource of type `MaskedStorage&lt;Seagl&gt;`[^1] from the `World`, but the resource does not exist. </span><span> </span><span>You may ensure the resource exists through one of the following methods: </span><span> </span><span>* Inserting it when the world is created: `world.insert(..)`. </span><span>* If the resource implements `Default`, include it in a system&#39;s `SystemData`, and ensure the system is registered in the dispatcher. </span><span>* If the resource does not implement `Default`, insert it in the world during `System::setup`. </span></code></pre> <p>Hmm. It seems like our <code>Seagl</code> Component isn't registered with Amethyst. This happens implicitly when we add a system that uses our component, but until we write a System, we'll have to explicitly register our Component with Amethyst.</p> <p>Add this toward the top of our <code>on_start</code> method:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span>data.world.register::&lt;Seagl&gt;(); </span></code></pre> <p>Let's try running again:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ cargo run </span><span>... </span><span>Error { inner: Inner { source: Some(Error { inner: Inner { source: None, backtrace: None, </span><span>error: Os { code: 2, kind: NotFound, message: &quot;No such file or directory&quot; } } }), backtrace: None, </span><span>error: StringError(&quot;Failed to fetch metadata for \&quot;/home/pop/seagl-talk/assets/texture/spritesheet.ron\&quot;&quot;) } } </span></code></pre> <p>Ah, a different runtime error. This time we forgot to add our spritesheet image and spritesheet config file. Lets add those.</p> <p>Add this code to a file in <code>assets/texture/spritesheet.ron</code>:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#6d6d6d;">// assets/texture/spritesheet.ron </span><span>List(( </span><span> texture_width: </span><span style="color:#fdb082;">32</span><span>, </span><span> texture_height: </span><span style="color:#fdb082;">16</span><span>, </span><span> sprites: [ </span><span> ( </span><span style="color:#6d6d6d;">// Seagl </span><span> x: </span><span style="color:#fdb082;">0</span><span>, </span><span> y: </span><span style="color:#fdb082;">0</span><span>, </span><span> width: </span><span style="color:#fdb082;">16</span><span>, </span><span> height: </span><span style="color:#fdb082;">16</span><span>, </span><span> ), </span><span> ( </span><span style="color:#6d6d6d;">// Burger </span><span> x: </span><span style="color:#fdb082;">16</span><span>, </span><span> y: </span><span style="color:#fdb082;">0</span><span>, </span><span> width: </span><span style="color:#fdb082;">10</span><span>, </span><span> height: </span><span style="color:#fdb082;">8</span><span>, </span><span> ), </span><span> ], </span><span>)) </span></code></pre> <p>And save this image to <code>assets/texture/spritesheet.png</code>:</p> <p><img src="/images/seagl-2020/spritesheet.png" alt="Seagl and Burger. 32x16. Pixel on LCD." /></p> <p>Now if we run <code>cargo run</code> we get the same blank orange window. This happened because we forgot to add a Camera to the scene!</p> <p>Add this to the end of our <code>on_start</code> function:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">let </span><span style="color:#ff5e5e;">mut</span><span> transform </span><span style="color:#ff5e5e;">= </span><span>Transform::default(); </span><span>transform.</span><span style="color:#6699cc;">set_translation_xyz</span><span>(</span><span style="color:#fdb082;">50.0</span><span>, </span><span style="color:#fdb082;">50.0</span><span>, </span><span style="color:#fdb082;">1.0</span><span>); </span><span>data.world </span><span> .</span><span style="color:#6699cc;">create_entity</span><span>() </span><span> .</span><span style="color:#6699cc;">with</span><span>(Camera::standard_2d(</span><span style="color:#fdb082;">100.0</span><span>, </span><span style="color:#fdb082;">100.0</span><span>)) </span><span> .</span><span style="color:#6699cc;">with</span><span>(transform) </span><span> .</span><span style="color:#6699cc;">build</span><span>(); </span></code></pre> <p><img src="/images/seagl-2020/window-with-seagl.png" alt="That&#39;s a nice looking Seagl there..." /></p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>It's so beautifull...</p> </span> </span> <h3 id="step-3-move-around-horse-racing">Step 3: Move Around 🏇</h3> <p>Thinking back to our ECS discussion, we have two of the three ingredients: an Entity, some Components, but no Systems!</p> <p>First, we need to create a System struct and implement <code>System</code> on it.</p> <p>Our System's run function looks like this in psuedocode:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>for every seagl that can move: </span><span> If the user input was to move horizontal: </span><span> Move the seagl horizontally </span><span> If the user input was to move vertical: </span><span> Move the seagl vertically </span></code></pre> <p>This doesn't look <em>exactly</em> the same in Rust, but it's pretty close.</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span>[</span><span style="color:#6699cc;">derive</span><span>(SystemDesc)] </span><span style="color:#ff5e5e;">pub </span><span style="font-style:italic;color:#fbdfb5;">struct </span><span>MoveSystem; </span><span> </span><span style="font-style:italic;color:#fbdfb5;">impl</span><span>&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>&gt; System&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>&gt; </span><span style="color:#ff5e5e;">for </span><span>MoveSystem { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">type </span><span>SystemData </span><span style="color:#ff5e5e;">= </span><span>( </span><span> WriteStorage&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>, Transform&gt;, </span><span> ReadStorage&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>, Seagl&gt;, </span><span> Read&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>, Time&gt;, </span><span> Read&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>, InputHandler&lt;StringBindings&gt;&gt;, </span><span> ); </span><span> </span><span> </span><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">run</span><span>(</span><span style="color:#ff5e5e;">&amp;mut </span><span style="font-style:italic;color:#fc9354;">self</span><span>, (</span><span style="font-style:italic;color:#fc9354;">mut transforms</span><span>, </span><span style="font-style:italic;color:#fc9354;">seagls</span><span>, </span><span style="font-style:italic;color:#fc9354;">time</span><span>, </span><span style="font-style:italic;color:#fc9354;">input</span><span>): </span><span style="font-style:italic;color:#fbdfb5;">Self::</span><span>SystemData) { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">let</span><span> speed: </span><span style="font-style:italic;color:#fbdfb5;">f32 </span><span style="color:#ff5e5e;">= </span><span style="color:#fdb082;">50.0</span><span>; </span><span> </span><span style="color:#ff5e5e;">for </span><span>(_seagl, transform) </span><span style="color:#ff5e5e;">in </span><span>(</span><span style="color:#ff5e5e;">&amp;</span><span>seagls, </span><span style="color:#ff5e5e;">&amp;mut</span><span> transforms).</span><span style="color:#6699cc;">join</span><span>() { </span><span> </span><span style="color:#ff5e5e;">if </span><span style="font-style:italic;color:#fbdfb5;">let </span><span style="color:#fbe3bf;">Some</span><span>(horizontal) </span><span style="color:#ff5e5e;">=</span><span> input.</span><span style="color:#6699cc;">axis_value</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">horizontal</span><span style="color:#ffffff;">&quot;</span><span>) { </span><span> transform.</span><span style="color:#6699cc;">prepend_translation_x</span><span>( </span><span> horizontal </span><span style="color:#ff5e5e;">*</span><span> time.</span><span style="color:#6699cc;">delta_seconds</span><span>() </span><span style="color:#ff5e5e;">*</span><span> speed </span><span style="color:#ff5e5e;">as </span><span style="font-style:italic;color:#fbdfb5;">f32 </span><span> ); </span><span> </span><span> }; </span><span> </span><span style="color:#ff5e5e;">if </span><span style="font-style:italic;color:#fbdfb5;">let </span><span style="color:#fbe3bf;">Some</span><span>(vertical) </span><span style="color:#ff5e5e;">=</span><span> input.</span><span style="color:#6699cc;">axis_value</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">vertical</span><span style="color:#ffffff;">&quot;</span><span>) { </span><span> transform.</span><span style="color:#6699cc;">prepend_translation_y</span><span>( </span><span> vertical </span><span style="color:#ff5e5e;">*</span><span> time.</span><span style="color:#6699cc;">delta_seconds</span><span>() </span><span style="color:#ff5e5e;">*</span><span> speed </span><span style="color:#ff5e5e;">as </span><span style="font-style:italic;color:#fbdfb5;">f32 </span><span> ); </span><span> }; </span><span> } </span><span> } </span><span>} </span></code></pre> <p>We declare a <code>SystemData</code> type which is a tuple of components. The <code>Transform</code> component will be modified, so we require it as <code>mut</code>, but everything else is <code>Read</code> for stuff that Amethyst provides and <code>ReadStorage</code> for things we created.</p> <p>We loop over every entity with the <code>Seagl</code> and <code>Transform</code> components, then we match against any user input:</p> <ul> <li>If we had "vertical" input, move the entity on the x axis.</li> <li>If we had "horizontal" input, move the entity on the y axis.</li> <li>We don't need to explicitly say "move left"/"move right" because the horizontal/vertical inputs can be positive or negative.</li> </ul> <p>Next we need to register this system with out game. Because we are using Inputs we also need to register the inputs bundle with the game.</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span style="color:#75715e;">+++ main.rs </span><span>@@ fn main() -&gt; amethyst::Result&lt;()&gt; </span><span> ) </span><span> .with_plugin(RenderFlat2D::default()); </span><span> </span><span style="color:#a6e22e;">+ let bindings_path = app_root.join(&quot;config&quot;).join(&quot;bindings.ron&quot;); </span><span style="color:#a6e22e;">+ let inputs = InputBundle::&lt;StringBindings&gt;::new().with_bindings_from_file(bindings_path)?; </span><span style="color:#a6e22e;">+ </span><span> let game_data = GameDataBuilder::default() </span><span> .with_bundle(transform)? </span><span> .with_bundle(renderer)? </span><span style="color:#a6e22e;">+ .with_bundle(inputs)? </span><span style="color:#a6e22e;">+ .with(MoveSystem, &quot;move_system&quot;, &amp;[&quot;input_system&quot;]); </span><span> </span><span> let mut game = Application::new(assets_dir, SeaglState, game_data)?; </span></code></pre> <p>We have a dependency on the <code>input_system</code>, so Amethyst will ensure that system runs before <code>move_system</code>.</p> <p>Next we need to create a config file for our movement bindings. Instead of hard-coding "Up arrow means move up, down arrow means down" we put that in config files:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#6d6d6d;">// config/bindings.ron </span><span>( </span><span> axes: { </span><span> </span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">horizontal</span><span style="color:#ffffff;">&quot;</span><span>: Emulated(pos: Key(Right), neg: Key(Left)), </span><span> </span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">vertical</span><span style="color:#ffffff;">&quot;</span><span>: Emulated(pos: Key(Up), neg: Key(Down)), </span><span> }, </span><span> actions: {}, </span><span>) </span></code></pre> <p>![It moves!](/images/seagl-2020/SeaGL-move.gif" class="align-center)</p> <p>This is a good start, but you'll notice the Seagl doesn't turn left and right, this <em>totally</em> breaks my suspension of disbelief so we're gonna need to fix that in our <code>run</code> method:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span>diff --git a/src/main.rs b/src/main.rs </span><span>@@ impl&lt;&#39;s&gt; System&lt;&#39;s&gt; for MoveSystem </span><span>@@ run(...) </span><span> if let Some(vertical) = input.axis_value(&quot;vertical&quot;) { </span><span> transform.prepend_translation_x( </span><span> horizontal * time.delta_seconds() * speed as f32 </span><span> ); </span><span style="color:#a6e22e;">+ if horizontal &gt; 0.0 { </span><span style="color:#a6e22e;">+ transform.set_rotation_y_axis(std::f32::consts::PI); </span><span style="color:#a6e22e;">+ } </span><span style="color:#a6e22e;">+ if horizontal &lt; 0.0 { </span><span style="color:#a6e22e;">+ transform.set_rotation_y_axis(0.0); </span><span style="color:#a6e22e;">+ } </span><span style="color:#a6e22e;">+ </span><span> }; </span><span> if let Some(vertical) = input.axis_value(&quot;vertical&quot;) { </span><span> transform.prepend_translation_y( </span></code></pre> <p>In our "horizontal" check we added:</p> <ul> <li>If the input was greater than 0, flip our sprite on the Y axis.</li> <li>If the input was less than 0, reset our sprite on the Y axis.</li> </ul> <p>This makes it look like our Seagl is facing the direction they're moving which should help boost our Metacritic score when we publish this at the end of the blogpost.</p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>We rotate by PI because our 2D sprite is in the 3D world and we're rotating it in radians.</p> <p>Do you ever feel like a 2D sprite in a 3D world? I know I do...</p> </span> </span> <p><img src="/images/seagl-2020/SeaGL-move-look.gif" alt="It moves!" /></p> <h3 id="step-4-eat-some-food-hamburger">Step 4: Eat some food! 🍔</h3> <p>I'm sure we could all get <em>minutes</em> of fun out of moving our seagl around the screen, but this game could really use something else... Something tastier.</p> <p>Let's add burgers.</p> <p>This will require us to do everything we just did, again:</p> <ul> <li>Add a Food Compnent.</li> <li>Create a Burger entity with the food component.</li> <li>Add an Eat system.</li> <li>Register our Eat system with the game.</li> </ul> <p>First we need to add a food Component.</p> <p>Add this component anywhere that feels right:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#e9fdac;">derive</span><span>(Default)] </span><span style="color:#ff5e5e;">pub </span><span style="font-style:italic;color:#fbdfb5;">struct </span><span>Food; </span><span> </span><span style="font-style:italic;color:#fbdfb5;">impl </span><span>Component </span><span style="color:#ff5e5e;">for </span><span>Food { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">type </span><span>Storage </span><span style="color:#ff5e5e;">= </span><span>DenseVecStorage&lt;</span><span style="font-style:italic;color:#fbdfb5;">Self</span><span>&gt;; </span><span>} </span></code></pre> <p>It's structurally identical to our Seagl, but with a different <code>struct</code> it's a totally different component.</p> <p>With a Food component we can add our Burger entity. Add this code to our <code>on_setup</code> function at the end:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="font-style:italic;color:#fbdfb5;">let</span><span> burger_sprite </span><span style="color:#ff5e5e;">= </span><span>SpriteRender::new(sprite_sheet_handle.</span><span style="color:#6699cc;">clone</span><span>(), </span><span style="color:#fdb082;">1</span><span>); </span><span style="font-style:italic;color:#fbdfb5;">let </span><span style="color:#ff5e5e;">mut</span><span> transform </span><span style="color:#ff5e5e;">= </span><span>Transform::default(); </span><span>transform.</span><span style="color:#6699cc;">set_translation_xyz</span><span>(</span><span style="color:#fdb082;">75.0</span><span>, </span><span style="color:#fdb082;">75.0</span><span>, </span><span style="color:#ff5e5e;">-</span><span style="color:#fdb082;">1.0</span><span>); </span><span>data.world </span><span> .</span><span style="color:#6699cc;">create_entity</span><span>() </span><span> .</span><span style="color:#6699cc;">with</span><span>(Food::default()) </span><span> .</span><span style="color:#6699cc;">with</span><span>(burger_sprite) </span><span> .</span><span style="color:#6699cc;">with</span><span>(transform) </span><span> .</span><span style="color:#6699cc;">build</span><span>(); </span></code></pre> <p>We create an entity spawning it at the point (75, 75, -1).</p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>We spawn the burger at <code>z=-1</code> to ensure the Seagl sprite is closer to the camera and thus is drawn on top of the burger.</p> <p>Have you ever seen a Seagull <em>behind</em> a burger? That's ridiculous.</p> </span> </span> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>A few exercises left to the reader:</p> <ol> <li>How would you spawn multiple burgers?</li> <li>How would you re-spawn burgers when one is eaten?</li> <li>How would you keep track of how many burgers were eaten?</li> <li>How would you display the number of burgers eaten?</li> </ol> <p>I've only covered enough in this post to answer the first two.</p> </span> </span> <p>And finally an "eat" system.</p> <p>This system's pseudocode looks like this:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>For each seagl with a location: </span><span> For each Food with a location: </span><span> If the Seagl overlaps with the Food: </span><span> Destory that food </span></code></pre> <p>This is a bit of a hack. If this were a real game we would keep track of how many burgers the Seagl ate, but for this demo, we'll be lazy:</p> <pre data-lang="rust" style="background-color:#191919;color:#f8f8f2;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#ff5e5e;">pub </span><span style="font-style:italic;color:#fbdfb5;">struct </span><span>EatSystem; </span><span> </span><span style="font-style:italic;color:#fbdfb5;">impl</span><span>&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>&gt; System&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>&gt; </span><span style="color:#ff5e5e;">for </span><span>EatSystem { </span><span> </span><span style="font-style:italic;color:#fbdfb5;">type </span><span>SystemData </span><span style="color:#ff5e5e;">= </span><span>( </span><span> ReadStorage&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>, Transform&gt;, </span><span> ReadStorage&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>, Seagl&gt;, </span><span> ReadStorage&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>, Food&gt;, </span><span> Entities&lt;</span><span style="color:#ff5e5e;">&#39;s</span><span>&gt;, </span><span> ); </span><span> </span><span> </span><span style="font-style:italic;color:#fbdfb5;">fn </span><span style="color:#8cdaff;">run</span><span>(</span><span style="color:#ff5e5e;">&amp;mut </span><span style="font-style:italic;color:#fc9354;">self</span><span>, (</span><span style="font-style:italic;color:#fc9354;">transforms</span><span>, </span><span style="font-style:italic;color:#fc9354;">seagls</span><span>, </span><span style="font-style:italic;color:#fc9354;">foods</span><span>, </span><span style="font-style:italic;color:#fc9354;">entities</span><span>): </span><span style="font-style:italic;color:#fbdfb5;">Self::</span><span>SystemData) { </span><span> </span><span style="color:#ff5e5e;">for </span><span>(_seagl, seagl_pos) </span><span style="color:#ff5e5e;">in </span><span>(</span><span style="color:#ff5e5e;">&amp;</span><span>seagls, </span><span style="color:#ff5e5e;">&amp;</span><span>transforms).</span><span style="color:#6699cc;">join</span><span>() { </span><span> </span><span style="color:#ff5e5e;">for </span><span>(_food, food_pos, entity) </span><span style="color:#ff5e5e;">in </span><span>(</span><span style="color:#ff5e5e;">&amp;</span><span>foods, </span><span style="color:#ff5e5e;">&amp;</span><span>transforms, </span><span style="color:#ff5e5e;">&amp;</span><span>entities).</span><span style="color:#6699cc;">join</span><span>() { </span><span> </span><span style="color:#6d6d6d;">// https://developer.mozilla.org/en-US/docs/Games/Techniques/2D_collision_detection </span><span> </span><span style="color:#ff5e5e;">if </span><span>(seagl_pos.</span><span style="color:#6699cc;">translation</span><span>().x </span><span style="color:#ff5e5e;">&lt;</span><span> food_pos.</span><span style="color:#6699cc;">translation</span><span>().x </span><span style="color:#ff5e5e;">+ </span><span style="color:#fdb082;">5.0</span><span>) </span><span style="color:#ff5e5e;">&amp;&amp; </span><span> (seagl_pos.</span><span style="color:#6699cc;">translation</span><span>().x </span><span style="color:#ff5e5e;">+ </span><span style="color:#fdb082;">8.0 </span><span style="color:#ff5e5e;">&gt;</span><span> food_pos.</span><span style="color:#6699cc;">translation</span><span>().x) </span><span style="color:#ff5e5e;">&amp;&amp; </span><span> (seagl_pos.</span><span style="color:#6699cc;">translation</span><span>().y </span><span style="color:#ff5e5e;">&lt;</span><span> food_pos.</span><span style="color:#6699cc;">translation</span><span>().y </span><span style="color:#ff5e5e;">+ </span><span style="color:#fdb082;">4.0</span><span>) </span><span style="color:#ff5e5e;">&amp;&amp; </span><span> (seagl_pos.</span><span style="color:#6699cc;">translation</span><span>().y </span><span style="color:#ff5e5e;">+ </span><span style="color:#fdb082;">8.0 </span><span style="color:#ff5e5e;">&gt;</span><span> food_pos.</span><span style="color:#6699cc;">translation</span><span>().y) </span><span> { </span><span> entities.</span><span style="color:#6699cc;">delete</span><span>(entity).</span><span style="color:#6699cc;">unwrap</span><span>(); </span><span> } </span><span> } </span><span> } </span><span> } </span><span>} </span></code></pre> <p>And last but not least, we need to register this system with our game:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span style="color:#75715e;">+++ main.rs </span><span>@@ fn main() -&gt; amethyst::Result&lt;()&gt; </span><span> let game_data = GameDataBuilder::default() </span><span> .with_bundle(transform)? </span><span> .with_bundle(renderer)? </span><span> .with_bundle(inputs)? </span><span> .with(MoveSystem, &quot;move_system&quot;, &amp;[&quot;input_system&quot;]) </span><span style="color:#a6e22e;">+ .with(EatSystem, &quot;eat_system&quot;, &amp;[&quot;move_system&quot;]); </span></code></pre> <p><img src="/images/seagl-2020/SeaGL-move-look-burger.gif" class="align-center" alt="It moves!" /></p> <h2 id="conclusions">Conclusions</h2> <p>We did it. We made a lil' game. It had a Seagl and a burger. And we had fun making it.</p> <p>I wouldn't say it's <em>easy</em> to make games in Rust, but we are <em>very far</em> from having to write games from scratch.</p> <p>If this post piqued your interest I hope you check out <a href="https://arewegameyet.rs">https://arewegameyet.rs</a> to learn more about the Rust Games ecosystem, and <a href="https://amethyst.rs">https://amethyst.rs</a> to learn more about this budding Game Engine.</p> <h2 id="links">Links</h2> <ul> <li>SeaGL conference website: <a href="https://seagl.org/">https://seagl.org/</a> (You should go if you're in the Pacific Northwest)</li> <li>The code for this post is avaliable at <a href="https://github.com/pop/lets-make-games-with-rust">https://github.com/pop/lets-make-games-with-rust</a>. I even tagged each step so you can see exactly what we added!</li> <li>Rust Language: <a href="https://www.rust-lang.org/">https://www.rust-lang.org/</a></li> <li>Are We Game Yet?: <a href="https://arewegameyet.rs/">https://arewegameyet.rs/</a></li> <li>Amethyst Game Engine website: <a href="https://amethyst.rs/">https://amethyst.rs/</a></li> <li>Amethyst Game Engine book has a great introduction and overview: <a href="https://book.amethyst.rs/stable/">https://book.amethyst.rs/stable/</a></li> <li>Bevy Game Engine is an interesting iteration on Game Engines in Rust: <a href="https://bevyengine.org/">https://bevyengine.org/</a></li> <li>My Source that C++ is the defacto language in the games industry: <a href="https://youtu.be/rX0ItVEVjHc">https://youtu.be/rX0ItVEVjHc</a></li> <li>Game Programming Patterns is an awesome book with a free &amp; legal copy online: <a href="https://gameprogrammingpatterns.com/">https://gameprogrammingpatterns.com/</a></li> <li><code>rustup</code> homepage for installation instructions: <a href="https://rustup.rs/">https://rustup.rs/</a></li> </ul> 2018-10-13T00:00:00+00:00 2018-10-13T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/kubernetes-as-an-external-service-proxy/ <p>Say you have a firewall restriction that creates the following situation:</p> <ol> <li>App1 cannot communicate directly with App2.</li> <li>App1 and App2 can both talk to a Kubernets cluster.</li> <li>Neither app is hosted on the Kubernetes cluster.</li> <li>How can you get messages between App1 and App2?</li> </ol> <p>The way the problem is stated makes it pretty obvious that the solution <em>involves</em> using a Kubernetes cluster, but how exactly?</p> <p>The naive solution might be to spin up a container which acts as a proxy; Nginx comes to mind. This would definitely work, but I am exceedingly lazy and don't want to learn how to configure Nginx. In fact, the solution I came to doesn't involve running any new pods!</p> <p>Here's the code. Below I'll explain what's happening here and why it works.</p> <pre data-lang="yaml" style="background-color:#191919;color:#f8f8f2;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">Endpoints </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">myapp-proxy </span><span style="color:#ff5e5e;">subsets</span><span>: </span><span>- </span><span style="color:#ff5e5e;">addresses</span><span>: </span><span> - </span><span style="color:#ff5e5e;">ip</span><span>: </span><span style="color:#fbe3bf;">2.3.4.5k </span><span style="color:#6d6d6d;"># App2&#39;s address </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">port</span><span>: </span><span style="color:#fdb082;">8080 </span><span style="color:#6d6d6d;"># App2&#39;s service port </span><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">v1 </span><span> </span><span>--- </span><span> </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">Service </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">myapp-proxy </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">type</span><span>: </span><span style="color:#fbe3bf;">LoadBalancer </span><span> </span><span style="color:#ff5e5e;">loadBalancerSourceRanges</span><span>: </span><span> - </span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">1.2.3.4/32</span><span style="color:#ffffff;">&quot; </span><span style="color:#6d6d6d;"># App1&#39;s address </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">protocol</span><span>: </span><span style="color:#fbe3bf;">TCP </span><span> </span><span style="color:#ff5e5e;">port</span><span>: </span><span style="color:#fdb082;">80 </span><span style="color:#6d6d6d;"># Redirect traffic hitting 80 to the app&#39;s service port </span><span> </span><span style="color:#ff5e5e;">targetPort</span><span>: </span><span style="color:#fdb082;">8080 </span><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">v1 </span><span> </span><span>--- </span><span> </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">Ingress </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">restaurant-proxy </span><span style="color:#ff5e5e;">spec</span><span>: </span><span style="color:#ff5e5e;">rules</span><span>: </span><span> - </span><span style="color:#ff5e5e;">host</span><span>: </span><span style="color:#fbe3bf;">myapp-proxy.somehost.net </span><span> </span><span style="color:#ff5e5e;">http</span><span>: </span><span> </span><span style="color:#ff5e5e;">paths</span><span>: </span><span> - </span><span style="color:#ff5e5e;">path</span><span>: </span><span style="color:#fbe3bf;">/ </span><span> </span><span style="color:#ff5e5e;">backend</span><span>: </span><span> </span><span style="color:#ff5e5e;">serviceName</span><span>: </span><span style="color:#fbe3bf;">myapp-proxy </span><span> </span><span style="color:#ff5e5e;">servicePort</span><span>: </span><span style="color:#fdb082;">80 </span><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">extensions/v1beta1 </span></code></pre> <h2 id="endpoints">Endpoints</h2> <p>Endpoints are the Kubernetes abstraction for IPs+Ports running the same application. It's how you group together N instances of an app into one pool.</p> <p>Under the hood Endpoints get created as a pre-requisite for every Service you deploy. You don't usually need to deal with these directly as they are created implicitly whenever a Deployment gets applied.</p> <p>By manually creating an endpoint we have imported our non-kubernetes app into Kubernetes. That means we can do Kubernetes things with it like expose it via a Service or even put it behind an Ingress Pretty neat!</p> <h2 id="service">Service</h2> <p>Services are how we expose an endpoint to the world. Most cloud providers will give you a public IP address for a service and load balance across all of that service's endpoints.</p> <p>This is as far as we need to proxy traffic between our two Apps. App1 makes a request to whatever IP Kubernetes gets for the <code>myapp-proxy</code> Service and relays it to the <code>myapp-proxy</code> endpoint, which ultimately routes the traffic to App2. What's really cool is that the endpoint that <strong>is</strong> App2 can be an self-hosted Virtual Machine, as long as the IP doesn't change this proxy will continue to work.</p> <h2 id="ingress">Ingress</h2> <p>Ingresses are my favorite part of Kubernetes. They're very convenient, incredibly powerful, and they work like... over half the time.</p> <p>While not strictly necessary, this Ingress gives us some nice-to-haves.</p> <ul> <li>Gives us an easy to manage host name via something like <a href="https://github.com/kubernetes-incubator/external-dns">External DNS</a>.</li> <li>Could be extended to terminate SSL, again "for free", with something like <a href="https://github.com/jetstack/cert-manager">Cert Manager</a>.</li> </ul> <p>So that's how we use Kubernetes to manage services (lower-case 's') which aren't running in Pods.</p> 2018-06-23T00:00:00+00:00 2018-06-23T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/chef-dynamic-attributes/ <h2 id="problem">Problem</h2> <p>You are trying to set a node attribute in a resource block but it isn't working.</p> <p>For example you have the following code:</p> <pre data-lang="ruby" style="background-color:#191919;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span>ruby_block </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">foo</span><span style="color:#ffffff;">&#39; </span><span style="color:#ff5e5e;">do </span><span> block </span><span style="color:#ff5e5e;">do </span><span> </span><span style="color:#6d6d6d;"># some ruby </span><span> node.override[</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">var</span><span style="color:#ffffff;">&#39;</span><span>] </span><span style="color:#ff5e5e;">= </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">value</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">end </span><span> action </span><span style="color:#fdb082;">:run </span><span style="color:#ff5e5e;">end </span><span> </span><span style="color:#ff5e5e;">if</span><span> node[</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">var</span><span style="color:#ffffff;">&#39;</span><span>] </span><span style="color:#ff5e5e;">== </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">value</span><span style="color:#ffffff;">&#39; </span><span> notifies </span><span style="color:#fdb082;">:run</span><span>, some_resource[</span><span style="color:#6699cc;">name</span><span>] </span><span style="color:#ff5e5e;">else </span><span> notifies </span><span style="color:#fdb082;">:run</span><span>, other_resource[whatever] </span><span style="color:#ff5e5e;">end </span></code></pre> <p>Even though you're setting <code>node['var']</code>, <code>other_resource[whatever]</code> is getting notified to <code>:run</code>. What gives?</p> <h2 id="solution">Solution</h2> <p>To jump ahead, because you're skimming this post for the answer, you're going to want to do something like this:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span># go-go-gadget diffs </span><span># Chef Version 13.5.8 </span><span> </span><span> ruby_block &#39;foo&#39; do </span><span> block do </span><span> # some ruby </span><span> node.override[&#39;var&#39;] = &#39;value&#39; </span><span> end </span><span style="color:#f92672;">- action :run </span><span style="color:#a6e22e;">+ action :nothing </span><span style="color:#f92672;">- end </span><span style="color:#a6e22e;">+ end.run_action(:run) </span><span> </span><span>if node[&#39;var&#39;] == &#39;value&#39; </span><span> notifies :run, some_resource[name] </span><span>else </span><span> notifies :run, other_resource[whatever] </span><span>end </span></code></pre> <p>Basically you want to execute the <code>ruby_block</code> that sets your attribute <strong>now</strong> so the rest of your code can consume the value either later in the compilation phase or during the execution phase of the Chef run.</p> <h2 id="how-why-did-that-work">How/why did that work?</h2> <p>If you've read his far you're actually reading. Welcome to the post! Let's begin.</p> <p>Let's dig into <em>exactly</em> what my problem was, because sometimes examples aren't good enough. You know... for fun.</p> <p>I was trying to trigger an application upgrade based on the current installed version. I was <em>not</em> installing a package-manager-managed package<sup class="footnote-reference"><a href="#1">1</a></sup> so I couldn't just pin the version a <code>yum</code> resource, I had to do this kludge of a solution.</p> <p>So for me <code>some_resource[name]</code> was downloading the latest version of the app (a <code>.tar.gz2</code> file from GitHub) and <code>other_resource[whatever]</code> was a no-op. The as part of the package download I unconditionally trigger a <code>:delayed</code> service restart. Chef was 'upgrading' and restarting my app every thirty minutes which broke core functionality.<sup class="footnote-reference"><a href="#2">2</a></sup></p> <h2 id="shit-i-tried-before-stumbling-upon-the-solution">Shit I tried before stumbling upon the solution</h2> <p>I tried a bunch of things, but they can all kinda be summed up in this:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span># yay more diffs </span><span> ruby_block &#39;foo&#39; do </span><span> block do </span><span> # some ruby </span><span> node.override[&#39;var&#39;] = &#39;value&#39; </span><span> end </span><span> notifies :create, &#39;tar[my-app]&#39;, :immediately # Triggers tar[my-app] to run now </span><span> end </span><span> </span><span style="color:#f92672;">- if node[&#39;var&#39;] == &#39;value&#39; </span><span style="color:#f92672;">- notify :run, some_resource[name] </span><span style="color:#f92672;">- else </span><span style="color:#f92672;">- notify :run, other_resource[whatever] </span><span style="color:#f92672;">- end </span><span> </span><span> tar &#39;my-app&#39; do </span><span> action :nothing # Unless triggered this does nothing </span><span style="color:#a6e22e;">+ only_if { node[&#39;var&#39;] == &#39;value&#39; } # This should also do nothing if this isn&#39;t true </span><span> # The rest of the resource block </span><span> end </span></code></pre> <p>Which ultimately didn't work either.</p> <p>I... I have no idea why this didn't work.</p> <p>Something something Chef is complicated.</p> <h2 id="why-the-solution-works-and-the-other-stuff-didn-t">Why the solution works (and the other stuff didn't)</h2> <p>The reason my ultimate solution <em>did</em> work is best summed up by this quote from the <a href="https://docs.chef.io/resource_common.html#run-action">Chef Docs</a>:</p> <blockquote> <p>Use <code>.run_action(:some_action)</code> at the end of a resource block to run the specified action during the compile phase.</p> </blockquote> <p>My original code (read: broken code) was running the <code>if</code> block during the compilation phase (happens earlier) and running the <code>ruby_block</code> in the execution phase (happens later). By telling it to run my <code>ruby_block</code> during the compilation phase we were ensuring it happened before the <code>if</code> block, and in a way running it like you would 'expect' a script to run.</p> <p>It ain't idiomatic but it gets the job done.</p> <h2 id="errata">Errata</h2> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p>Say that three times fast.</p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p>The service was in question was Prometheus. When Chef ran every 30 minutes, it killed the Prometheus process and thus killed all of our pending alerts. TLDR we didn't get any alerts that took more than 30 minutes to trigger We also <em>kept</em> getting alerts that should taken a few hours to re-notify.</p> </div> 2017-12-12T00:00:00+00:00 2017-12-12T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/graduation/ <p>Today I checked my grades for the last time. I can safely say that I have <strong>finally</strong> gotten a gorramn undergraduate degree in Computer Science, with a minor in Mathematics, from Oregon State University.[^1]<sup class="footnote-reference"><a href="#2">1</a></sup></p> <p>I hear that this happens on the reg<sup class="footnote-reference"><a href="#3">2</a></sup> but I'm sure that every single person who <em>actually graduates</em> feels amazing when they do.</p> <p>It's not amazing because graduation is a Herculean feat. I know I'm not a 60 year old grandmother who dreamt of getting a college degree for 45 years and finally scraped together the cash and time in retirement. I'm also not a refugee that escaped a war-torn country for an education and a better life. I'm just another white dude that checked all the boxes.</p> <p>It wasn't hard for me. I just had to put my head down and do the work. But it did take a long ass time.</p> <p>Honestly my story is exactly like <em>most</em> college graduation stories. I went to kindergarten, then elementary school, then middle school, then high school, then college, and now life. I plugged away at school, did what I needed to pass my classes, stressed and lamented, studied hard and eventually the system said "Cool.You're done." Along the way I learned how to "play the game"<sup class="footnote-reference"><a href="#4">3</a></sup>, albeit a bit late compared to a lot of my peers.</p> <p>I'm thankful to be in a field which doesn't require a masters or PHD to get a jorb. Heck my field doesn't really require the degree I got; a surprising number of my friends got well paying jobs sans-degree.</p> <p>I'm also thankful to graduate without debt. This came from some kickass parents + grandparents, some work during school, and some bad stuff that happened which ended up paying for abou half of my tuition.</p> <p>I'm thankful to graduate with wonderful friends, a job, and some passions which make my work-related decisions simple and profitable.</p> <p>Last, I'm thankful to be <em>done</em>. Now I can finally focus on <em>learning</em>, my real passion.</p> <p>I'm gonna have a <a href="https://10barrel.com/beer/crush-cucumber-sour/">Cucumber Sour</a> and celebrate. Thanks for reading.</p> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">4</sup> <p>I don't claim to have the best sources, but at least it's not Yahoo Answers: <a href="https://www.reference.com/education/many-people-graduate-college-year-8b0ccf821050d39e">https://www.reference.com/education/many-people-graduate-college-year-8b0ccf821050d39e</a></p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">1</sup> <p>Go beavs.</p> </div> <div class="footnote-definition" id="3"><sup class="footnote-definition-label">2</sup> <p>I'm 99% sure I graduated. Something horribly kafkaesque might still happen, but fuckit.</p> </div> <div class="footnote-definition" id="4"><sup class="footnote-definition-label">3</sup> <p>This is shorthand for tips and tricks to get through school.</p> <p>School isn't really focused on making sure you <em>learned</em>, it's mostly designed to make sure you can pass tests. Once you figure this out you can focus on learning for the test instead of trying to grok the material. I believe the kids call it a "life hack".</p> <p>Not that you shouldn't try to grok the material, that just happens to take <em>way</em> longer than test-based learning. So if you want a personal life, learn for the test; if you want a PHD, learn for the material.</p> <p>I didn't figure this system out until about Junior year of High School, too late to apply this zen skill to the ACT and SAT, but it was useful in college.</p> </div> 2017-11-27T00:00:00+00:00 2017-11-27T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/independent-crypto-conclusion/ <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>This is a part of a series of blog posts I wrote for an Independent Study on cryptography at Oregon State University. To read all of the posts, check out the 'Independent Crypto' tag.</p> </span> </span> <p>Wait, is it over already? It feels like we just started! Those 10 weeks always fly by, and this term was no exception.</p> <p>Independent Crypto has been a thrill, an honor, and of course immensely educational. It's been a blast to say the least. The weekly check-ins with my mentor <a href="http://web.engr.oregonstate.edu/~rosulekm/">Mike Rosulek</a>, forcing myself to engage with academic papers and online lectures, and finally having complex topics like Garbled Circuits and Elliptic Curves "make sense", it was all very enjoyable. I am very fortunate to have been given the opportunity to both create this course and take it. Excited as I was before it began, it turned out even better than I could have hoped.</p> <p>Let's reflect on the some of the things we learned...</p> <h2 id="elliptic-curve-cryptography">Elliptic Curve Cryptography</h2> <p>We learned that Elliptic Curves look like this:</p> <p><img src="/images/independent-crypto/ecc-1.png" alt="A straight forward ECC." /></p> <p>and this:</p> <p><img src="/images/independent-crypto/ecc-3.png" alt="ECC with the line L illustrated" /></p> <p>We also learned that you can implement a version of the Diffie-hellman key exchange protocol by "adding" points on an Elliptic Curve over a finite field.</p> <p>We also learned that despite how weird Elliptic Curve Cryptography sounds when you describe it, it can be used in very secure and efficient crypto.</p> <h2 id="memory-hard-functions">Memory Hard Functions</h2> <p>We learned that Memory Hard Functions (MHFs) are a solution to the arms race that is hash-cracking hardware. Importantly, it relies on the fact that while there are specialized hashing processors, there is no specialized RAM for the same task (or any task really).</p> <p>The goal of an MHF is to make it as hard (or harder) for an adversary to compute a given hash as it was for you, assuming you're running on non-specialized hardware and they have specialized hash-cracking hardware.</p> <p>We learned that there are two types of MHF's: data-dependent and data-independent. Data-dependent MFHs (dMHFs) have predictable memory usage patterns so they may be susceptible to cache-timing attacks. Data-independent MHFs (iMHFs) are not susceptible to this attack as their memory patterns are not predictable. While there are dMHFs in the wild, like scrypt which performs exceptionally well, there are not any any proven iMHFs in use.</p> <p>We also learned that this is what it looks like to "Pebble an iMHF Directed Acyclic Graph":</p> <p><img src="/images/independent-crypto/dag-animated.gif" alt="Animated DAG traversal." /></p> <p>We also learned that the way to attack an iMHF is by performing a breadth-first search on the graph, then once you hit a wall, fill in the necessary nodes (pebbles) until you can compute the next node. Much of the active research into iMHFs is in figuring out the best graphs, or types of graphs, to combat these kinds of feather/balloon attacks.</p> <h2 id="garbled-circuits">Garbled Circuits</h2> <p>We learned that Garbled Circuits are a way of achieved two-party secure function evaluation.</p> <p>Alice and Bob agree on a circuit (program) to garble. One party encrypts the circuit by encrypting each logic-gate. Both parties obfuscate their inputs and evaluate the garbled circuit. This gives both parties the result of the original program without either party knowing the exact inputs.</p> <p>This isn't a fool-proof security measure. Sometimes it is good to ask the party garbling said circuits to create a few extras. The evaluating party opens some of them to make sure they're on the up-and-up. The other party evaluates the remaining circuits and verifies that the outputs are consistent.</p> <p>We also learned that this game is surprisingly fun for only having <strong>four levels</strong>.</p> <iframe src="/garbled-circuits-game.html" height="400px" width="100%"></iframe> <p>Permalink: <a href="http://elijah.run/garbled-circuits-game.html">http://elijah.run/garbled-circuits-game.html</a></p> <h2 id="remote-timing-attacks">Remote Timing Attacks</h2> <p>And now for something completely different! We broke from the math and theory to focus on a problem involving real tangible bits!</p> <p>Based on response timing differences an adversary can sometimes determine private information <strong>like your OpenSSL Private Keys</strong>. This is scary, but the problem has mostly been fixed and now there are standards the crypto community follows to write code which is secure against these attacks.</p> <p>If you're writing crypto and you know what you're doing, make sure you're using the defacto constant-time libraries for bit-wise comparisons, mathematical operations, and pretty much anything involving secrets.</p> <h2 id="miscellaneous">Miscellaneous</h2> <p>I personally gained a lot from this course. I wrote annotated bibliographies, read academic papers, implemented a remote timing attack, and made a fun little game which was totally relevant and 100% not just an excuse to play around making games.</p> <p>Despite sinking hours into this course I barely scratched the surface of every topic. I've got <em>loads</em> more I could cover if I was inclined to do so. I'm not saying I'm <em>going</em> to get a graduate degree, but if it's anything like this I'd be up for it.</p> 2017-11-20T00:00:00+00:00 2017-11-20T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/remote-timing-attacks/ <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>This is a part of a series of blog posts I wrote for an Independent Study on cryptography at Oregon State University. To read all of the posts, check out the 'Independent Crypto' tag.</p> </span> </span> <p>Fade in.</p> <p>It is 2002.</p> <p>You are a Linux system administrator. You and thousands of other admins are running OpenSSL on you battle tested Linux servers. You trust that your data is transfered securely from host to host because... why wouldn't it be?! OpenSSL makes things secure. Duh.</p> <p>A few months into running that server you figure out that your private keys have been compromised! They were stolen somehow but you can't figure out what happened. You check the logs to see if somebody hacked into your system, but nothing obvious catches your eye.</p> <p>Combing through the logs you do see an IP address that tried (and failed) hundreds of thousands of times to authenticate with your OpenSSL server. It attempted authentication over and over and failed relentlessly until suddenly, after about two days, it stopped. Weird.</p> <p>A year later you read an academic paper from Stanford.<sup class="footnote-reference"><a href="#1">1</a></sup> It clicks. You've you've been Remote Timing Attacked!</p> <h2 id="definition">Definition</h2> <p>Remote Timing Attacks are a special brand of Side Channel Attack where adversaries use differences in response times to determine private information. Creepy I know.</p> <p>A lot of code and Statistics^TM goes into figuring out secret information based on this, but let's start small. Take the following comparison which lives in a hypothetical SSL library:</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#ff5e5e;">if </span><span style="color:#e9fdac;">length</span><span>(recieved_key) </span><span style="color:#ff5e5e;">!= </span><span style="color:#e9fdac;">PRIVATE_KEY_LENGTH</span><span>: </span><span> </span><span style="color:#ff5e5e;">return </span><span>KeyLengthError </span><span> </span><span style="color:#ff5e5e;">if </span><span>recieved_key </span><span style="color:#ff5e5e;">== </span><span>expected_key: </span><span> </span><span style="color:#ff5e5e;">return </span><span>Thing </span><span style="color:#ff5e5e;">else</span><span>: </span><span> </span><span style="color:#ff5e5e;">return </span><span>UnknownKeyError </span></code></pre> <p>This might seem innocent enough but as it turns out this leaks information! The <code>==</code> operator does not <em>always</em> take the same amount of time to return a response. In fact, this operator tends to compare two inputs bit-by-bit. When it finds a difference it short-circuits and returns True or False. If <code>recieved_key</code> shares the first N bits with <code>expected_key</code>, the program will return slightly sooner than if they only shared the first N-1 bits.</p> <p>What's the timing difference if it's just returning one or two cycles earlier; does it really matter? As it turns out, there is <em>enough</em> of a difference to break security and enable an adversary to decrypt entire private keys!<sup class="footnote-reference"><a href="#2">2</a></sup> This is put well by an article on chosenplaintext.ca:<sup class="footnote-reference"><a href="#3">3</a></sup></p> <blockquote> <p>Now, it may not seem significant that an attacker can see how many bytes of their key were a match, but it can actually be <strong>fatal</strong> to security. The attacker can crack the first byte of the key by trying all 256 possibilities, and observing which one caused the comparison to take longer. Now, armed with the first byte, they can do the same with the second byte, and the third, and so on, until they have recovered the entire key.</p> </blockquote> <p>With a lot of patience an adversary can recover secret information from an OpenSSL processes on the same host OS, an OpenSSL processes on a Virtual Machine on the same host OS, and even an OpenSSL on a separate host processes across a network. It gets harder (read: more time consuming) to hack the farther away adversary, but they're all possible with sufficient patience, compute power, and Statistics^TM.</p> <h2 id="oh-god-fix-it-please">Oh god fix it please</h2> <p>Clam down! We've come a long way since 2003 when this was proven to be a viable attack. Most SSL libraries have fixed this vulnerability so you're fine as long as you updated in the past decade.</p> <p>If you <em>haven't</em> updated in the past decade... burn that server. Even the silicon atoms are compromised. It's not even worth trying a fresh install. The thing is just too far gone, start fresh. Goodnight, sweet prince.</p> <h2 id="constant-time-algorithms">Constant-time Algorithms</h2> <blockquote> <p>How did the crypto libraries solve this problem?</p> <blockquote> <p><strong>Constant-time Algorithms</strong></p> </blockquote> <p>Oh fancy, tell me more.</p> </blockquote> <p>Constant-time Algorithms are a way of implementing an algorithm in a way that always takes the same amount of time to compute regardless of the input.</p> <p>Specifically, these perform in constant-time while processing <em>secret</em> information. This distinction means processing a secret key <em>always</em> takes N cycles while checking that a configuration file is correctly formatted might take a M cycles or maybe M+5.</p> <p>There are a lot of coding practices to be aware of in crypto which help us to avoid leaking information for Remote Timing Attacks. Let's go over a few.</p> <h2 id="limit-conditionals-on-secrets">Limit conditionals on secrets</h2> <blockquote> <p>Avoid conditioning on secret information to avoid (among other things) CPU branch predictions.</p> </blockquote> <p>Take this code for example:</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#ff5e5e;">if </span><span>usually_true: </span><span> </span><span style="color:#e9fdac;">do_usual_thing</span><span>() </span><span style="color:#6d6d6d;"># Path A </span><span style="color:#ff5e5e;">else</span><span>: </span><span> </span><span style="color:#e9fdac;">do_weird_thing</span><span>() </span><span style="color:#6d6d6d;"># Path B </span></code></pre> <p>The CPU eventually will recognize that Path A is going to happen more than Path B so it will try to optimize for that path, making the "usual thing" faster. This makes sense from a CPU designer standpoint<sup class="footnote-reference"><a href="#4">4</a></sup>, but it leaks information about which branch is being taken. When the "unusual thing" happens the CPU has to backpedal before continuing, which takes a notable amount of time. This backpedaling gives an adversary enough information to craft an attack the path they're on and extrapolate secret information based on that path-awareness.</p> <p><em>This type of attack (conditioning on private information) is explored in the Constant Time Algorithm example and Remote Timing Attack demo near the end of the post.</em></p> <h2 id="division-multiplication-tricky-stuff">Division/Multiplication: tricky stuff</h2> <blockquote> <p>Multiplication is not always constant-time.</p> </blockquote> <p>Take this piece of code.</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>t1_a = current_time </span><span>small_number_a * small_number_b </span><span>t2_a = current_time </span><span> </span><span>t1_b = current_time </span><span>big_number_a * big_number_b </span><span>t2_b = current_time </span><span> </span><span>assert(t2_a - t1_a == t2_b - t1_b) </span></code></pre> <p>Believe it or not, even though the first and second blocks are just multiplying two numbers together they can take different amounts of time depending on your CPU and compiler.</p> <p>This is triggered by some CPUs just not being equipped to handle large numbers, so they have to perform the large number multiplication in software. Other CPUs optimize for small numbers since those get handled more frequently than large numbers. These are pretty old hardware limitations, and the issue has mostly been resolved in newer 64-bit CPUs. That said... you know... still something to lookout for.</p> <p>The same goes for division. Many CPUs don't have hardware support for division so the compiler needs to handle the operation in software.</p> <p>TLDR: the same piece of code which is constant-time on one architecture (X86_64) might not be constant-time on another piece of hardware (x86_32 for example).<sup class="footnote-reference"><a href="#5">5</a></sup></p> <h2 id="compilers-and-undefined-behavior">Compilers and undefined behavior</h2> <blockquote> <p>Watch out for compiler's "undefined behavior".</p> </blockquote> <p>The C programming language, and <em>most</em> programming languages, have a formal specification of some kind. This formal specification gives the programmer a good idea about what their code will do when they compile and run it.</p> <p>For example if I wrote the following C:</p> <pre data-lang="c" style="background-color:#191919;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#fbdfb5;">int </span><span style="color:#8cdaff;">main</span><span>() { </span><span> </span><span style="color:#ff5e5e;">return </span><span style="color:#fdb082;">10 </span><span style="color:#ff5e5e;">+ </span><span style="color:#fdb082;">20 </span><span>} </span></code></pre> <p>it <em>should</em> run and return <code>30</code>, because the specification tells us that the <code>+</code> operator adds two numbers and <code>return</code> returns a given value from a function. It also says that <code>main</code> returns a given integer as the exit status.</p> <p>If I ran that code and it returned <code>-1</code> I'd be very confused; that breaks specification! Compiler authors know this and follow the specification of the C language very carefully to make sure specified inputs produce specified outputs.</p> <p>But what about behavior the specification <em>doesn't</em> mention?</p> <p>Take for instance this:</p> <pre data-lang="c" style="background-color:#191919;color:#f8f8f2;" class="language-c "><code class="language-c" data-lang="c"><span style="font-style:italic;color:#fbdfb5;">int </span><span style="color:#8cdaff;">main</span><span>() { </span><span> </span><span style="color:#6699cc;">fprintf</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fdb082;">%d</span><span style="color:#ffffff;">&quot;</span><span>, </span><span style="color:#fdb082;">214748300 </span><span style="color:#ff5e5e;">* </span><span style="color:#fdb082;">214745000</span><span>); </span><span>} </span></code></pre> <p>This might not be explicitly covered by the specification because it's pretty weird. 214748300 and 214745000 are close to the signed integer maximum, so when multiplied together what happens? Will the program print an unsigned integer value of 4611686014132420609 or does it return a signed integer which has overflowed but is still signed?</p> <p>This kind headache is called Undefined Behavior and it basically means the compiler, or rather the compiler's authors, <em>choose</em> which behavior they think is best because the language spec didn't say what should happen.</p> <p>Another side effect hinted at is that while you can be confident what the end behavior of your code will be you can't predict how that behavior is achieved. This didn't used to be an issue when C compilers were just 1:1 mapping your C loops and functions to sane assembly. Fortunately compilers have gotten much better at producing fast and efficient executables. <strong>Unfortunately</strong>, we aren't easily able to predict the runtime of our code because our compilers are liable to pour some black-magic voodoo on any and all binaries it produces.</p> <p>These points are condensed really well by the BearSSL website:</p> <blockquote> <p>The C programming language is defined to run in an abstract machine under the "as if" rule, so the compiler is free to translate your code in any sequence of instructions that yield the expected result, with execution time not being part of the observable elements that must be preserved.</p> </blockquote> <p>Even though we know that a function will always <em>work</em>, the C language (and compiler spec) doesn't care about <em>how</em> it gets done.</p> <p>When you're trying to write crypto code this can feel like you're a parent telling your kid to clean their room. They'll make it look clean, but they just shoved everything under the bed. Technically they did what you wanted, the room looks cleaner, but they missed the point. Something about building character in constant-time.</p> <p>So what do we do? We need to trick the compiler.</p> <p>There are a handful of tricks to<sup class="footnote-reference"><a href="#6">6</a></sup> to get the compiler to (a) avoid unknown behavior and (b) enforce a specific assembly output. These are tricks include:</p> <ul> <li>Using a bit-wise operations instead of the equivalent mathematical operation.</li> <li>Mark important <em>secret</em> variables as volatile.</li> <li>Manually zero out important <em>secret filled</em> memory.</li> <li>Use multiple sources of entropy; as many as you can get your hands on.</li> <li>Read the output assembly and become a Jedi.</li> </ul> <p>These are very high-level solutions to some of the problems, solutions I'm only going to hand-wavily describe, but honestly that's because I don't really grok the solutions and don't want to lead you astray. Check out the end of this post for further reading by really smart people that get paid to do this stuff.</p> <h2 id="why-can-t-i-just">"Why can't I just..."</h2> <p>Just <em>wait</em>? If only.</p> <p>So the first thought I had (and every other crypto novice has) is something like this:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>do time sensitive operation </span><span>sleep N seconds </span></code></pre> <p>This doesn't work because this just shifts the amount of time it takes to do an operation, literally <em>just</em> making your crypto take longer. Then the <em>second</em> thought I and every other crypto novice has is something like:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>record start time </span><span>do time sensitive operation </span><span>record end time </span><span>sleep (expected time - elapsed time) seconds </span></code></pre> <p>This is <em>better</em> but you'll never <em>nail</em> the expected time. It'll either be too long or too short This means that either some amount of information is leaked, when <code>expected time</code> is too short, or the crypto is needlessly slow, which is just a silly compromise.<sup class="footnote-reference"><a href="#7">7</a></sup></p> <p>We <em>can</em> make it algorithmically secure without this <code>sleep</code> hack, so we <em>will</em>.</p> <h2 id="practical-solutions-to-timing-attacks">Practical Solutions to Timing Attacks</h2> <p>If you're implementing crypto, STOP.</p> <p>If you're implementing crypto and you know what you're doing, your language of choice <em>probably</em> has a constant-time library which implements some primitives that you can take advantage of for simple tasks like comparisons. Those can be found at the end of this post in Errata. <sup class="footnote-reference"><a href="#CTLibs">8</a></sup></p> <p>After looking into constant-time-ifying your code, do some reading and testing! There have been a lot of developments in analyzing program constant-time-ness and much of this work is Open Source! These are implemented using a lot of Statistics^TM, code inspection, and even Valgrind.[^8][^9]<sup class="footnote-reference"><a href="#10">9</a></sup> If you're really concerned that <em>Your Crypto Library</em> isn't secure against Remote Timing Attacks, take one of those tools on a test drive and see what happens. It's pretty likely that you'll find a <em>notable</em> timing difference based on different inputs and you'll probably need to make changes for your library to be secure against timing attacks.</p> <p>It's for the greater good. Because of your contributions the crypto community is even stronger.</p> <h3 id="a-crazy-idea-constant-time-language-spec">A crazy idea: constant time language spec</h3> <p>My crazy idea, for my <em>very</em> hypothetical grad-school studies would be to implement a Constant-time <em>compiler</em> and/or <em>language</em>. This would perform transformations to your code in an attempt to make it constant-time or warning you when your code isn't going to run in constant-time when it ought to.</p> <p>Of course this would take a very long time, and honestly I haven't thought it through entirely, but I imagine something like this:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>regular code </span><span> </span><span>ct { </span><span> thing that needs to be constant-time. </span><span>} </span><span> </span><span>regular code </span></code></pre> <p>Where you tell the compiler "This needs to be constant-time". The compiler does it's best to convert loops and statements into constant-time and when it's done it tells you if it was able to convert your code into constant-time execution or not.</p> <p>It's probably overkill; you don't usually just willy-nilly write constant-time code. That said, just like C and Python help produce less error prone code than writing straight Assembly, so too might a constant-time language help produce code that hits less of the tricky pitfalls of implementing Constant-time algorithms.</p> <h2 id="constant-time-algorithm-example">Constant-time Algorithm example</h2> <p>Take our code block from the beginning, the one that did the leaky comparison. That takes different amounts of time when processing a given key against a known private key. How would we write <em>that</em> in constant-time?</p> <p>Something like this:</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#6d6d6d;"># Short circuit based on user input, does not leak private information </span><span style="color:#ff5e5e;">if </span><span style="color:#e9fdac;">length</span><span>(recieved_key) </span><span style="color:#ff5e5e;">!= </span><span style="color:#e9fdac;">PRIVATE_KEY_LENGTH</span><span>: </span><span> </span><span style="color:#ff5e5e;">return </span><span>KeyLengthError </span><span> </span><span>recvied_bits </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">bits</span><span>(recieved_key) </span><span style="color:#6d6d6d;"># Taken to be constant-time </span><span>expected_bits </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">bits</span><span>(expected_key) </span><span style="color:#6d6d6d;"># Taken to be constant-time </span><span> </span><span>ret </span><span style="color:#ff5e5e;">= </span><span>Thing </span><span> </span><span style="color:#ff5e5e;">for </span><span>i </span><span style="color:#ff5e5e;">in </span><span style="color:#6699cc;">range</span><span>(</span><span style="color:#e9fdac;">PRIVATE_KEY_LENGTH</span><span>): </span><span> </span><span style="color:#ff5e5e;">if </span><span>recieved_bits[i] </span><span style="color:#ff5e5e;">!= </span><span>expected_bits[i]: </span><span> ret </span><span style="color:#ff5e5e;">= </span><span>KeyLengthError </span><span style="color:#6d6d6d;"># This sets the output, but the loop does not break </span><span> </span><span style="color:#ff5e5e;">return </span><span>ret </span></code></pre> <p>This is similar to our original code but it does a few things differently:</p> <ol> <li>We convert our keys to a variable which can be operated on bit-wise.</li> <li>We manually compare each bit of the inputs. This is what the <code>==</code> operator does, but instead of returning when we get a difference we essentially set a switch. <code>ret = Error</code> from <code>ret = Thing</code>.</li> <li>Outside of the loop we return our response after processing all of our bits.</li> </ol> <p>Behaviorally this is almost identical to our original code, but it does not return earlier or later depending on the user's input.</p> <p>Yet another implementation avoids the direct comparison:</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#6d6d6d;"># Short circuit based on user input, does not leak private information </span><span style="color:#ff5e5e;">if </span><span style="color:#e9fdac;">length</span><span>(recieved_key) </span><span style="color:#ff5e5e;">!= </span><span style="color:#e9fdac;">PRIVATE_KEY_LENGTH</span><span>: </span><span> </span><span style="color:#ff5e5e;">return </span><span>KeyLengthError </span><span> </span><span>recvied_bits </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">bits</span><span>(recieved_key) </span><span style="color:#6d6d6d;"># Taken to be constant-time </span><span>expected_bits </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">bits</span><span>(expected_key) </span><span style="color:#6d6d6d;"># Taken to be constant-time </span><span> </span><span>matching_bits </span><span style="color:#ff5e5e;">= </span><span style="color:#fdb082;">0 </span><span> </span><span style="color:#ff5e5e;">for </span><span style="color:#e9fdac;">_ </span><span style="color:#ff5e5e;">in </span><span style="color:#6699cc;">range</span><span>(</span><span style="color:#e9fdac;">PRIVATE_KEY_LENGTH</span><span>): </span><span> matching_bits </span><span style="color:#ff5e5e;">+= </span><span style="color:#fbe3bf;">int</span><span>(</span><span style="color:#e9fdac;">xor</span><span>(recieved_bits[i], expected_bits[i])) </span><span> </span><span style="color:#ff5e5e;">if </span><span>matching_bits </span><span style="color:#ff5e5e;">== </span><span style="color:#e9fdac;">PRIVATE_KEY_LENGTH</span><span>: </span><span> </span><span style="color:#ff5e5e;">return </span><span>thing </span><span style="color:#ff5e5e;">else</span><span>: </span><span> </span><span style="color:#ff5e5e;">return </span><span>KeyLengthError </span></code></pre> <h2 id="remote-timing-attack-demo">Remote Timing Attack demo</h2> <blockquote> <p>Examples are fine, but what about a demo! You said this was a real threat!</p> </blockquote> <p>True, I did say that... so we'll do a small demonstration.</p> <p>Below is a bit of Python code that checks a user's input against some hard-coded secret.</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#6d6d6d;"># secret.py </span><span style="color:#ff5e5e;">from </span><span>time </span><span style="color:#ff5e5e;">import </span><span>sleep </span><span style="color:#6d6d6d;"># Used to exaggerate time difference. </span><span style="color:#ff5e5e;">from </span><span>sys </span><span style="color:#ff5e5e;">import </span><span>argv </span><span style="color:#6d6d6d;"># Used to read user input. </span><span> </span><span style="font-style:italic;color:#ff5e5e;">def </span><span style="color:#8cdaff;">is_equal</span><span>(</span><span style="font-style:italic;color:#fc9354;">a</span><span>,</span><span style="font-style:italic;color:#fc9354;">b</span><span>): </span><span> </span><span style="color:#6d6d6d;">&quot;&quot;&quot;Custom `==` operator&quot;&quot;&quot; </span><span> </span><span style="color:#6d6d6d;"># Fail if the strings aren&#39;t the right length </span><span> </span><span style="color:#ff5e5e;">if </span><span style="color:#6699cc;">len</span><span>(a) </span><span style="color:#ff5e5e;">!= </span><span style="color:#6699cc;">len</span><span>(b): </span><span> </span><span style="color:#ff5e5e;">return </span><span style="color:#ff8942;">False </span><span> </span><span> </span><span style="color:#ff5e5e;">for </span><span>i </span><span style="color:#ff5e5e;">in </span><span style="color:#6699cc;">range</span><span>(</span><span style="color:#6699cc;">len</span><span>(a)): </span><span> </span><span style="color:#6d6d6d;"># Short-circuit if the strings don&#39;t match </span><span> </span><span style="color:#ff5e5e;">if </span><span>a[i] </span><span style="color:#ff5e5e;">!= </span><span>b[i]: </span><span> </span><span style="color:#ff5e5e;">return </span><span style="color:#ff8942;">False </span><span> </span><span> </span><span style="color:#e9fdac;">sleep</span><span>(</span><span style="color:#fdb082;">0.15</span><span>) </span><span style="color:#6d6d6d;"># This exaggerates it just enough for our purposes </span><span> </span><span> </span><span style="color:#ff5e5e;">return </span><span style="color:#ff8942;">True </span><span> </span><span style="color:#6d6d6d;"># Hard-coded secret globals FOR DEMONSTRATIONS ONLY </span><span>secret </span><span style="color:#ff5e5e;">= </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">l33t</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#6d6d6d;"># This is python for &quot;If someone uses you as a script, do this&quot; </span><span style="color:#ff5e5e;">if </span><span>__name__ </span><span style="color:#ff5e5e;">== </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">__main__</span><span style="color:#ffffff;">&#39;</span><span>: </span><span> </span><span> </span><span style="color:#ff5e5e;">try</span><span>: </span><span> </span><span style="color:#6d6d6d;"># The user got it right! </span><span> </span><span style="color:#ff5e5e;">if </span><span style="color:#e9fdac;">is_equal</span><span>(</span><span style="color:#fbe3bf;">str</span><span>(argv[</span><span style="color:#fdb082;">1</span><span>]), secret): </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">You got the secret!</span><span style="color:#ffffff;">&#39;</span><span>) </span><span> </span><span> </span><span style="color:#6d6d6d;"># The user got it wrong </span><span> </span><span style="color:#ff5e5e;">else</span><span>: </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">Try again!</span><span style="color:#ffffff;">&#39;</span><span>) </span><span> </span><span> </span><span style="color:#6d6d6d;"># The user forgot to enter a guess. </span><span> </span><span style="color:#ff5e5e;">except </span><span style="color:#fbe3bf;">IndexError</span><span>: </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">Usage: python secret.py yourguess</span><span style="color:#fdb082;">\n</span><span style="color:#ffffff;">&#39; </span><span>\ </span><span> </span><span style="color:#ff5e5e;">+</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">The secret may consist of characters in [a-z0-9] </span><span style="color:#ffffff;">&#39;</span><span>\ </span><span> </span><span style="color:#ff5e5e;">+</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">and is </span><span style="color:#fdb082;">{}</span><span style="color:#fbe3bf;"> characters long.</span><span style="color:#ffffff;">&#39;</span><span>.</span><span style="color:#e9fdac;">format</span><span>(</span><span style="color:#6699cc;">len</span><span>(secret))) </span></code></pre> <p>The thing that makes the above code particularly useful for our purposes is that it exaggerates the time it takes to evaluate the <code>is_equal</code> function. Think of this as the 'backpedaling' the CPU does... turned up to 11. Most important <em>for the author</em> we don't need to use Statistics^TM to figure the secret, evaluating each input multiple times and collecting/processing that timing data, it already takes about one magnitude longer to evaluate a matching letter than it does to evaluate a non-matching letter.</p> <p>Next we've got the attack code.</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#6d6d6d;"># attack.py </span><span style="color:#ff5e5e;">from </span><span>time </span><span style="color:#ff5e5e;">import </span><span>time </span><span style="color:#6d6d6d;"># Used to get a timing difference. </span><span style="color:#ff5e5e;">from </span><span>string </span><span style="color:#ff5e5e;">import </span><span>ascii_lowercase </span><span style="color:#6d6d6d;"># All lowercase characters </span><span style="color:#ff5e5e;">from </span><span>string </span><span style="color:#ff5e5e;">import </span><span>digits </span><span style="color:#6d6d6d;"># All digits as strings </span><span style="color:#ff5e5e;">from </span><span>subprocess </span><span style="color:#ff5e5e;">import </span><span>call </span><span style="color:#6d6d6d;"># used to exec a secret.py </span><span style="color:#ff5e5e;">from </span><span>os </span><span style="color:#ff5e5e;">import </span><span>devnull </span><span style="color:#6d6d6d;"># Suppress output of secret.py </span><span> </span><span>current </span><span style="color:#ff5e5e;">= </span><span style="color:#fbe3bf;">list</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">aaaa</span><span style="color:#ffffff;">&#39;</span><span>) </span><span style="color:#6d6d6d;"># Initial guess </span><span>characters </span><span style="color:#ff5e5e;">= </span><span>ascii_lowercase</span><span style="color:#ff5e5e;">+</span><span>digits </span><span style="color:#6d6d6d;"># All possible characters in the secret </span><span> </span><span style="color:#6d6d6d;"># We know the string is the same length as our initial guess </span><span style="color:#ff5e5e;">for </span><span>i </span><span style="color:#ff5e5e;">in </span><span style="color:#6699cc;">range</span><span>(</span><span style="color:#6699cc;">len</span><span>(current)): </span><span> guess_times </span><span style="color:#ff5e5e;">= </span><span>[] </span><span style="color:#6d6d6d;"># Keep track of execution times </span><span> </span><span> </span><span style="color:#ff5e5e;">for </span><span>x </span><span style="color:#ff5e5e;">in </span><span>characters: </span><span> current[i] </span><span style="color:#ff5e5e;">= </span><span>x </span><span style="color:#6d6d6d;"># Swap the current letter with the current guess </span><span> </span><span> </span><span style="color:#6d6d6d;"># Uncomment the following line for fun debug output </span><span> </span><span style="color:#6d6d6d;"># print(&#39;Making guess {}&#39;.format(&#39;&#39;.join(current))) </span><span> </span><span> </span><span style="color:#6d6d6d;"># Execute `secret.py` and time it </span><span> start </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">time</span><span>() </span><span> a </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">call</span><span>([</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">python</span><span style="color:#ffffff;">&#39;</span><span>, </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">secret.py</span><span style="color:#ffffff;">&#39;</span><span>, </span><span style="color:#ffffff;">&#39;&#39;</span><span>.</span><span style="color:#e9fdac;">join</span><span>(current)], </span><span style="font-style:italic;color:#fc9354;">stdout</span><span style="color:#ff5e5e;">=</span><span style="color:#6699cc;">open</span><span>(devnull, </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">wb</span><span style="color:#ffffff;">&#39;</span><span>)) </span><span> end </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">time</span><span>() </span><span> </span><span> </span><span style="color:#6d6d6d;"># Add that time to the list </span><span> guess_times.</span><span style="color:#e9fdac;">append</span><span>(end</span><span style="color:#ff5e5e;">-</span><span>start) </span><span> </span><span> </span><span style="color:#6d6d6d;"># Uncomment the following line for fun debug output </span><span> </span><span style="color:#6d6d6d;"># print(&#39;max {} min {}&#39;.format(max(guess_times), min(guess_times))) </span><span> </span><span> </span><span style="color:#6d6d6d;"># This is a hackey-looking way of getting the outlier time. </span><span> current[i] </span><span style="color:#ff5e5e;">= </span><span>characters[guess_times.</span><span style="color:#e9fdac;">index</span><span>(</span><span style="color:#6699cc;">max</span><span>(guess_times))] </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">character </span><span style="color:#fdb082;">{}</span><span style="color:#fbe3bf;"> is </span><span style="color:#fdb082;">{}</span><span style="color:#ffffff;">&#39;</span><span>.</span><span style="color:#e9fdac;">format</span><span>(i, current[i])) </span><span> </span><span> </span><span style="color:#6d6d6d;"># Uncomment the following line for fun debug output </span><span> </span><span style="color:#6d6d6d;"># print(guess_times.index(max(guess_times))) </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">Final guess is </span><span style="color:#fdb082;">{}</span><span style="color:#ffffff;">&#39;</span><span>.</span><span style="color:#e9fdac;">format</span><span>(</span><span style="color:#ffffff;">&#39;&#39;</span><span>.</span><span style="color:#e9fdac;">join</span><span>(current))) </span></code></pre> <p>It's a bit convoluted in parts but if you stare at it for a while and read the enlightening comments you should see why this gets the right answer.</p> <p>I encourage you to copy that code into two files, <code>secret.py</code> and <code>attack.py</code> and run it like so:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>$ time python attack.py </span><span>character 0 is l </span><span>character 1 is 3 </span><span>character 2 is 3 </span><span>character 3 is t </span><span>Final guess is l33t </span><span> </span><span>real 0m35.176s </span><span>user 0m1.300s </span><span>sys 0m0.485s </span><span> </span><span>$ python secret.py l33t </span><span>You got the secret! </span></code></pre> <p>You'll need a working Python installation and probably a shell of some kind. I'm on CentOS Linux but any *nix system will <em>probably</em> work. With some fiddling you can probably get it to work on Windows ;-)</p> <blockquote> <p>This isn't a real threat! You exaggerated the problem!</p> </blockquote> <p>Hush now. It demonstrates the principles of the attack. Besides, the post is over. We've only got time for the conclusion and then you're off to bed.</p> <h2 id="constant-time-blogpost">Constant-time blogpost</h2> <p>This topic has been a break from the theory/math-heavy term thus far. Honestly I'm jazzed about it.</p> <p>We learned that something as small as a comparison (a <em>comparison</em>!) can leak information to an adversary. Your algorithm might be secure, but if you're not careful you can leak information in the most menial code.</p> <p>This isn't a lost cause. We don't need to throw this <em>security</em> thing out the window. If we're aware of the gotchas we can craft code that solves these problems. It's hard work but the peace of mind should make it worth it.</p> <p>Learning about this seemingly obscure (<em>terrifying</em>) exploit in algorithmically secure code is just the kind of headache I enjoy in Computer Science. Although I don't feel like I did Remote Timing Attacks justice, I could probably spend weeks on it, I had to call it quits. I could keep working on this for <em>another 10 weeks</em>, but it's over. Just walk away.</p> <h2 id="annotated-bibliography">Annotated Bibliography</h2> <p>BearSSL[^11]<sup class="footnote-reference"><a href="#12">10</a></sup> : BearSSL is a project which aims to make an architecture-independent constant-time implementation of various Crypto Libraries, largely mirroring compatibility with OpenSSL and related Open Source crypro libraries. Not only is the project interesting but a handful of blogposts and analysis are posted on the website covering topics like how to implement RSA in constant-time to the compatibility of various CPU models with assumptions about constant-time operations (e.g., multiplication).</p> <p>Remote Timing Attacks are Practical<sup class="footnote-reference"><a href="#13">11</a></sup> : This paper was very easy to read for an academic article and covered the creation of various timing attacks in practical conditions (e.g., hacking an RSA private key across a network).</p> <p>Beginner focused blogs[^14]<sup class="footnote-reference"><a href="#15">12</a></sup> : These were two blogposts which laid out what timing attack, why they were viable, and how you can avoid them. Protip: never assume a library you're using is constant-time.</p> <p>Adam Langley's blog[^16]<sup class="footnote-reference"><a href="#17">13</a></sup> : Adam Langley has great Intermediate-level blogposts about the Lucky13 attack and analyzing code for constant-time execution. These aren't for the weak of heart, but are much more accessible than a lot of academic articles on similar topics.</p> <p>CryptoCoding.net Coding Rules<sup class="footnote-reference"><a href="#18">14</a></sup> : This wiki outlines some common pitfalls when writing constant-time code and how to avoid it. It assumes you're writing C code, but many of the principles carry to more exotic languages.</p> <p>Constant Time Testing Papers[^19]<sup class="footnote-reference"><a href="#20">15</a></sup> : These academic papers outline tools developed for studying how constant-time a program is and analyzes various programs with these tools. They're a great (surprisingly recent) survey of this topic.</p> <h2 id="errata">Errata</h2> <div class="footnote-definition" id="CTLibs"><sup class="footnote-definition-label">8</sup> <ul> <li><a href="https://golang.org/pkg/crypto/subtle/">https://golang.org/pkg/crypto/subtle/</a> - <a href="https://cryptography.io/en/latest/hazmat/primitives/constant-time/">https://cryptography.io/en/latest/hazmat/primitives/constant-time/</a> - <a href="https://github.com/isislovecruft/subtle">https://github.com/isislovecruft/subtle</a></li> </ul> </div> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p>Remote Timing Attacks are Pratical; David Brumley, Dan Boneh; <a href="http://crypto.stanford.edu/~dabo/pubs/papers/ssl-timing.pdf">http://crypto.stanford.edu/~dabo/pubs/papers/ssl-timing.pdf</a></p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p>Remote Timing Attacks are Pratical; David Brumley, Dan Boneh; <a href="http://crypto.stanford.edu/~dabo/pubs/papers/ssl-timing.pdf">http://crypto.stanford.edu/~dabo/pubs/papers/ssl-timing.pdf</a></p> </div> <div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup> <p>Chosen Plaintext: A beginner's guide to contant-time cryptography; <a href="https://www.chosenplaintext.ca/articles/beginners-guide-constant-time-cryptography.html">https://www.chosenplaintext.ca/articles/beginners-guide-constant-time-cryptography.html</a></p> </div> <div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup> <p>As Sonic the Hedgehog always says, Gotta Go Fast!</p> </div> <div class="footnote-definition" id="5"><sup class="footnote-definition-label">5</sup> <p>BearSSL: Constant Time Multiplication <a href="https://bearssl.org/ctmul.html">https://bearssl.org/ctmul.html</a></p> </div> <div class="footnote-definition" id="6"><sup class="footnote-definition-label">6</sup> <p>Cryptography Coding Standards: Coding rules; <a href="https://cryptocoding.net/index.php/Coding_rules">https://cryptocoding.net/index.php/Coding_rules</a></p> </div> <div class="footnote-definition" id="7"><sup class="footnote-definition-label">7</sup> <p>Chosen Plaintext: A beginner's guide to contant-time cryptography; <a href="https://www.chosenplaintext.ca/articles/beginners-guide-constant-time-cryptography.html">https://www.chosenplaintext.ca/articles/beginners-guide-constant-time-cryptography.html</a></p> </div> <div class="footnote-definition" id="8"><sup class="footnote-definition-label">16</sup> <p>Dude, is my code constant-time? Oscar Reparaz, josep Balasch, Ingrid Vebauwhede; <a href="https://eprint.iacr.org/2016/1123.pdf">https://eprint.iacr.org/2016/1123.pdf</a></p> </div> <div class="footnote-definition" id="9"><sup class="footnote-definition-label">17</sup> <p>Veryfying Constnat-Time Implementations, via Usenix; <a href="http://haslab.uminho.pt/jba/files/16usenix.pdf">http://haslab.uminho.pt/jba/files/16usenix.pdf</a></p> </div> <div class="footnote-definition" id="10"><sup class="footnote-definition-label">9</sup> <p>Lucky Thirteen attack on TLS CBC Adam langley via ImperialViolet; <a href="https://www.imperialviolet.org/2013/02/04/luckythirteen.html">https://www.imperialviolet.org/2013/02/04/luckythirteen.html</a></p> </div> <div class="footnote-definition" id="11"><sup class="footnote-definition-label">18</sup> <p>BearSSL: Constant Time Multiplication <a href="https://bearssl.org/ctmul.html">https://bearssl.org/ctmul.html</a></p> </div> <div class="footnote-definition" id="12"><sup class="footnote-definition-label">10</sup> <p>BearSSL: Why Consant-Time Crypto? <a href="https://bearssl.org/constanttime.html">https://bearssl.org/constanttime.html</a></p> </div> <div class="footnote-definition" id="13"><sup class="footnote-definition-label">11</sup> <p>Remote Timing Attacks are Pratical; David Brumley, Dan Boneh; <a href="http://crypto.stanford.edu/~dabo/pubs/papers/ssl-timing.pdf">http://crypto.stanford.edu/~dabo/pubs/papers/ssl-timing.pdf</a></p> </div> <div class="footnote-definition" id="14"><sup class="footnote-definition-label">19</sup> <p>Chosen Plaintext: A beginner's guide to contant-time cryptography; <a href="https://www.chosenplaintext.ca/articles/beginners-guide-constant-time-cryptography.html">https://www.chosenplaintext.ca/articles/beginners-guide-constant-time-cryptography.html</a></p> </div> <div class="footnote-definition" id="15"><sup class="footnote-definition-label">12</sup> <p>A Lesson In Timing Attacks (or, Don't use <code>MessageDigest.isEquals</code>); Coda Hale; <a href="https://codahale.com/a-lesson-in-timing-attacks/">https://codahale.com/a-lesson-in-timing-attacks/</a></p> </div> <div class="footnote-definition" id="16"><sup class="footnote-definition-label">20</sup> <p>Checking that functions are constant-time with Valgrind; Adam langley via ImperialViolet; <a href="https://www.imperialviolet.org/2010/04/01/ctgrind.html">https://www.imperialviolet.org/2010/04/01/ctgrind.html</a></p> </div> <div class="footnote-definition" id="17"><sup class="footnote-definition-label">13</sup> <p>Lucky Thirteen attack on TLS CBC Adam langley via ImperialViolet; <a href="https://www.imperialviolet.org/2013/02/04/luckythirteen.html">https://www.imperialviolet.org/2013/02/04/luckythirteen.html</a></p> </div> <div class="footnote-definition" id="18"><sup class="footnote-definition-label">14</sup> <p>Cryptography Coding Standards: Coding rules; <a href="https://cryptocoding.net/index.php/Coding_rules">https://cryptocoding.net/index.php/Coding_rules</a></p> </div> <div class="footnote-definition" id="19"><sup class="footnote-definition-label">21</sup> <p>Veryfying Constnat-Time Implementations, via Usenix; <a href="http://haslab.uminho.pt/jba/files/16usenix.pdf">http://haslab.uminho.pt/jba/files/16usenix.pdf</a></p> </div> <div class="footnote-definition" id="20"><sup class="footnote-definition-label">15</sup> <p>Dude, is my code constant-time? Oscar Reparaz, josep Balasch, Ingrid Vebauwhede; <a href="https://eprint.iacr.org/2016/1123.pdf">https://eprint.iacr.org/2016/1123.pdf</a></p> </div> 2017-10-17T00:00:00+00:00 2017-10-17T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/memory-hard-functions/ <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>This is a part of a series of blog posts I wrote for an Independent Study on cryptography at Oregon State University. To read all of the posts, check out the 'Independent Crypto' tag.</p> </span> </span> <h2 id="problem-storing-passwords-is-hard">Problem: storing passwords is hard</h2> <p>You're a system administrator and -- oh no! A hacker stole your database!</p> <p>Well, not yet... but they <em>could</em>. Once you get popular enough it's bound to happen.<sup class="footnote-reference"><a href="#1">1</a></sup> Can you make sure your users data is safe <strong>when</strong> that happens?</p> <p>When you store passwords in a database you never store them in plain text. Instead, you store a <a href="https://en.wikipedia.org/wiki/Hash_function">hash</a> of that password. For example:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>Password: 12345678 </span><span>sha256sum (hash): 2634c3097f98e36865f0c572009c4ffd73316bc8b88ccfe8d196af35f46e2394 </span></code></pre> <p>The hash is generated when the user tries to login. The <em>hash</em> of the password the user sends at login is compared against the corresponding password hash for that user. If it matches that means the user sent the right password and so they are authenticated.</p> <p>What happens if the hacker pre-computes a bunch of popular passwords? This might sound crazy, but there are lots of people that re-use passwords, like <code>123456</code>. The hacker can pre-compute the hash for the 1,000,000 most popular passwords and more or less reverse-search for any user's password once they have a database dump.</p> <h2 id="solution-1-add-salt">Solution 1: add salt</h2> <p>Our first naive solution to solve this problem is to make the adversary's life harder by adding <em>salt</em> to our passwords. This is a piece of known information which is added to the password so adversaries can't pre-compute a hash-table, they have to compute this after they have the database and figure out the salt. For example:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>Password: 12345678 </span><span>Salt: cryptoHeckYeah! </span><span>New Password: 12345678cryptoHeckYeah! </span><span>sha256sum: 6e8a7780df48a0b687e9e272e8d082f5f4c0c3a8c43b63461c3f62618b111e9d </span></code></pre> <p>Unfortunately we live in 2017 and Graphics processors and <a href="https://en.wikipedia.org/wiki/Application-specific_integrated_circuit">ASICs</a> are cheap and can compute sha256sums <strong>super fast</strong> for <strong>really cheap</strong>. This means that it might be more of a pain, but the adversary can still crack a password with relative ease and efficiency because they've got a computer <em>designed</em> to generate lots of hashes. Curses.</p> <h2 id="solution-2-h-h-h-x">Solution 2: H(H(..H(x)..)</h2> <p>Computing a single sha256sum is easy, but what if the hacker had to compute like... 1000 sha256sums for each password! That sounds pretty hard... right? If we compute the hash of the hash of the hash (etc) it would take like... 1000x longer to compute each user's password. Something like this:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>p = &#39;12345678cryptoHeckYeah!&#39; </span><span>for x in 1..1000 </span><span> p = sha256sum( p ) </span><span>end </span><span>return p </span><span> </span><span>Result: 47c76630def739ede9c05fd974065b1200d4712aa2421eefb1f6b241a1ca6bea </span><span>Time: 0m1.547s </span></code></pre> <p>Unfortunately this hurts more than it helps.</p> <p>In bash on non-specialized hardware, this took about 1.6 seconds. On specialized hardware, written in a systems programming language, and implemented in parallel it'd be much less costly for an adversary to crack passwords hashed this way.</p> <p>Worst of all, this is <em>easier for an adversary to compute than it is for the the "good guys"</em> because the non-malicious actor is using generalized hardware and the adversary is using specialized hardware to compute the hashes. It's like trying to beat a Roadster in a drag race when you're behind the wheel of a Minivan; the Minivan ("good guys") <em>can't win</em> because they weren't built for drag races.</p> <h2 id="solution-3-memory-hard-functions">Solution 3: Memory Hard Functions</h2> <p>The big problem we have is that CPUs can be specialized to crack passwords <em>very quickly</em>. No matter how fast your AWS EC2 instance is, or even that top of the line IBM server you just bought, it will <em>never</em> be faster than a cheap custom designed ASIC. At around 3000$/box it won't break the adversary's bank to break into yours.<sup class="footnote-reference"><a href="#2">2</a></sup></p> <p>While there specialized hash-cracking CPUs <strong>do</strong> exist, specialized hash-cracking <em>memory</em> does <strong>not</strong> exist.<sup class="footnote-reference"><a href="#3">3</a></sup> If we were to create an algorithm which depends on lots of memory, instead of lots of CPU cycles, we could "level the playing field". This should help stop adversaries from reversing passwords as fast as they currently can.</p> <p>This theoretical hash-function is called a Memory Hard Function (MHF). These are difficult to perform unless you have a certain threshold of memory. As a result non-malicious actors can perform a hash in M seconds and it will take a malicious actor <em>at least</em> M seconds to perform the same hash.</p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>TLDR: We want a hash function that takes as long for an adversary to compute as it does for the "good guys" to compute. Since nobody has specialized hash-cracking RAM we should be able to create a hash function which is memory-intensive and fits our criteria. If we have a function that fits this we will have got a <em>Memory hard Function</em> (MHF).</p> </span> </span> <h2 id="scrypt-a-wild-mhf">scrypt: a wild MHF</h2> <p>scrpyt is a key derivation function <a href="http://www.tarsnap.com/scrypt.html">developed for the Tarsnap project</a>. It was designed explicitly to solve this problem and has some pretty impressive results. Some especially impressive results include:</p> <ul> <li>scrypt is about 2⁵ times more expensive to attack for logins than bcrypt.</li> <li>scrypt is about 2¹⁵ times more espensive to attack for logins than MD5 CRYPT.</li> <li>scrypt is about 2³⁷ times more expensive to attack for file encryption than MD5.</li> </ul> <p>scrypt also happens to be a MHF. Yay we found one!</p> <p>So... how does it work?</p> <p>Given a hash function H, an input B, and an integer N, compute:</p> <blockquote> <p>V_i = Hⁱ(B), given 0 ≤ i &lt; N,</p> </blockquote> <p>and</p> <blockquote> <p>X = H^N(B)</p> </blockquote> <p>then iterate</p> <blockquote> <ul> <li>j &lt;- Integrify(X) mod N</li> <li>X &lt;- H(X ⊕ V_j)</li> </ul> <p>N times; and output X</p> <p>The function Integrify can be any bijection<sup class="footnote-reference"><a href="#4">4</a></sup> from {0,1}^k to {0...2^k - 1}.</p> </blockquote> <p>Breaking that down a bit:</p> <ul> <li>The function is given a different hash function (H), an input to compute the hash of (B), and a modulus (N).</li> <li>N hashes are generated with variations of H and the input B called V_0..N.</li> <li>X is initialized with a hash value and a loop begins: <ol> <li>j is set to a psuedo-random integer mod N.</li> <li>X is set to the hash of the existing X value xor'd with one of the V values.</li> </ol> </li> <li>Loop N times and output the final X.</li> </ul> <p>One of the biggest gripes with scrypt is that it has a very predictable runtime. This means that the running of the function is predictable based on the user's input and so can be victim to a cache-timing side-channel attack. We won't be able to get into what this attack means, but basically you can say "scrypt is good, but not perfect".</p> <h2 id="data-independent-mhfs-imhfs">Data-independent MHFs (iMHFs)</h2> <p><img src="/images/independent-crypto/DAG.gif" alt="A directed acyclic graph map." /></p> <p>iMHFs are supposed to solve the problem that scrypt has (side-channel attacks) by have unpredictable runtimes which still result in the same output.</p> <p>iMHFs can be thought of as Directed Acyclic Graphs (DAGs) which are traversed during runtime.</p> <p>Some specifics:</p> <ul> <li>The function depends on a random oracle H: {0,1}^2k -&gt; {0,1}^k</li> <li>The function provides a Directed Acyclic Graph Directed Acyclic Graph (DAG) G used to encode data-dependencies</li> <li>The initial input is a password and a salt.</li> <li>Each other node is labeled with the hash of it's parent nodes.</li> <li>The output is the hash of the value of the last node.</li> </ul> <p>As mentioned before, a very nice feature of iMHFs is that their memory usage pattern does not depend on the user's input (password) and so is not vulnerable to side-channel attacks.</p> <h2 id="pebbling-a-directed-acyclic-graph-dag">Pebbling a Directed Acyclic Graph (DAG)</h2> <p><img src="/images/independent-crypto/dag-animated.gif" alt="A directed acyclic graph traversal." /></p> <p>We can think of the process of computing the output of an iMHF as pebbling a graph where:</p> <ul> <li>Computing the value of a node is to pebble it.</li> <li>There are rules about which nodes can be pebbled at any time.</li> <li>When a pebble is removed from a node it is freed from memory.</li> <li>Our goal is to pebble the last node.</li> </ul> <p>Rules:</p> <ul> <li>We can only place a pebble on a node if we have pebbles on all of it's parents nodes.</li> <li>Our goal is to get to the sink node (exit node).</li> </ul> <p>The naive pebbling algorithm, the one the 'good guy' user would utilize is as follows:</p> <ul> <li>Only one pebble can be placed per time-step.</li> <li>The graph is pebbled in in topological order.</li> <li>Pebbles (calculated nodes) are never discarded until the end of the function.</li> <li>Expected cost: scales with n² where n is the number of nodes.</li> </ul> <p>This does take up considerable resources, but it isn't prohibitive for users on commodity hardware. This means it won't take <em>too long</em> to get your account authenticated. More importantly, it will take about as long for the bad guys to calculate a token as it took you to calculate a token, as opposed to a small fraction it would take if this was a "normal" hash function.</p> <h2 id="attacks-on-imhfs">Attacks on iMHFs</h2> <p>An attack is defined as when cost of calculating a hash from an iMHF is lower than via the nieve approach.</p> <p>The general idea of an iMHF attack is that it has two phases: light phase and balloon phase.</p> <h3 id="light-phase">Light Phase</h3> <p>In the light phase the algorithm races through the DAG discarding as many pebbles as possible, essentially performing a breadth first search for the end of the graph, computing nodes in parallel when possible. Once a node is computed and it isn't immediately needed it is discarded.</p> <p>If the DAG were a straight line from beginning to end this would be fairly memory efficient.</p> <h3 id="balloon-phase">Balloon Phase</h3> <p>In the balloon phase the algorithm has 'hit a wall' and back-computes the nodes it needs to compute the next node whose parent's have already been discarded. This causes a slow-down.</p> <p>An attack described like this has the following complexity:</p> <blockquote> <p>E_R(A) = O(en + √(n³d))</p> </blockquote> <p>For small values of e and d this results in an attack as:</p> <blockquote> <p>E_R(A) = O(n²) for e,d = O(n)</p> </blockquote> <p>Preventing against this type of attack is where much of the research into iMHF's is focused. An ideal iMHF DAG minimize the disparity between the attackers compute time and the "good guy's" compute time.</p> <h2 id="conclusion">Conclusion</h2> <p>This has been a rough overview of Memory Hard functions, how they work, and how variations of MHFs differ.</p> <p>MHFs are functions which remove the advantage that adversaries have to crack passwords by depending heavily on memory. This reduces the adversary's advantage if they have an ASIC or GPU processor(s) to brute-force a password crack and ought to make it very difficult (ideally <em>impractical</em>) for adversaries to crack a password hashed with an MHF.</p> <p>Some existing MHFs, like scrypt, are vulnerable to side-channel attacks so iMHFs have been theorized which do not have a predictable runtime and so are not vulnerable to side-channel attacks. No iMHFs exist yet, however many functions have been developed with get <em>close</em> and offer many of the benefits of iMFHs. Some of these include Argon2i, Catena, and Balloon hashing, which we did not cover in this post.</p> <h2 id="annotated-bibliography">Annotated Bibliography</h2> <p>Conference Presentations by Jeremiah Blocki[^5][^6]<sup class="footnote-reference"><a href="#7">5</a></sup> The video presentations online by Jeremiah were a very important resource for getting a grasp on what MHFs are, and more specifically what iMHFS and how they worked. The three videos cited in this post cover largely the same content and present the material, including the problem, naive solution, MHF solution, iMFH solution, and possible attacks against iMHFs in about 30 minutes. I like to think I'm pretty good at public speaking, but this material was very complicated and presented in a very digestible format.</p> <p>I cannot stress enough how useful these videos were. I learned an incredible amount from these videos and referenced them for the majority of this content.</p> <p>Strict Memory Hard Hashing Functions<sup class="footnote-reference"><a href="#8">6</a></sup> This paper was very short and presented some essential knowledge to discuss the differences between MHFs and iMHFs. I didn't directly use or reference this content, however it did present an easily understandable academic definition and comparison of iMHF compared to MHFs.</p> <p>Practical Graphs for Optimal Side-Channel Resistant Memory-Hard Functions<sup class="footnote-reference"><a href="#9">7</a></sup> This paper was used in the writing of this post, however it was very long and dense, so it was never directly cited.</p> <p>scrypt: A new key derivation function<sup class="footnote-reference"><a href="#10">8</a></sup> This was the soul reference for the scrypt section of this post. There is an academic paper published too, but the slides were simple and presented all of the same knowledge (I think) sans any proofs.</p> <p>If I feel an existential hole in my heart I might read the proofs, but in the interest of time I chose not to right now.</p> <h2 id="errata">Errata</h2> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p><a href="https://haveibeenpwned.com/PwnedWebsites">https://haveibeenpwned.com/PwnedWebsites</a></p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p>Antminer "Bitcoin Miner" <a href="http://a.co/2E20HW8">http://a.co/2E20HW8</a></p> </div> <div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup> <p>Yet.</p> </div> <div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup> <p>Bijection: A function which creates a 1-to-1 relationship between inputs and outputs.</p> </div> <div class="footnote-definition" id="5"><sup class="footnote-definition-label">9</sup> <p>Efficiently Computing Data Independent Memory Hard Functions (Video) Joël Alwen and Jeremiah Blocki, Crypto 2016, September 26, 2016, <a href="https://youtu.be/ujpvPtn_N5Y">https://youtu.be/ujpvPtn_N5Y</a></p> </div> <div class="footnote-definition" id="6"><sup class="footnote-definition-label">10</sup> <p>Towards a Theory of Data-Independent Memory Hard Functions (Video), Jeremiah Blocki with Joel Alwen, Krzysztof Pietrzak 2017, Real World Crypto conference, February 1, 2017, <a href="https://youtu.be/YtfVLzUkwME">https://youtu.be/YtfVLzUkwME</a></p> </div> <div class="footnote-definition" id="7"><sup class="footnote-definition-label">5</sup> <p>Memory Hard Functions and Password Hashings (Video), CERIAS Symposium 2017 - TechTalk, Jeremiah M. Blocki - Assistant Professor, Computer Science - Purdue University, May 1, 2017, <a href="https://youtu.be/9yX4v89m5oo">https://youtu.be/9yX4v89m5oo</a></p> </div> <div class="footnote-definition" id="8"><sup class="footnote-definition-label">6</sup> <p>Strict Memory Hard Hashing Functions, Sergio Demian Lerner, (Preliminary v0.3, 01-19-14), <a href="http://www.hashcash.org/papers/memohash.pdf">http://www.hashcash.org/papers/memohash.pdf</a></p> </div> <div class="footnote-definition" id="9"><sup class="footnote-definition-label">7</sup> <p>Practical Graphs for Optimal Side-Channel Resistant Memory-Hard Functions Joel Alwen, Jeremiah Blocki, Ben Harsha IACR Cryptography ePrint Archive, 2017, <a href="https://eprint.iacr.org/2017/443.pdf">https://eprint.iacr.org/2017/443.pdf</a></p> </div> <div class="footnote-definition" id="10"><sup class="footnote-definition-label">8</sup> <p>scrypt: A new key derivation function (variable subtitles) Colin Percival, May 9, 2009, <a href="http://www.tarsnap.com/scrypt/scrypt-slides.pdf">http://www.tarsnap.com/scrypt/scrypt-slides.pdf</a></p> </div> 2017-10-10T00:00:00+00:00 2017-10-10T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/garbled-circuits/ <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>This is a part of a series of blog posts I wrote for an Independent Study on cryptography at Oregon State University. To read all of the posts, check out the 'Independent Crypto' tag.</p> </span> </span> <p>Let's imagine you are a billionaire. You want to know if you have more money than your billionaire friend Bob, but for some reason it's <em>very</em> faux pas to let anybody know how much money you have, even your good friend Bob.</p> <p>But you're a billionaire! You're not used to the phrase "that isn't possible". In your frustration you try to figure out if you can use some of your billions to find a solution.</p> <p>The first solution you come up with is Trusty Tina. You tell Tina how much you make, Bob tells Tina how much he makes, and then Tina tells both of you who makes more.</p> <p>The only problem is that Tina isn't <em>that</em> trustworthy, her parent's just named her that. Like... you wouldn't trust her with your life or anything. You could pay her to keep your assets secret but Bob might pay her a bit more reveal your number to him. Tina reminds you of this so you have to pay her <em>more</em> to keep the secret, and eventually you have an arms-race type situation at hand. With your cunning accountant skills you can already tell that Tina might be more trouble than she's worth.</p> <p>If you won't tell Bob directly, and you can't depend on Manipulative Tina, is there <em>any</em> way to determine out who has more money?</p> <p>Yes.</p> <h2 id="2-party-secure-function-evaluation">2-Party Secure Function Evaluation</h2> <p>The problem above is the Millionaires problem<sup class="footnote-reference"><a href="#1">1</a></sup> and it is solved by the use of 2-party secure function evaluation (SFE). The general idea<sup class="footnote-reference"><a href="#2">2</a></sup> is that you have a function <code>f</code> which takes as input <code>x</code> and <code>y</code>. This function is garbled by one of the two parties into <code>f'</code> and the inputs are garbled into <code>x'</code> and <code>y'</code> by each of the parties. <code>f'(x', y') == f(x,y)</code> but does not leak any of the inputs, because <code>x</code> and <code>y</code> were obfuscated.</p> <p>In other words...</p> <ol> <li>Alice and Bob agree on a function (i.e., circuit) <code>f(a,b)</code>.</li> <li>Alice garbles the function (<em>cough</em> circuit) <code>f</code> and her input <code>a</code>.</li> <li>Alice sends <code>f'</code> and <code>a'</code> to Bob.</li> <li>Bob gets his input <code>b</code> garbled into <code>b'</code>.</li> <li>Bob evaluates <code>f'(a', b')</code>.</li> <li>This reveals the result of <code>f(a,b)</code> to Bob, but does not reveal Alice's input.</li> </ol> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>The first half of this post focuses on honest garbled circuit uses, meaning both parties are acting honestly (and don't pull any fast-ones).</p> <p>The latter portion focuses on problems with that 'vanilla' garbled circuit implementation and potential solutions.</p> </span> </span> <h2 id="garbling-a-gate">Garbling a gate</h2> <p>How does Alice actually... 'garble' a circuit? It sounds kinda dirty.</p> <p>Each gate (OR, AND, XOR, etc) has two inputs. Stick with me. Each input is encrypted. Keeping up? So you need a <em>key</em> to use <em>each gate</em>. It gets better. But if you have the keys, you don't know which key corresponds with a <code>1</code> or a <code>0</code> so you can compute a function without knowing the actual values you put in. Whoa.</p> <p>Let's use the OR gate as an example.</p> <p>Remember truth-tables for OR? Here's a reminder:</p> <table><thead><tr><th>OR</th><th><strong>0</strong></th><th><strong>1</strong></th></tr></thead><tbody> <tr><td><strong>0</strong></td><td>0</td><td>1</td></tr> <tr><td><strong>1</strong></td><td>1</td><td>1</td></tr> </tbody></table> <p>This table is going to be important for Alice's part of this dance.</p> <h3 id="alice-s-circuit-input">Alice's circuit &amp; input</h3> <p>Alice definitely does the heavy lifting in this transaction.</p> <ol> <li>Alice generates 4 keys <code>Kx0</code>, <code>Kx1</code>, <code>Ky0</code>, and <code>Ky1</code>.</li> <li>Alice creates four variables corresponding with the four values in the OR table:</li> </ol> <table><thead><tr><th>OR</th><th>0</th><th>1</th></tr></thead><tbody> <tr><td><strong>0</strong></td><td><code>B00</code> = 0</td><td><code>B01</code> = 1</td></tr> <tr><td><strong>1</strong></td><td><code>B10</code> = 1</td><td><code>B11</code> = 1</td></tr> </tbody></table> <ol start="3"> <li>Each box is encrypted with the two keys:</li> </ol> <table><thead><tr><th>OR</th><th>0</th><th>1</th></tr></thead><tbody> <tr><td><strong>0</strong></td><td>`E(Kx0</td><td></td></tr> <tr><td><strong>1</strong></td><td>`E(Kx1</td><td></td></tr> </tbody></table> <ol start="4"> <li>Alice sends these ciphertexts (unordered) to Bob.</li> </ol> <h3 id="bob-s-input">Bob's input</h3> <ol start="5"> <li>Bob gets Alice's input, key <code>KxA</code>, from Alice.</li> <li>Bob uses oblivious transfer<sup class="footnote-reference"><a href="#3">3</a></sup> to get his input <code>KyB</code>.</li> <li>With these two keys Bob is able to process the circuit (an OR gate).</li> </ol> <p>Bob has enough information to get one of the four possible outputs of the circuit, but doesn't know if Alice's input is a 1 or a 0.</p> <p>Importantly, while Bob can share the output of the circuit, he should <strong>not</strong> share his input. That would make using OT (step 6) obtuse.</p> <h3 id="extending-the-garbled-gate">Extending the garbled gate</h3> <p><img src="/images/independent-crypto/garbled-circuit.jpg" alt="Garbled circuit example diagram" /></p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>The UTF-8 Padlock symbol doesn't render on my browser because I seem to have gone back in the time to the late 90s. Being stuck in the past, we have to comprmise. The ⛨ symbol is meant to represent a lock and the ⚿ represents a key.</p> </span> </span> <p>Multiple gates can be connected together to build more complicated circuits. One important difference is that while each intermediate circuit still has four cipher-texts, for the four outcomes of a truth-table, those decrypt to a <em>key</em> and not a 1 or 0. The only gates which decrypt to a plain-text of 0 or 1 are output gates, not the intermediate gates.</p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p><em>PSST</em> Check out the end of this post for a <strong>GAME</strong>!</p> </span> </span> <h2 id="problems-with-garbled-circuits">Problems with garbled circuits</h2> <p>There are a few important flaws in the <em>security</em> of garbled circuits as they have been described. The first is that although Alice and Bob agree on a circuit to garble there is no guarantee that the circuit one is evaluating (if you're Bob) is the circuit you agreed upon.</p> <p>For example:</p> <ol> <li>Alice and Bob 'agree' on a function <code>f(a,b)</code>.</li> <li>Alice creates her own function <code>g(a,b)</code>.</li> <li>Alice garbles <code>g</code> and her input <code>a</code> and sends it to Bob as <code>f'</code> and <code>a'</code>.</li> <li>Bob evaluates <code>g'(a',b')</code> and reveals the output to Alice. Alice now knows something other than than Bob agreed to.</li> </ol> <h2 id="improvements-on-garbled-circuit-security">Improvements on garbled circuit security</h2> <p>To prevent the above adversarial attack we do something called "Cut-and-Choose". This is when Bob checks Alice's work to make sure she's not cheating.</p> <p>Remember that Alice and Bob agreed on a given circuit.</p> <ol> <li>Alice generates M garbled circuits for the agreed upon function where M &gt; 1.</li> <li>All secrets for a randomly chosen N circuits are revealed where 1 ≤ N &lt; M, the circuits are "opened".</li> <li>Bob selects one of the remaining M-N circuits to evaluate as outlined earlier.</li> </ol> <p>This ensures that Alice is not nefarious to some statistical certainty. She had control over how the circuits were garbled but she does not have control over which are revealed or evaluated. If she made one (or two or three...) nefarious circuits that bad behavior is <em>probably</em> revealed in step 2, if all the checked circuits are good Alice is <em>probably</em> being honest.</p> <p>This doesn't break garbled circuits for the following reasons:</p> <ul> <li>While Alice reveals the secrets of the N circuits, she doesn't reveal anything about her input. We are only un-garbling the circuit not the inputs (revealing all possible inputs, not Alice's).</li> <li>We're not un-garbling the M-N circuits which may be evaluated, so those are still secret.</li> </ul> <p>As M grows and N approaches M this method gets more secure at the cost of computation cycles and bandwidth in transferring the garbled circuits.</p> <h2 id="the-free-xor-optimization">The "Free XOR" Optimization</h2> <p>I'm definitely not a circuits person. You can show me a circuit diagram and I'll say "Yep, that's a circuit. What's it do?" I couldn't even even identify which gate is which without Wikipedia.</p> <p>I was told during my research for this post that XOR gates are very popular with garbled circuit design, and more broadly circuit design in general. This was shared to me in the form of a cryptic hint so I figured I'd investigate and share my findings here.</p> <p>As it turns out the Wikipedia page notes that this XOR optimization exists and even cites the original paper published on the topic.<sup class="footnote-reference"><a href="#4">4</a></sup></p> <p>The jist of this optimization is that one can very efficiently garble an XOR gate such that the output of the gate is encoded as the XOR of the keys used to unlock the gate and some known global constant. This is in contrast with the implementations discussed in the beginning where each gate had to be decrypted with two cipher-texts and revealed another key.</p> <p>Basically using XOR, which is pretty fast, we can avoid generating four keys per gate and instead craft 1 key which is produced as the result of 'unlocking' a gate.</p> <p>Put a bit more formally:</p> <blockquote> <p>Given a gate G with input wires A and B and output wire C and a random string R, the garbled gate is obtained by XORing the garbled gates inputs C¹ = C⁰ ⊕ R:</p> <blockquote> <p>C⁰ = A⁰ ⊕ B⁰ = (A⁰ ⊕ R ) ⊕ (B⁰ ⊕ R) = A¹ ⊕ B¹ C¹ = C⁰ ⊕ R = A⁰ (B⁰ ⊕ R) = A⁰ ⊕ B¹ = (A⁰ ⊕ R) ⊕ B⁰ = A¹ ⊕ B⁰</p> </blockquote> </blockquote> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>LETTER^{0,1} is short-hand for the True or False output of the given gate.</p> </span> </span> <p>This isn't super intuitive, and honestly I just put those equations up there to prove that I read a paper about this.</p> <p>The main takeaway is that 'free XOR' saves us computation generating and processing cryptographic keys by simply performing the XOR operation. This optimization is so powerful that using <em>mostly</em> XOR gates makes garbled circuits notably faster and more useful for secure computation.<sup class="footnote-reference"><a href="#5">5</a></sup></p> <h2 id="annotated-bibliography">Annotated Bibliography</h2> <p>Foundations of Garbled Circuits<sup class="footnote-reference"><a href="#6">6</a></sup> This is by far the most thorough academic source I have. If I had a better foundation in academic reading this might be the perfect paper but most of it went way over my head. That said the overview of each section was fairly human-readable and gave me a good rough overview for many of the topics covered in this post.</p> <p>A Brief History of Practical Garbled Circuits<sup class="footnote-reference"><a href="#7">7</a></sup> This was the first source I checked out to get a feel for how difficult garbled circuits are as a topic. It quickly glanced at the basics of garbled circuits and then quickly went into the optimizations on garbled circuits. This was overwhelming, but as I started to learn more about garbled circuits and filled in the knowledge gaps it gained significant value.</p> <p>It's a great talk about Garbled Circuits which wasn't ideal for beginners, but did give me a good breadth of the topic and what I could dive into.</p> <p>Improved garbled Circuit: Free XOR Gates and Applications<sup class="footnote-reference"><a href="#8">8</a></sup> This paper was useful in giving me an academic preview of the XOR optimization in Garbled Circuits. I quickly started looking at the many other papers referenced by this one, kind of like following down the Wikipedia wormhole, but with more PDFs and less pictures.</p> <p>SFE: Yao's Garbled Circuit<sup class="footnote-reference"><a href="#9">9</a></sup> This slide-deck was very useful as a reference for basic GCs and securing GC's with cut-and-choose. It wasn't a great initial source for this material, but it was useful <em>after</em> I had a basic understanding of a topic to solidify it with pretty pictures and Comic Sans.</p> <p>Mike Rosulek on Stack Exchange[^10]<sup class="footnote-reference"><a href="#11">10</a></sup> This is more of a shout-out than a citation. Mike Rosulek's posts on crypto.stackexchange.com were very helpful in breaking down core concepts like what garbled circuits are and why XOR is "free". They also provided a good list of further reading which was helpful in addition to the resources provided in the syllabus.</p> <h2 id="errata">Errata</h2> <iframe src="/garbled-circuits-game.html" height="400px" width="100%"></iframe> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>Yes, the name is a misnomer. The goal is to <em>evaluate</em> a garbled circuit, but that just doesn't roll off the tongue the same.</p> </span> </span> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p>The original problem was developed in the 80's. This post adjusts the scenario for inflation.</p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p>To completely level with you, it's been anecdotally proven that there is at least 1 definition of Garbled Circuits for each paper on the topic.</p> </div> <div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup> <p>Oblivious Transfer has been described to me as:</p> <ul> <li>Alice sends two possible options to a box labeled OT.</li> <li>Bob sends a choice to the box labeled OT.</li> <li>Bob gets back one of the two options, without knowledge of the other.</li> <li>Alice does not know which option Bob got.</li> </ul> <p>This is a cryptographic primitive which is very useful for tasks like generating Bob's input to the garbled circuit <code>f'</code>.</p> </div> <div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup> <p>Improved garbled Circuit: Free XOR Gates and Applications, Written by Vladimir Kolesnikov and Thomas Shneider, Published July 2008. <a href="http://www.cs.toronto.edu/~vlad/papers/XOR_ICALP08.pdf">http://www.cs.toronto.edu/~vlad/papers/XOR_ICALP08.pdf</a></p> </div> <div class="footnote-definition" id="5"><sup class="footnote-definition-label">5</sup> <p>A Brief History of Practical Garbled Circuit Optimizations, Presented by Mike Rosulek, Published by the Simons Institute, June 15, 2015. <a href="https://youtu.be/FTxh908u9y8">https://youtu.be/FTxh908u9y8</a></p> </div> <div class="footnote-definition" id="6"><sup class="footnote-definition-label">6</sup> <p>Foundations of Garbled Circuits, Written by Mihir Bellare, Viet tung Hoang, and Phillip Rogaway, Published October, 2012. <a href="https://eprint.iacr.org/2012/265.pdf">https://eprint.iacr.org/2012/265.pdf</a></p> </div> <div class="footnote-definition" id="7"><sup class="footnote-definition-label">7</sup> <p>A Brief History of Practical Garbled Circuit Optimizations, Presented by Mike Rosulek, Published by the Simons Institute, June 15, 2015. <a href="https://youtu.be/FTxh908u9y8">https://youtu.be/FTxh908u9y8</a></p> </div> <div class="footnote-definition" id="8"><sup class="footnote-definition-label">8</sup> <p>Improved garbled Circuit: Free XOR Gates and Applications, Written by Vladimir Kolesnikov and Thomas Shneider, Published July 2008. <a href="http://www.cs.toronto.edu/~vlad/papers/XOR_ICALP08.pdf">http://www.cs.toronto.edu/~vlad/papers/XOR_ICALP08.pdf</a></p> </div> <div class="footnote-definition" id="9"><sup class="footnote-definition-label">9</sup> <p>SFE: Yao's Garbled Circuit, Published by engr.illinois.edu, for the course CS 598, Fall 2009. <a href="https://courses.engr.illinois.edu/cs598man/fa2009/slides/ac-f09-lect16-yao.pdf">https://courses.engr.illinois.edu/cs598man/fa2009/slides/ac-f09-lect16-yao.pdf</a></p> </div> <div class="footnote-definition" id="10"><sup class="footnote-definition-label">11</sup> <p>What exactly is a "garbled circuit"? Asked by user Ella Rose, Answered by user Mikero on on July 27, 2016. <a href="https://crypto.stackexchange.com/a/37993">https://crypto.stackexchange.com/a/37993</a></p> </div> <div class="footnote-definition" id="11"><sup class="footnote-definition-label">10</sup> <p>Why XOR and NOT is free in garbled circuits Asked by user Jason, Answered by user Mikero on February 28, 2017. <a href="https://crypto.stackexchange.com/a/44278">https://crypto.stackexchange.com/a/44278</a></p> </div> 2017-10-04T00:00:00+00:00 2017-10-04T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/elliptic-curve-cryptography/ <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>This is a part of a series of blog posts I wrote for an Independent Study on cryptography at Oregon State University. To read all of the posts, check out the 'Independent Crypto' tag.</p> </span> </span> <div class="warning"> <span class="warning-title"> Warning </span> <span class="warning-body"> <p>This post is jumps around a bit. We'll start by showing how Elliptic Curve Cryptography works at a high level, then create a list of questions about how/why Elliptic Curve Cryptography works and how it is useful to cryptogrpahy. Once those questions are answered we will end with a recap. Hopefully we will zero in on what Elliptic Curves are and what Elliptic Curve Cryptography is.</p> </span> </div> <h2 id="diffie-hellman-key-exchange">Diffie-Hellman key exchange ++</h2> <p>You find yourself day-dreaming during a walk around campus wondering if there is an alternative cryptography system to the very popular RSA. You want something that has improved computational and network efficiency. You want smaller keys that are harder to crack. Could such a system exist?</p> <p>You share this fantasy with a friend, you share all of your crypto fantasies with this friend, and they tell you that Elliptic Curve Cryptography is promising and it perfectly fits your needs. ... but how does it work?</p> <h3 id="diffie-hellman-key-exchange-a-recap">Diffie-Hellman key exchange (a recap)</h3> <p>To create a useful crypto out of Elliptic Curves we need to implement <a href="https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange">Diffie–Hellman key exchange</a> (DHKE). Once we have DHKE we more or less have a valid crypto system which we can build upon to encrypt and decrypt private information.</p> <p>The reader (you) is assumed to be familiar with DHKE. While DHKE is fairly simple, it is not unforgettable, so here is quick reminder:</p> <ol> <li>Alice and Bob agree on a public modulus (p) and a base (g).</li> <li>Alice and Bob both choose secret integers (a and b).</li> <li>Alice sends Bob g^a (mod p) (we call it A) and Bob sends Alice g^b (mod p) (we call it B).</li> <li>Alice computes B^a (mod p) and Bob computes A^b (mod p). These are equivalent (mod p). This is Alice and Bob's shared secret.</li> </ol> <p>How do we use Elliptic Curves get a similar 'shared secret'?</p> <h3 id="elliptic-diffie-hellman-key-exchange">Elliptic Diffie-Hellman key exchange</h3> <p>At a (very) high level the algorithm is as follows:</p> <ol> <li>Alice and Bob agree to use a given Elliptic Curve over a finite field, E(F_p), and a public point P ∈ E(F_p).</li> <li>Alice chooses a secret integer n_A and Bob choose secret integers n_B.</li> <li>Alice computes Q_A = n_AP and Bob computes Q_B = n_BP. These are the "Public Keys"</li> <li>Alice sends Bob her public key, Bob send Alice his public key.</li> <li>Alice computes n_AQ_B, Bob computes n_BQ_A.</li> <li>The shared secret value is n_AQ_B = n_A(n_BP) = n_B(n_AP) = n_BQ_A</li> </ol> <p>It looks similar to the given DHKE algorithm, and seems promising, but... how does it work?</p> <h2 id="elliptic-curves-and-elliptic-curve-cryptography-q-a">Elliptic Curves and Elliptic Curve Cryptography Q&amp;A</h2> <p>To answer that we are going to answer the following:</p> <ul> <li>What are Elliptic Curves?</li> <li>What does an Elliptic Curve look like?</li> <li>What does it mean to multiply P by n?</li> <li>What about a finite field?</li> <li>How are the pubic keys used? Why are these a shared secret?</li> <li>Why is Elliptic Curve Cryptography useful?</li> </ul> <h3 id="what-are-elliptic-curves">What are Elliptic Curves?</h3> <p>A Elliptic Curve is the set of solutions to an equation of the form</p> <blockquote> <p>y² = x³ + AX + B</p> </blockquote> <h3 id="what-does-an-elliptic-curve-look-like">What does an Elliptic Curve look like?</h3> <p>Two examples of Elliptic Curves are as follows:</p> <p><img src="/images/independent-crypto/ecc-1.png" alt="A simple elliptic curve" /></p> <p>and:</p> <p><img src="/images/independent-crypto/ecc-2.png" alt="Another simple elliptic curve" /></p> <h3 id="adding-p-and-q">Adding P and Q</h3> <p>Multiplication is <em>just</em> repeated addition. Oh shoot we haven't said how "addition" happens on an Elliptic Curve. Let's do that.</p> <p>Addition is the process of drawing a line L between P and Q. The third point that the line L intersects is point R. When R is reflected over the X axis we call this R'. The result of P ⊕ Q (read: P 'plus' Q) is R'.</p> <p>We can enumerate these steps as:</p> <ol> <li>Take two points P and Q on the Elliptic Curve E.</li> <li>Draw a line L which passes through these two points.</li> <li>L should ultimately pass through <em>three</em> points: P, Q, and R.</li> <li>Multiply the Y coordinate of R by -1, this is R'.</li> <li>P ⊕ Q = R'.</li> </ol> <p>Here's a visualization of straight forward addition.</p> <p><img src="/images/independent-crypto/ecc-3.png" alt="Annotated curve E with points P, Q, R, R&#39; and line L labeled." /></p> <p>You might think "What happens when P is tangent a point on E?" In that case we say P = Q, so R = P ⊕ P, or R = 2P. It looks like this:</p> <p><img src="/images/independent-crypto/ecc-4.png" alt="Annotated curve E with points P, R, R&#39; and line L labeled. P is tangent to the curve." /></p> <p>Wait a second, 2P looks like n*P which was one of the questions we had! Don't worry, we'll get there soon.</p> <h3 id="that-thing-about-finite-fields">That thing about Finite Fields</h3> <p>In practice we bound the curve over a field F_p with p ≥ 3. We input {1, 2, ..., p-1} as the value of X in E and select the results which are squares modulo 13.</p> <p>For example:</p> <blockquote> <p>E : y² = x³ + 3X + 8 over F₁₃ X = 1 1 + 3 + 8 = 12 12 is a square (mod 13)</p> </blockquote> <p>Repeating this gives us the set of points in E(F₁₃):</p> <blockquote> <p>E(F₁₃) = {O, (1,5), (1,8), (2,3), (2,10), (9,6), (9,7), (12,2), (12,11)}</p> </blockquote> <p>In practice this bounds the graph of E and forces us to draw a strange modulus graph shown below:</p> <p><img src="/images/independent-crypto/ecc-5.gif" alt="Elliptic Curves illustrated where each point is a valid coordinate. There are no curves." /></p> <p><em>Image source: A (relatively easy to understand) primer on elliptic curve cryptography</em><sup class="footnote-reference"><a href="#1">1</a></sup></p> <h3 id="multiplying-p-by-an-integer-with-the-double-and-add-algorithm">Multiplying P by an integer with The Double-and-Add Algorithm</h3> <p>To "multiply" P by n we need to use the Double-and-Add Algorithm. Here's how that looks:</p> <ol start="0"> <li>Take a point P ∈ E(F_p) and an integer n ≥ 1.</li> <li>Set Q = P and R = O.</li> <li>Loop while n &gt; 0. 3. If n ≡ 1 (mod 2), set R = R + Q 4. Set Q = 2Q and n = floor(n/2).</li> <li>Return the point R, which equals nP.</li> </ol> <p><em>Recall that the algorithm for finding point 2Q was covered in the above section</em> <a href="https://elijah.run/blog/elliptic-curve-cryptography/#adding-p-and-q">Adding P and Q</a></p> <h3 id="what-is-the-shared-secret">What <em>is</em> the shared secret?</h3> <p>Let's review. The shared secret is the second point n_An_BP, which is a point on the public curve E(F_p). This point can be used to encrypt information as it is a shared secret (necessary for DHKE). How exactly it is used to encrypt information is left as an exercise for readers in charge of cryptographic implementation standards.</p> <p>The reason this is a shared secret is because an adversary needs to solve the following Elliptic Curve Discrete Logarithm Problem</p> <blockquote> <p>nP = Q_A</p> </blockquote> <p>Which is a very hard problem, as mentioned in the next section.</p> <h2 id="an-example-of-elliptic-curve-cryptography">An example of Elliptic Curve Cryptography</h2> <p>This sounds good in theory, but let's give it a test drive.</p> <p>Alice and Bob are given the following shared information:</p> <blockquote> <p>p = 3851, E: y² = x³ + 324X + 1287, P = (920, 303) ∈ E(F₃₈₅₁)</p> </blockquote> <p>Alice and Bob choose their secret integers:</p> <blockquote> <p>n_A = 1194 n_B = 1759</p> </blockquote> <p>Alice and Bob then compute their public keys:</p> <blockquote> <p>Alice computes Q_A = 1194P = (2067, 2178) ∈ E(F₃₈₅₁) Bob computes Q_B = 1759P = (3684, 3125) ∈ E(F₃₈₅₁)</p> </blockquote> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>Remember that we use the Double-and-Add algorithm to compute Q_A and Q_B. This invloves iteratively computing the tangent line at a point, the intersection with E at that intersection, and reflecting that point over the X axis.</p> </span> </span> <p>Alice and Bob trade public keys and calculate their shared secret:</p> <blockquote> <p>Alice computes n_AQ_B = 1194(3684, 3125) = (3347, 1242) ∈ E(F₃₈₅₁) Bob computes n_BQ_A = 1759(2067, 2178) = (3347, 1242) ∈ E(F₃₈₅₁)</p> </blockquote> <p>Therefore (3347, 1242) is the shared secret.</p> <h2 id="why-elliptic-curve-cryptography-is-useful">Why Elliptic Curve Cryptography is useful</h2> <p>While it is harder than simply multiplying mod p for Alice to compute her shared secret (which is the case in RSA), it is <em>even harder</em> for a malicious actor to figure out that same shared secret. This point is best put by the source <em>A (relatively easy to understand) primer on elliptic curve cryptography</em><sup class="footnote-reference"><a href="#2">2</a></sup>:</p> <blockquote> <p>You can compute how much energy is needed to break a cryptographic algorithm and compare that with how much water that energy could boil. This is a kind of a cryptographic carbon footprint. By this measure, breaking a 228-bit RSA key requires less energy than it takes to boil a teaspoon of water. Comparatively, breaking a 228-bit elliptic curve &gt; key requires enough energy to boil all the water on earth. For this &gt; level of security with RSA, you'd need a key with 2,380 bits.</p> </blockquote> <p>So an Elliptic Curve Cryptography key can be one magnitude smaller in size and offer the same level of security as RSA.</p> <p>We can put this in more concrete terms: the fastest algorithm to solve the Elliptic Curve Discrete Logarithm Problem, which Elliptic DHKE security is built upon, in E(F_p) takes √p steps. This is much more difficult than the 'vanilla' Discrete Logarithm Problem.</p> <h2 id="notes-and-edge-cases">Notes and edge cases</h2> <p>Elliptic Curve Cryptography, much like the rest of Cryptography, deals heavily with <a href="https://en.wikipedia.org/wiki/Number_theory">Number Theory</a>. Despite my best efforts most of the nitty-gritty Number Theory in this topic went <em>way</em> over my head. As a result I didn't include much of that kind of stuff and instead focused on the things I <em>could</em> share and sound smart about.</p> <p>Here are some other things about Elliptic Curve Cryptography I didn't cover that deserve more air time:</p> <ul> <li>The Elliptic Curve chosen must meet a special set of criteria; any old Elliptic Curve won't do. This was the cause of a cryptographic breach with Elliptic Curve Cryptography a few years ago that triggered doubts about Elliptic Curve Cryptography as a whole.</li> <li>Some primes cause solving the Elliptic Curve Discrete Logarithm Problem for E(F_p) to be easier than the Discrete Logarithm Problem, these primes can be computed and should be avoided.</li> <li>If you want a deeper understanding of the theory of Elliptic Curves (addition of points on these curves, etc) you should look into <a href="https://en.wikipedia.org/wiki/Algebraic_geometry">algebraic geometry</a>.</li> </ul> <h2 id="annotated-bibliography">Annotated Bibliography</h2> <p>An Introduction to Mathematical Cryptography<sup class="footnote-reference"><a href="#3">3</a></sup> The chapter in this textbook on Elliptic Curves in Cryptography established the bedrock understanding of the topic of Elliptic Curve Cryptography. This ended up being the main resource for this post and offered a great median between "Regular Joe's guide to Elliptic Curve Cryptography" and "The graduate student's guide to Elliptic Curve Cryptography" which were my other two resources. It was also the source of all examples, which were very useful in gaining an intuitive understanding of the material.</p> <p>A (relatively easy to understand) primer on elliptic curve cryptography<sup class="footnote-reference"><a href="#4">4</a></sup> This blog post was my <em>second</em> source and did a good job of taking the proofs and dense material in Intro to Math Cyrpto (above) and boiled it down to the important stuff. It drastically improved further readings of the original textbook and provided that great animated image of adding P ⊕ Q in E(F_p). It didn't cover any of the Number Theory, but explained the historical context of Elliptic Curve Cryptography, roughly how/why it works, and did a good job of describing it's impact in our world today.</p> <p>Cryptography: An Introduction<sup class="footnote-reference"><a href="#5">5</a></sup> This wasn't a resource I actually <em>used</em>, but I did read the chapter on Elliptic Curve Cryptography (chapter 2!). It gave me an appreciation for the previous two sources and some exposure to the other ways Elliptic Curves can be taught.</p> <h2 id="errata">Errata</h2> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p>A (relatively easy to understand) primer on elliptic curve cryptography, October 24, 2013, Nick Sullivan, Cloudflare blog, reposted on Ars Technica, <a href="https://arstechnica.com/information-technology/2013/10/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/">https://arstechnica.com/information-technology/2013/10/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/</a></p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p>A (relatively easy to understand) primer on elliptic curve cryptography, October 24, 2013, Nick Sullivan, Cloudflare blog, reposted on Ars Technica, <a href="https://arstechnica.com/information-technology/2013/10/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/">https://arstechnica.com/information-technology/2013/10/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/</a></p> </div> <div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup> <p>An Introduction to Mathematical Cryptography, 2008, Jeffery Hoffstein, Jill Pipher, Joseph H. Silverman, Springer Publishing, ISBN 978-0-387-77993-5</p> </div> <div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup> <p>A (relatively easy to understand) primer on elliptic curve cryptography, October 24, 2013, Nick Sullivan, Cloudflare blog, reposted on Ars Technica, <a href="https://arstechnica.com/information-technology/2013/10/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/">https://arstechnica.com/information-technology/2013/10/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/</a></p> </div> <div class="footnote-definition" id="5"><sup class="footnote-definition-label">5</sup> <p>Cryptography: An Introduction (Third Edition), May 19, 2016, Nigel Smart, <a href="https://www.cs.umd.edu/~waa/414-F11/IntroToCrypto.pdf">https://www.cs.umd.edu/~waa/414-F11/IntroToCrypto.pdf</a></p> </div> 2017-07-10T00:00:00+00:00 2017-07-10T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/independent-crypto/ <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>This is a part of a series of blog posts I wrote for an Independent Study on cryptography at Oregon State University. To read all of the posts, check out the 'Independent Crypto' tag. {% end() %}</p> <p>{% warning() %} This syllabus was written by an Oregon State University undergraduate student and not by an Oregon State University staff member. This should explain any irregularities in the structure and substance of the document.</p> </span> </span> <p>The purpose of this course ("Independent Crypto") is to give students an opportunity to dive deeper into interesting topics of Cryptography.</p> <p>By the end of this course you should <em>grok</em> the following topics:</p> <ul> <li>Elliptic curve cryptography</li> <li>Memory hard functions</li> <li>Garbled circuits</li> <li>An topic of your choosing (get creative!)</li> </ul> <p>Each of these should take about 40 hours of work to complete. An overview of what that means is outlined below.</p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>This course is designed to be a 4 credit hour independent study.</p> <p>As is standard Oregon State University policy, this corresponds with 160 hours of work over a 10 week period. Plan accordingly.</p> </span> </span> <hr /> <p>Each topic ought to take about 40 hours of work to complete over the course of a 10 week term. If you schedule your time well this will be a piece of cake at just 16 hours per week. That's 2.28 hours per day, 3.2 hours per week-day, or 16 hours the day before your check-in is due!</p> <p>The basic structure is as follows:</p> <ol> <li>Research a topic. Read papers, watch informative videos, ask questions and learn as much as you can about a given topic.</li> <li>Maintain an <em>annotated bibliography</em>. This should include materials found while studying a given topic, a summary of each of the materials, and a final summary of the topic as a whole.</li> <li>Meet weekly with the mentoring professor.</li> <li>Repeat.</li> <li>???</li> <li>Profit.</li> </ol> <p>The end goal, in addition to learning about a breadth of topics in modern security, is to produce an annotated bibliography. This will exercise the student's ability to read and process academic topics, journals, and videos.</p> <p>Of course if you are particularly passionate about a topic you are encouraged to go further: implementing things of interest, investigating new questions, and generally 'digging deeper' as you gain interest in different topics.</p> <p>The following topics do not <em>need</em> to be completed in order, however doing so will result in an optimal 'difficulty curve' as the kids say. The kids do still say that right?</p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>Included are a few resources grabbed in a quick internet search. These are meant to be starting places for each topic, generating questions and providing external resources. You will need to find additional resources for each topic.</p> </span> </span> <h2 id="elliptic-curve-cryptography">Elliptic Curve Cryptography</h2> <p>Weeks 0-2 will be dedicated to Elliptic Curve Cryptography.</p> <h3 id="kickoff-resources">Kickoff Resources</h3> <ul> <li>A (relatively easy to understand) primer on Elliptic Curve Cryptography: <a href="https://arstechnica.com/security/2013/10/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/">https://arstechnica.com/security/2013/10/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/</a></li> <li>Elliptic Curve Cryptography in Practice: <a href="https://eprint.iacr.org/2013/734.pdf">https://eprint.iacr.org/2013/734.pdf</a></li> <li>Elliptic Curve Cryptography, a gentle introduction: <a href="http://andrea.corbellini.name/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/">http://andrea.corbellini.name/2015/05/17/elliptic-curve-cryptography-a-gentle-introduction/</a></li> </ul> <h3 id="kickoff-questions">Kickoff Questions</h3> <ul> <li>What are elliptic curves?</li> <li>How do elliptic curves relate to cryptography?</li> <li>How are Elliptic Curve Cryptography functions different from similar ones like RSA?</li> </ul> <h2 id="memory-hard-functions">Memory-hard functions</h2> <p>Weeks 3 and 4 of the course should be dedicated to the topic of Memory Hard Functions.</p> <h3 id="kickoff-resources-1">Kickoff Resources</h3> <ul> <li>Memory-hard functions and tradeoff cryptanalysis with applications to password hashing, cryptocurrencies, and white-box cryptography: <a href="https://www.cryptolux.org/images/d/d1/Tradeoff-slides.pdf">https://www.cryptolux.org/images/d/d1/Tradeoff-slides.pdf</a></li> <li>Strict Memory Hard Hashing Functions: <a href="http://www.hashcash.org/papers/memohash.pdf">http://www.hashcash.org/papers/memohash.pdf</a></li> </ul> <h3 id="kickoff-questions-1">Kickoff Questions</h3> <ul> <li>What are Memory-hard functions?</li> <li>What purposes are Memory-hard functions used for?</li> <li>What are some examples of Memory-hard functions and how do they work?</li> </ul> <h2 id="garbled-circuits">Garbled circuits</h2> <p>Weeks 5-7 should be dedicated to garbled circuits.</p> <h3 id="kickoff-resources-2">Kickoff Resources</h3> <ul> <li>Garbled Circuits: <a href="https://youtu.be/TxCu1L_tzlU">https://youtu.be/TxCu1L_tzlU</a></li> <li>Foundations of Garbled Circuits: <a href="https://eprint.iacr.org/2012/265.pdf">https://eprint.iacr.org/2012/265.pdf</a></li> <li>Faster Secure Two-Party Computation Using Garbled Circuits: <a href="https://www.usenix.org/legacy/event/sec11/tech/full_papers/Huang.pdf">https://www.usenix.org/legacy/event/sec11/tech/full_papers/Huang.pdf</a></li> <li>SFE: Yao’s Garbled Circuit: <a href="https://courses.engr.illinois.edu/cs598man/fa2009/slides/ac-f09-lect16-yao.pdf">https://courses.engr.illinois.edu/cs598man/fa2009/slides/ac-f09-lect16-yao.pdf</a> (bonus points if you find the talk for these slides).</li> <li>Garbled Circuts, Cryptowiki, <a href="http://cryptowiki.net/index.php?title=Garbled_circuits">http://cryptowiki.net/index.php?title=Garbled_circuits</a> (probably don't cite this one in a paper)</li> <li>Amortizing Garbled Circuits: <a href="https://eprint.iacr.org/2015/081.pdf">https://eprint.iacr.org/2015/081.pdf</a></li> </ul> <h2 id="independent-study">Independent study</h2> <p>In the last seven or so weeks you've learned a lot. You've read papers, watch informative lectures, and had insightful conversations with peers and mentors. Many of these probably sparked your attention in a particular topic. Use these last few weeks to investigate one of those sparks that you've been itching to learn more about.</p> <p>If you truly feel uninspired you can use this time to learn about Private Set Intersection.</p> <h3 id="kickoff-resources-for-private-set-intersection">Kickoff Resources for Private Set Intersection</h3> <ul> <li>BIU Winter School on Cryptography on Youtube.</li> <li>CSCI E-127, Introduction to Cryptography <a href="http://cm.dce.harvard.edu/2014/01/14301/publicationListing.shtml">http://cm.dce.harvard.edu/2014/01/14301/publicationListing.shtml</a></li> </ul> <h3 id="generic-kickoff-questions">Generic Kickoff Questions</h3> <ul> <li>What topic are you investigating?</li> <li>How does this relate to Cryptography?</li> <li>How would you explain this topic to your friends or parents?</li> <li>Why is this topic important?</li> <li>What interests you about this topic?</li> </ul> <h2 id="additional-resources">Additional resources</h2> <p>OSU Professor <a href="http://web.engr.oregonstate.edu/~rosulekm/">Mike Rosulek</a> volunteered the following additional resources:</p> <ul> <li>Scrypt is maximally memory-hard, <a href="http://www.cs.bu.edu/fac/reyzin/papers/BostonCryptoDayTalk-Leo.pptx">http://www.cs.bu.edu/fac/reyzin/papers/BostonCryptoDayTalk-Leo.pptx</a></li> <li>Practical Graphs for Optimal Side-Channel Resistant Memory-Hard Functions, <a href="https://eprint.iacr.org/2017/443.pdf">https://eprint.iacr.org/2017/443.pdf</a></li> <li>Efficiently Computing Data Independent Memory Hard Functions, <a href="https://youtu.be/ujpvPtn_N5Y">https://youtu.be/ujpvPtn_N5Y</a></li> <li>Memory hard Functions and Password Hashing, <a href="https://youtu.be/9yX4v89m5oo">https://youtu.be/9yX4v89m5oo</a></li> <li>Towards a Theory of Data-Independent Memory Hard Functions, <a href="https://youtu.be/YtfVLzUkwME">https://youtu.be/YtfVLzUkwME</a></li> <li>Depth-Robust Graphs and Their Cumulative Memory Complexity, <a href="https://eprint.iacr.org/2016/875.pdf">https://eprint.iacr.org/2016/875.pdf</a></li> <li>Practical Garbled Circuit Optimizations, <a href="http://web.engr.oregonstate.edu/~rosulekm/pubs/gc-survey-talk.pdf">http://web.engr.oregonstate.edu/~rosulekm/pubs/gc-survey-talk.pdf</a></li> <li>Pratical Garbled Circuit Optimizations, <a href="https://youtu.be/FTxh908u9y8">https://youtu.be/FTxh908u9y8</a></li> <li>Cache-timing attacks on AES, <a href="http://cr.yp.to/antiforgery/cachetiming-20050414.pdf">http://cr.yp.to/antiforgery/cachetiming-20050414.pdf</a></li> <li>Lucky Thirteen attack on TLS CBC, <a href="https://www.imperialviolet.org/2013/02/04/luckythirteen.html">https://www.imperialviolet.org/2013/02/04/luckythirteen.html</a></li> </ul> 2016-12-12T00:00:00+00:00 2016-12-12T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/buildbot-on-kubernetes/ <p><img src="/images/buildbot-on-k8s/kubernetes-logo.png" alt="kubernetes logo" /></p> <hr /> <blockquote> <p><strong>TLDR:</strong> If you just want to see the end-result of this post, the results can be found at this GitHub Repository: <a href="https://github.com/ElijahCaine/buildbot-on-kubernetes">https://github.com/ElijahCaine/buildbot-on-kubernetes</a></p> </blockquote> <h2 id="preamble">Preamble</h2> <p>I'm learning Kubernetes (K8s) for work and decided to try my hand at deploying Buildbot with K8s because we all know the universal law discovered made by Science McSmartyPants in the 1758 which stated: <strong>doing cool shit &gt; reading docs</strong> <sup class="footnote-reference"><a href="#docs-should">1</a></sup>. I would describe K8s and Buildbot, but they each already did that:</p> <blockquote> <p>"Buildbot is an open-source framework for automating software build, test, and release processes."</p> <p>- <a href="http://buildbot.net/">Buildbot.net</a></p> </blockquote> <hr /> <blockquote> <p>Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.</p> <p>- <a href="http://kubernetes.io/">Kubernetes.io</a></p> </blockquote> <hr /> <p>If that still didn't make sense, don't worry you should keep reading. This is a fun post.</p> <blockquote> <p><strong>NOTE:</strong> As you read this post keep in mind: <em>it might walk like a tutorial, quack like a tutorial, and even read like a tutorial, but I promise you that it is in fact</em> <strong>not</strong> <em>a tutorial.</em></p> <p><strong>This post is an adventure.</strong></p> </blockquote> <h3 id="squad-goals">Squad Goals</h3> <p>Here's a quick rundown of what I want to achieve:</p> <ul> <li>Deploy an instance of Buildbot on K8s.</li> <li>Have that instance scale it's number of workers automagically depending on the amount of work being asked of it.</li> <li>Share all relevant storage between replicated containers (e.g., databases, builds, etc).</li> </ul> <h3 id="why">Why?</h3> <p>I have never deployed Buildbot ever for anything. I have also not really worked with K8s until starting my recent jorb at <a href="https://coreos.com">CoreOS</a>. Why Buildbot and why K8s?</p> <ol> <li> <p>I have to learn K8s sooner or later for work.</p> </li> <li> <p>I want to deploy something a little more complicated than Nginx. As fun as that is, every K8s tutorial uses that as an example and it's getting <em>oooooold</em>.</p> </li> <li> <p>Buildbot seems like a good K8s challenge b/c:</p> <ol> <li>It has 3 moving parts (master, worker, database)</li> <li>It could benefit from the fancy-dancy auto-scaling features built into K8s (e.g., high workload -&gt; add more workers)</li> </ol> </li> </ol> <p>Anyway, let's get started.</p> <h3 id="my-sick-rig">My sick rig</h3> <p>I'm using the latest version of K8s and <a href="https://github.com/kubernetes/minikube#minikube">Minikube</a>, which is backed by Virtualbox on a 2014 MacBook Pro running OSX.</p> <p>Minikube version and associated OS:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ minikube version </span><span>minikube version: v0.13.1 </span><span>$ minikube ssh </span><span>... </span><span>Boot2Docker version 1.11.1, build master : 901340f - Fri Jul 1 22:52:19 UTC 2016 </span><span>Docker version 1.11.1, build 5604cbe </span><span>docker@minikube:~$ </span></code></pre> <p>There's some Minikube-specific semantics in this post, but you can probably get by with whatever K8s back-end you want/have lying around.</p> <p>Kubernetes version:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl version </span><span>Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.6&quot;, GitCommit:&quot;e569a27d02001e343cb68086bc06d47804f62af6&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-11-12T05:22:15Z&quot;, GoVersion:&quot;go1.7.1&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;} </span><span>Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.6&quot;, GitCommit:&quot;e569a27d02001e343cb68086bc06d47804f62af6&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;1970-01-01T00:00:00Z&quot;, GoVersion:&quot;go1.7.1&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;} </span></code></pre> <p>Between finishing editing and pushing this post the above versions will probably be out of date. <em>SHRUG</em>. What are you gonna do. Software, amirite?</p> <p>Virtualbox version:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ virtualbox --help </span><span>Oracle VM VirtualBox Manager 5.1.6 </span><span>... </span></code></pre> <p>OSX Version and hardware:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ sw_vers </span><span>ProductName: Mac OS X </span><span>ProductVersion: 10.10.5 </span><span>BuildVersion: 14F1912 </span><span>$ system_profiler </span><span>... </span><span> Hardware Overview: </span><span> Model Name: MacBook Pro </span><span> Model Identifier: MacBookPro11,1 </span><span> Processor Name: Intel Core i5 </span><span> Processor Speed: 2.6 GHz </span><span> Number of Processors: 1 </span><span> Total Number of Cores: 2 </span><span> ... </span><span> Memory: 16 GB </span><span> ... </span><span>... </span></code></pre> <p>The hardware and Virtualbox versions are a little less important, but might as well be included for completeness.</p> <h2 id="first-pass">First Pass</h2> <p>The Buildbot project is nice enough to provide <a href="https://docs.buildbot.net/current/tutorial/docker.html">some Buidlbot Docker infrastructure</a> to start working with. It uses a <a href="https://docs.docker.com/compose/">Docker Compose</a> YAML file to deploy one worker container, one master service, and one PostgreSQL service, each of which is linked together and <em>just works™</em></p> <p>Great! In theory <sup class="footnote-reference"><a href="#theory-vs-practice">2</a></sup> we can just translate the options in <code>docker-compose.yml</code> to K8s options to get a simple cluster up and running. Which, to clarify, isn't an established <strong>thing</strong>, it's just a process I'm guessing should work based on the fact that Docker Compose and K8s are both orchestration tools. One is <strong>way</strong> more complicated and robust, but At least <strong>some</strong> of their features should over-lap in that Venn diagram.</p> <p>Once we've got a nieve translated-docker-compose k8s setup running then we can (hopefully) tweak some knobs and get persistent storage and auto-scaling working <sup class="footnote-reference"><a href="#why">3</a></sup>.</p> <h3 id="throw-some-containers-at-the-wall-and-see-what-sticks">Throw some containers at the wall and see what sticks</h3> <p>Let's start really basic and just try to get something running with <code>kubectl run</code>. We'll use K8s to deploy a Buildbot <code>master</code> container mentioned in that <code>docker-compose.yml</code> with translated configuration options from that file. Remember, we're just mapping a <code>docker-compose.yml</code> into a K8s setup to begin. Nothing fancy, no pre-emptive optimizations, just this:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl run master \ </span><span> --image=buildbot/buildbot-master:master \ </span><span> --env=&quot;BUILDBOT_CONFIG_DIR=config&quot; \ </span><span> --env=&quot;BUILDBOT_CONFIG_URL=https://github.com/buildbot/buildbot-docker-example-config/archive/master.tar.gz&quot; \ </span><span> --env=&quot;BUILDBOT_WORKER_PORT=9989&quot; \ </span><span> --env=&quot;BUILDBOT_WEB_URL=http://localhost:8080/&quot; \ </span><span> --env=&quot;BUILDBOT_WEB_PORT=8080&quot; \ </span><span> --port=8080 </span></code></pre> <p>Some reading later and I can tell you that command started a <em>Deployment</em> of <em>Pods</em>. To see if it worked, let's run <code>kubect get pods</code>.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get pods </span><span>NAME READY STATUS RESTARTS AGE </span><span>master-4259088255-afsfk 1/1 Running 1 10s </span></code></pre> <p>This looks pretty good...</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get pods </span><span>NAME READY STATUS RESTARTS AGE </span><span>master-4259088255-afsfk 0/1 CrashLoopBackOff 2 1m </span></code></pre> <p>... nooo! The thing was at 1/1 and now it's at 0/1 <strong>and</strong> it says <strong>CrashLoopBackOff</strong>. Numbers going down when they're supposed to stay the same is never a good sign, and crashing is almost never what you want.</p> <p>If I've learned <em>anything</em> about fixing stuff that's broke it's <em>always check the logs</em>.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl logs po/master-4259088255-afsfk </span><span>[...] </span><span>2016-12-09 22:31:42+0000 [-] Setting up database with URL &#39;sqlite:&#39; </span><span>2016-12-09 22:31:42+0000 [-] The Buildmaster database needs to be upgraded before this version of </span><span>[...] </span><span>2016-12-09 22:31:42+0000 [-] BuildMaster startup failed </span><span>2016-12-09 22:31:42+0000 [-] BuildMaster is stopped </span><span>2016-12-09 22:31:42+0000 [-] Main loop terminated. </span><span>2016-12-09 22:31:42+0000 [-] Server Shut Down. </span></code></pre> <p>Gross, but probably useful. How? Good question:</p> <ol> <li>I saw the word <code>database</code>.</li> <li>We <em>didn't</em> deploy a database.</li> <li>QED let's do that.</li> </ol> <h3 id="add-one-cup-of-postgres-to-the-mix">Add one cup of Postgres to the mix</h3> <p>Just like with the <code>master</code> container, we're just going to use CLI arguments to get a database running.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl run postgres \ </span><span> --image=postgres:9.4\ </span><span> --env=&quot;POSTGRES_PASSWORD=change_me&quot; \ </span><span> --env=&quot;POSTGRES_USER=buildbot&quot; \ </span><span> --env=&quot;POSTGRES_DB=buildbot&quot; \ </span><span> --env=&quot;BUILDBOT_DB_URL=postgresql+psycopg2://{POSTGRES_USER}:{POSTGRES_PASSWORD}@db/{POSTGRES_DB}&quot;\ </span><span> --port=5432 </span></code></pre> <p>Cross fingers aaand...</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get pods </span><span>NAME READY STATUS RESTARTS AGE </span><span>master-4259088255-afsfk 0/1 CrashLoopBackOff 6 9m </span><span>postgres-2443857112-3ermh 0/1 ContainerCreating 0 15s </span></code></pre> <p><em>/me holds breath</em></p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get pods </span><span>NAME READY STATUS RESTARTS AGE </span><span>master-4259088255-afsfk 0/1 CrashLoopBackOff 6 9m </span><span>postgres-2443857112-3ermh 1/1 Running 0 54s </span></code></pre> <p>Yuss! Wait, for real?</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl logs postgres-2443857112-3ermh </span><span>[... hey look a bunch of useful Postgres garbage ...] </span><span>PostgreSQL init process complete; ready for start up. </span><span>[... some more useful Postgres garbage ...] </span></code></pre> <p>Good 'nuff. Now how does this database plug into the master container?</p> <p>Well, in the Docker Compose world, containers talked to one-another on a private network with the <code>link</code> directive. There's probably some way to do we do that with K8s right?</p> <h3 id="welcome-yaml-config-files-to-the-class">Welcome YAML config files to the class</h3> <p>Running Command-Line Interface (CLI) commands is fun, but the easiest way to get this system running seems to be with configuration files. I'm sure I <em>can</em> use CLI commands to orchestrate this entire project, but... honestly all the tutorials talk about how to do things in YAML so we're doing it in YAML now.</p> <p>Some research later it looks like declaring a <a href="http://kubernetes.io/docs/user-guide/pods/">Pod</a> is the way to go? For context, here's where I got the idea from the K8s docs:</p> <blockquote> <p>A pod (as in a pod of whales or pea pod) is a group of one or more containers (such as Docker containers), the shared storage for those containers, and options about how to run the containers.</p> </blockquote> <p>Well that sounds <em>roughly</em> like what we're doing. I've got some containers, I want them to be able to talk to each other, and they're all logically connected to one another. Let's go down this rabbit hole.</p> <p><img src="/images/buildbot-on-k8s/alice-down.gif" alt="Alice down the rabbit hole..." /></p> <p>Here I have Frankensteined this config <code>buildbot.yaml</code> from examples in the configs:</p> <pre data-lang="yaml" style="background-color:#191919;color:#f8f8f2;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">v1</span><span style="color:#ffffff;">&#39; </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">Pod</span><span style="color:#ffffff;">&#39; </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">buildbot</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">buildbot</span><span style="color:#ffffff;">&#39; </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">containers</span><span>: </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">master</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">image</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">buildbot/buildbot-master:master</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">containerPort</span><span>: </span><span style="color:#fdb082;">8080 </span><span> </span><span style="color:#ff5e5e;">env</span><span>: </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">BUILDBOT_CONFIG_DIR</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">config</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">BUILDBOT_CONFIG_URL</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">https://github.com/buildbot/buildbot-docker-example-config/archive/master.tar.gz</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">BUILDBOT_WORKER_PORT</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">9989</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">BUILDBOT_WEB_URL</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">http://localhost:8080/</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">BUILDBOT_WEB_PORT</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">8080</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">POSTGRES_PASSWORD</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">change_me</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">POSTGRES_USER</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">buildbot</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">POSTGRES_DB</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">buildbot</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">BUILDBOT_DB_URL</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">postgresql+psycopg2://{POSTGRES_USER}:{POSTGRES_PASSWORD}@db/{POSTGRES_DB}</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">postgres</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">image</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">postgres:9.4</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">containerPort</span><span>: </span><span style="color:#fdb082;">5432 </span><span> </span><span style="color:#ff5e5e;">env</span><span>: </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">POSTGRES_PASSWORD</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">change_me</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">POSTGRES_USER</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">buildbot</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">POSTGRES_DB</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">buildbot</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">BUILDBOT_DB_URL</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">postgresql+psycopg2://{POSTGRES_USER}:{POSTGRES_PASSWORD}@db/{POSTGRES_DB}</span><span style="color:#ffffff;">&#39; </span></code></pre> <p>What happens when we run it?</p> <p>Well first we need to clean up that CLI-created garbage we were doing earlier.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl delete deployment master postgres </span><span>deployment &quot;master&quot; deleted </span><span>deployment &quot;postgres&quot; deleted </span></code></pre> <p>Then deploy the new pod:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl create -f buildbot.yaml </span><span>pod &quot;buildbot&quot; created </span></code></pre> <p>Aaaand:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get pods </span><span>buildbot 2/2 Running 0 23s </span></code></pre> <p>I'm suspicious...</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl logs po/buildbot master </span><span>checking basedir </span><span>/usr/lib/python2.7/site-packages/buildbot/config.py:85: ConfigWarning: [0.9.0 and later] `buildbotNetUsageData` is not configured and defaults to basic </span><span>[...] </span><span>Failure: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name &quot;db&quot; to address: Try again </span><span>[...] </span><span>problem while upgrading!: </span><span>Traceback (most recent call last): </span><span>[... python traceback ...] </span><span>OperationalError: (psycopg2.OperationalError) could not translate host name &quot;db&quot; to address: Try again </span></code></pre> <p>Whelp let's play 'Where do I fix <em>that</em>?'</p> <p>Is the problem in:</p> <ol> <li>The <code>buildbot-master</code> image</li> <li>The <code>postgres</code> image</li> <li>The <code>buildbot.yml</code> file</li> </ol> <p>If you guessed <strong>c</strong> you would be right so let's start in <code>buildbot.yaml</code>.</p> <p>Based on my experience with databases, specifically the startup time of database containers like Postgres and MariaDB, I'd <em>guess</em> that the database was taking too long to start. After some digging around, I found out that Pods were the entirely wrong way to go. Here's how that discovery went:</p> <ul> <li><strong>Q</strong>: Does K8s have anything like docker-compose's <code>depends_on:</code>.</li> <li><strong>A</strong>: No.</li> <li><strong>Q</strong>: Okay, so how... actually, is there anything like what I'm doing already out there?</li> <li><strong>A</strong>: Yes.</li> <li><strong>Q</strong>: ... Where?</li> <li><strong>A</strong>: <a href="https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-wordpress-pd">https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-wordpress-pd</a></li> <li><strong>Q</strong>: Noice. So the <em>Pods</em> thing was the wrong rabbit hole to go down?</li> <li><strong>A</strong>: Right.</li> <li><strong>Q</strong>: Right like correct or right like...?</li> <li><strong>A</strong>: Stop.</li> </ul> <h3 id="yay-examples">Yay examples</h3> <p>So let's run that example to make sure it all works fine.</p> <ol> <li>Clone repo</li> <li><code>cd</code> to example</li> <li>Follow README instructions</li> </ol> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl create -f local-volumes.yaml </span><span>persistentvolume &quot;local-pv-1&quot; created </span><span>persistentvolume &quot;local-pv-2&quot; created </span><span>$ kubectl create -f mysql-deployment.yaml </span><span>service &quot;wordpress-mysql&quot; created </span><span>persistentvolumeclaim &quot;mysql-pv-claim&quot; created </span><span>deployment &quot;wordpress-mysql&quot; created </span><span>$ kubectl create -f wordpress-deployment.yaml </span><span>service &quot;wordpress&quot; created </span><span>persistentvolumeclaim &quot;wp-pv-claim&quot; created </span><span>deployment &quot;wordpress&quot; created </span><span>$ kubectl get all </span><span>NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE </span><span>svc/kubernetes 10.0.0.1 &lt;none&gt; 443/TCP 2h </span><span>svc/wordpress 10.0.0.28 &lt;pending&gt; 80/TCP 5s </span><span>svc/wordpress-mysql None &lt;none&gt; 3306/TCP 17s </span><span>NAME READY STATUS RESTARTS AGE </span><span>po/buildbot 2/2 Running 0 8m </span><span>po/wordpress-1618093523-4if9k 0/1 ContainerCreating 0 5s </span><span>po/wordpress-mysql-2379610080-mqvll 0/1 ContainerCreating 0 17s </span><span>NAME STATUS VOLUME CAPACITY ACCESSMODES AGE </span><span>pvc/mysql-pv-claim Bound local-pv-1 20Gi RWO 17s </span><span>pvc/wp-pv-claim Bound local-pv-2 20Gi RWO 5s </span></code></pre> <p>And I can go to the site?</p> <p><img src="/images/buildbot-on-k8s/wordpress-working.png" alt="Wordpress working correctly" /></p> <p>Great.</p> <h3 id="now-use-the-example">Now Use the example</h3> <p>Now we need to morph <code>wordpress-deployment.yaml</code> into <code>postgres.yaml</code>, and <code>wordpress-deployment.yaml</code> into <code>master.yaml</code> Also, let's ignore <code>local-volumes.yaml</code> for now, just to be safe. Don't forget: we're not doing anything fancy yet.</p> <p><code>master.yaml</code>:</p> <pre data-lang="yaml" style="background-color:#191919;color:#f8f8f2;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">v1 </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">Service </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">master </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">port</span><span>: </span><span style="color:#fdb082;">8080 </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">frontend </span><span> </span><span style="color:#ff5e5e;">selector</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> </span><span style="color:#ff5e5e;">tier</span><span>: </span><span style="color:#fbe3bf;">master </span><span> </span><span style="color:#ff5e5e;">type</span><span>: </span><span style="color:#fbe3bf;">NodePort </span><span>--- </span><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">extensions/v1beta1 </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">Deployment </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">master </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">strategy</span><span>: </span><span> </span><span style="color:#ff5e5e;">type</span><span>: </span><span style="color:#fbe3bf;">Recreate </span><span> </span><span style="color:#ff5e5e;">template</span><span>: </span><span> </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> </span><span style="color:#ff5e5e;">tier</span><span>: </span><span style="color:#fbe3bf;">master </span><span> </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">containers</span><span>: </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">master </span><span> </span><span style="color:#ff5e5e;">image</span><span>: </span><span style="color:#fbe3bf;">buildbot/buildbot-master:master </span><span> </span><span style="color:#ff5e5e;">env</span><span>: </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">BUILDBOT_CONFIG_DIR </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">config </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">BUILDBOT_CONFIG_URL </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">https://raw.githubusercontent.com/buildbot/buildbot-docker-example-config/master/master.cfg</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">BUILDBOT_WORKER_PORT </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">9989</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">BUILDBOT_WEB_URL </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">http://localhost:8080/</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">BUILDBOT_WEB_PORT </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">8080</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">POSTGRES_PASSWORD </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">change_me </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">POSTGRES_USER </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">POSTGRES_DB </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">POSTGRES_DB_HOST </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">postgres </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">BUILDBOT_DB_URL </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">postgresql+psycopg2://{POSTGRES_USER}:{POSTGRES_PASSWORD}@{POSTGRES_DB_HOST}/{POSTGRES_DB}</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">containerPort</span><span>: </span><span style="color:#fdb082;">8080 </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">frontend </span></code></pre> <p><code>postgres.yaml:</code></p> <pre data-lang="yaml" style="background-color:#191919;color:#f8f8f2;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">v1 </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">Service </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">postgres </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">port</span><span>: </span><span style="color:#fdb082;">5432 </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">postgres </span><span> </span><span style="color:#ff5e5e;">selector</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> </span><span style="color:#ff5e5e;">tier</span><span>: </span><span style="color:#fbe3bf;">postgres </span><span> </span><span style="color:#ff5e5e;">clusterIP</span><span>: </span><span style="color:#fbe3bf;">None </span><span>--- </span><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">extensions/v1beta1 </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">Deployment </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">postgres </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">strategy</span><span>: </span><span> </span><span style="color:#ff5e5e;">type</span><span>: </span><span style="color:#fbe3bf;">Recreate </span><span> </span><span style="color:#ff5e5e;">template</span><span>: </span><span> </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> </span><span style="color:#ff5e5e;">tier</span><span>: </span><span style="color:#fbe3bf;">postgres </span><span> </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">containers</span><span>: </span><span> - </span><span style="color:#ff5e5e;">image</span><span>: </span><span style="color:#fbe3bf;">postgres:9.4 </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">postgres </span><span> </span><span style="color:#ff5e5e;">env</span><span>: </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">POSTGRES_PASSWORD </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">change_me </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">POSTGRES_USER </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">POSTGRES_DB </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">POSTGRES_DB_HOST </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">postgres </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">BUILDBOT_DB_URL </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">postgresql+psycopg2://{POSTGRES_USER}:{POSTGRES_PASSWORD}@{POSTGRES_DB_HOST}/{POSTGRES_DB}</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">containerPort</span><span>: </span><span style="color:#fdb082;">5432 </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">postgres </span></code></pre> <p>And with a flick of my wand:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl delete pods,deployment,service --all # cleanup the old stuff </span><span>pod &quot;buildbot&quot; deleted </span><span>pod &quot;wordpress-1618093523-4if9k&quot; deleted </span><span>pod &quot;wordpress-mysql-2379610080-mqvll&quot; deleted </span><span>deployment &quot;wordpress&quot; deleted </span><span>deployment &quot;wordpress-mysql&quot; deleted </span><span>service &quot;kubernetes&quot; deleted </span><span>service &quot;wordpress&quot; deleted </span><span>service &quot;wordpress-mysql&quot; deleted </span><span>$ kubectl create -f postrges.yml </span><span>service &quot;postgres&quot; created </span><span>deployment &quot;postgres&quot; created </span><span>$ kubectl create -f master.yml </span><span>service &quot;master&quot; created </span><span>deployment &quot;master&quot; created </span><span>$ minikube service master </span></code></pre> <p>Opens up this wonderful site:</p> <p><img src="/images/buildbot-on-k8s/buildbot-working.png" alt="Buildbot working" /></p> <p>Great, they can talk <sup class="footnote-reference"><a href="#same-app-note">4</a></sup>.</p> <h3 id="hire-a-worker">Hire a worker</h3> <p>So we've got a website and a database, but we're not actually running any builds yet. Let's fix that.</p> <p>Based on what the <code>docker-compose.yml</code> says we should be able to throw this together and have it work:</p> <p><code>worker.yaml</code>:</p> <pre data-lang="yaml" style="background-color:#191919;color:#f8f8f2;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">v1 </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">Service </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">worker </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">port</span><span>: </span><span style="color:#fdb082;">9989 </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">worker </span><span> </span><span style="color:#ff5e5e;">selector</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> </span><span style="color:#ff5e5e;">tier</span><span>: </span><span style="color:#fbe3bf;">worker </span><span> </span><span style="color:#ff5e5e;">clusterIP</span><span>: </span><span style="color:#fbe3bf;">None </span><span>--- </span><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">extensions/v1beta1 </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">Deployment </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">worker </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">strategy</span><span>: </span><span> </span><span style="color:#ff5e5e;">type</span><span>: </span><span style="color:#fbe3bf;">Recreate </span><span> </span><span style="color:#ff5e5e;">template</span><span>: </span><span> </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">app</span><span>: </span><span style="color:#fbe3bf;">buildbot </span><span> </span><span style="color:#ff5e5e;">tier</span><span>: </span><span style="color:#fbe3bf;">worker </span><span> </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">containers</span><span>: </span><span> - </span><span style="color:#ff5e5e;">image</span><span>: </span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">buildbot/buildbot-worker:master</span><span style="color:#ffffff;">&quot; </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">worker </span><span> </span><span style="color:#ff5e5e;">env</span><span>: </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">BUILDMASTER </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">master </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">BUILDMASTER_PORT </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">9989</span><span style="color:#ffffff;">&#39; </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">WORKERNAME </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">example-worker </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">WORKERPASS </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#fbe3bf;">pass </span><span> - </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">WORKER_ENVIRONMENT_BLACKLIST </span><span> </span><span style="color:#ff5e5e;">value</span><span>: </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">DOCKER_BUILDBOT* BUILDBOT_ENV_* BUILDBOT_1* WORKER_ENVIRONMENT_BLACKLIST</span><span style="color:#ffffff;">&#39; </span><span> </span><span style="color:#ff5e5e;">ports</span><span>: </span><span> - </span><span style="color:#ff5e5e;">containerPort</span><span>: </span><span style="color:#fdb082;">9989 </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">worker </span></code></pre> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl create -f worker.yaml </span><span>service &quot;worker&quot; created </span><span>deployment &quot;worker&quot; created </span><span>$ kubectl get -f worker.yaml </span><span>NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE </span><span>svc/worker None &lt;none&gt; 9989/TCP 44s </span><span>NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE </span><span>deploy/worker 1 1 1 0 44s </span></code></pre> <p>Looks promising...</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get pods -l tier=worker </span><span>NAME READY STATUS RESTARTS AGE </span><span>worker-3607067131-qdpfd 1/1 Running 0 1m </span><span>$ kubectl logs worker-3607067131-ke4zb -f </span><span>2016-11-21 19:46:45+0000 [-] Loading buildbot.tac... </span><span>2016-11-21 19:46:45+0000 [-] Loaded. </span><span>2016-11-21 19:46:45+0000 [-] twistd 16.5.0 (/usr/bin/python 2.7.12) starting up. </span><span>2016-11-21 19:46:45+0000 [-] reactor class: twisted.internet.epollreactor.EPollReactor. 2016-11-21 19:46:45+0000 [-] Starting Worker -- version: latest </span><span>2016-11-21 19:46:45+0000 [-] recording hostname in twistd.hostname </span><span>2016-11-21 19:46:45+0000 [-] Starting factory &lt;buildbot_worker.pb.BotFactory instance at 0x7fd8c2339cf8&gt; </span><span>2016-11-21 19:46:45+0000 [-] Connecting to buildbot:9989 </span><span>2016-11-21 19:46:56+0000 [Uninitialized] Connection to buildbot:9989 failed: Connection Refused </span><span>2016-11-21 19:46:56+0000 [Uninitialized] &lt;twisted.internet.tcp.Connector instance at 0x7fd8c233a170&gt; will retry in 2 seconds </span><span>2016-11-21 19:46:56+0000 [-] Stopping factory &lt;buildbot_worker.pb.BotFactory instance at 0x7fd8c2339cf8&gt; </span><span>2016-11-21 19:46:59+0000 [-] Starting factory &lt;buildbot_worker.pb.BotFactory instance at 0x7fd8c2339cf8&gt; </span><span>2016-11-21 19:46:59+0000 [-] Connecting to buildbot:9989 </span><span>2016-11-21 19:47:29+0000 [-] Connection to buildbot:9989 failed: [Failure instance: Traceback (failure with no frames): &lt;class &#39;twisted.internet.error.TimeoutError&#39;&gt;: User timeout caused connection failure. </span><span> ] </span><span>2016-11-21 19:47:29+0000 [-] &lt;twisted.internet.tcp.Connector instance at 0x7fd8c233a170&gt; will retry in 5 seconds </span><span>2016-11-21 19:47:29+0000 [-] Stopping factory &lt;buildbot_worker.pb.BotFactory instance at 0x7fd8c2339cf8&gt; </span></code></pre> <p>Khaaaaaan!</p> <h3 id="connection-issues">Connection issues</h3> <p>Fine, let's get to work debugging. First we'll login to the worker pod and try to ping the <code>buildbot</code> pod since the output makes it seem like there was a timeout between the host and the worker. This usually means they can't reach each other.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl exec -it worker-3607067131-qdpfd bash </span><span>buildbot@worker-3607067131-qdpfd:/buildbot$ ping master </span><span>bash: ping: command not found </span></code></pre> <p>Uhh...</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>buildbot@worker-3607067131-ke4zb:/buildbot$ curl http://master:8080 </span><span>&lt;!DOCTYPE html&gt;[... definitely actually a webpage ...] </span></code></pre> <p>So the worker can reach the master, but can the master reach the worker?</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get pods -l tier=master </span><span>NAME READY STATUS RESTARTS AGE </span><span>master-2152810066-9ip8b 1/1 Running 0 21m </span><span>$ kubectl exec -it master-2152810066-9ip8b sh </span><span>/var/lib/buildbot # ping worker </span><span>PING worker (172.17.0.6): 56 data bytes </span><span>64 bytes from 172.17.0.6: seq=0 ttl=64 time=0.083 ms </span><span>64 bytes from 172.17.0.6: seq=1 ttl=64 time=0.101 ms </span><span>^C </span><span>--- worker ping statistics --- </span><span>2 packets transmitted, 2 packets received, 0% packet loss </span><span>round-trip min/avg/max = 0.083/0.092/0.101 ms </span></code></pre> <p>Okay, so they can connect to one-another. What's the problem then? My best guess is that the <code>master</code> pod needs to have it's special worker port (<code>9989</code>) exposed. So let's add those lines to <code>master.yaml</code>:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span style="color:#75715e;">--- a/blogpost/updated-master.yaml </span><span style="color:#75715e;">+++ b/blogpost/updated-master.yaml </span><span style="color:#75715e;">@@ -8,6 +8,8 @@ spec: </span><span> ports: </span><span> - port: 8080 </span><span> name: frontend </span><span style="color:#a6e22e;">+ - port: 9989 </span><span style="color:#a6e22e;">+ name: worker </span><span> selector: </span><span> app: buildbot </span><span> tier: master </span><span style="color:#75715e;">@@ -55,3 +57,5 @@ spec: </span><span> ports: </span><span> - containerPort: 8080 </span><span> name: frontend </span><span style="color:#a6e22e;">+ - containerPort: 9989 </span><span style="color:#a6e22e;">+ name: worker </span></code></pre> <p>And after tearing everything down and bringing it back up:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get pods -l tier=worker </span><span>NAME READY STATUS RESTARTS AGE </span><span>worker-2552724660-l4kmv 1/1 Running 0 12s </span><span>$ kubectl logs worker-2552724660-l4kmv </span><span>[... logs logs logs ...] </span><span>2016-12-09 23:30:18+0000 [-] Connecting to master:9989 </span><span>2016-12-09 23:30:18+0000 [HangCheckProtocol,client] message from master: attached </span><span>2016-12-09 23:30:18+0000 [HangCheckProtocol,client] message from master: attached </span><span>2016-12-09 23:30:18+0000 [HangCheckProtocol,client] Connected to master:9989; worker is ready </span><span>2016-12-09 23:30:18+0000 [HangCheckProtocol,client] sending application-level keepalives every 600 seconds </span></code></pre> <p>Hey that looks like success to me! How does the front-end look?</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get pods -l tier=master </span><span>NAME READY STATUS RESTARTS AGE </span><span>master-3038604518-jim6q 1/1 Running 0 2m </span><span>$ kubectl port-forward master-3038604518-jim6q 8080 </span></code></pre> <p>And in a browser:</p> <p><img src="/images/buildbot-on-k8s/build-success.png" alt="Buildbot still working." /></p> <p>You may now do a happy dance.</p> <p><img src="/images/buildbot-on-k8s/happy-dance.gif" alt="Happy Dance .gif" /></p> <h2 id="storage">Storage</h2> <p>We've got the three core moving parts of our system, next on our list is adding some persistent storage. Thankfully, we can recycle the MySQL+Wordpress example we used before. Yay examples.</p> <p>So let's add the storage bits back in that we ignored before. This results in configs that look like so (abbreviated for your scrollbar's convenience):</p> <p><code>volumes.yaml</code>:</p> <pre data-lang="yaml" style="background-color:#191919;color:#f8f8f2;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">v1 </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">PersistentVolume </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">local-pv-1 </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">type</span><span>: </span><span style="color:#fbe3bf;">local </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">capacity</span><span>: </span><span> </span><span style="color:#ff5e5e;">storage</span><span>: </span><span style="color:#fbe3bf;">1Gi </span><span> </span><span style="color:#ff5e5e;">accessModes</span><span>: </span><span> - </span><span style="color:#fbe3bf;">ReadWriteOnce </span><span> </span><span style="color:#ff5e5e;">persistentVolumeReclaimPolicy</span><span>: </span><span style="color:#fbe3bf;">Recycle </span><span> </span><span style="color:#ff5e5e;">hostPath</span><span>: </span><span> </span><span style="color:#ff5e5e;">path</span><span>: </span><span style="color:#fbe3bf;">/data/pv-1 </span><span>--- </span><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">v1 </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">PersistentVolume </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">local-pv-2 </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">type</span><span>: </span><span style="color:#fbe3bf;">local </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">capacity</span><span>: </span><span> </span><span style="color:#ff5e5e;">storage</span><span>: </span><span style="color:#fbe3bf;">5Gi </span><span> </span><span style="color:#ff5e5e;">accessModes</span><span>: </span><span> - </span><span style="color:#fbe3bf;">ReadWriteOnce </span><span> </span><span style="color:#ff5e5e;">persistentVolumeReclaimPolicy</span><span>: </span><span style="color:#fbe3bf;">Recycle </span><span> </span><span style="color:#ff5e5e;">hostPath</span><span>: </span><span> </span><span style="color:#ff5e5e;">path</span><span>: </span><span style="color:#fbe3bf;">/data/pv-2 </span><span>--- </span><span style="color:#ff5e5e;">apiVersion</span><span>: </span><span style="color:#fbe3bf;">v1 </span><span style="color:#ff5e5e;">kind</span><span>: </span><span style="color:#fbe3bf;">PersistentVolume </span><span style="color:#ff5e5e;">metadata</span><span>: </span><span> </span><span style="color:#ff5e5e;">name</span><span>: </span><span style="color:#fbe3bf;">local-pv-3 </span><span> </span><span style="color:#ff5e5e;">labels</span><span>: </span><span> </span><span style="color:#ff5e5e;">type</span><span>: </span><span style="color:#fbe3bf;">local </span><span style="color:#ff5e5e;">spec</span><span>: </span><span> </span><span style="color:#ff5e5e;">capacity</span><span>: </span><span> </span><span style="color:#ff5e5e;">storage</span><span>: </span><span style="color:#fbe3bf;">5Gi </span><span> </span><span style="color:#ff5e5e;">accessModes</span><span>: </span><span> - </span><span style="color:#fbe3bf;">ReadWriteOnce </span><span> </span><span style="color:#ff5e5e;">persistentVolumeReclaimPolicy</span><span>: </span><span style="color:#fbe3bf;">Recycle </span><span> </span><span style="color:#ff5e5e;">hostPath</span><span>: </span><span> </span><span style="color:#ff5e5e;">path</span><span>: </span><span style="color:#fbe3bf;">/data/pv-3 </span></code></pre> <p><code>master.yaml</code>:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span>diff --git a/blogpost/volumes-master.yaml b/blogpost/volumes-master.yaml </span><span>index 7cbffd7..c7479e3 100644 </span><span style="color:#75715e;">--- a/blogpost/volumes-master.yaml </span><span style="color:#75715e;">+++ b/blogpost/volumes-master.yaml </span><span style="color:#75715e;">@@ -15,6 +15,19 @@ spec: </span><span> tier: master </span><span> type: NodePort </span><span> --- </span><span style="color:#a6e22e;">+apiVersion: v1 </span><span style="color:#a6e22e;">+kind: PersistentVolumeClaim </span><span style="color:#a6e22e;">+metadata: </span><span style="color:#a6e22e;">+ name: master-pv-claim </span><span style="color:#a6e22e;">+ labels: </span><span style="color:#a6e22e;">+ app: buildbot </span><span style="color:#a6e22e;">+spec: </span><span style="color:#a6e22e;">+ accessModes: </span><span style="color:#a6e22e;">+ - ReadWriteOnce </span><span style="color:#a6e22e;">+ resources: </span><span style="color:#a6e22e;">+ requests: </span><span style="color:#a6e22e;">+ storage: 1Gi </span><span style="color:#a6e22e;">+--- </span><span> apiVersion: extensions/v1beta1 </span><span> kind: Deployment </span><span> metadata: </span><span style="color:#75715e;">@@ -37,7 +50,7 @@ spec: </span><span> - name: BUILDBOT_CONFIG_DIR </span><span> value: config </span><span> - name: BUILDBOT_CONFIG_URL </span><span style="color:#f92672;">- value: &#39;https://raw.githubusercontent.com/buildbot/buildbot-docker-example-config/master/master.cfg&#39; </span><span style="color:#a6e22e;">+ value: &#39;https://raw.githubusercontent.com/ElijahCaine/buildbot-on-kubernetes/master/simple/master.cfg&#39; </span><span> - name: BUILDBOT_WORKER_PORT </span><span> value: &#39;9989&#39; </span><span> - name: BUILDBOT_WEB_URL </span><span style="color:#75715e;">@@ -59,3 +72,10 @@ spec: </span><span> name: frontend </span><span> - containerPort: 9989 </span><span> name: worker </span><span style="color:#a6e22e;">+ volumeMounts: </span><span style="color:#a6e22e;">+ - name: master-persistent-storage </span><span style="color:#a6e22e;">+ mountPath: /var/lib/buildbot/builds </span><span style="color:#a6e22e;">+ volumes: </span><span style="color:#a6e22e;">+ - name: master-persistent-storage </span><span style="color:#a6e22e;">+ persistentVolumeClaim: </span><span style="color:#a6e22e;">+ claimName: master-pv-claim </span></code></pre> <p><code>postgres.yaml</code>:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span style="color:#75715e;">--- a/blogpost/volumes-postgres.yaml </span><span style="color:#75715e;">+++ b/blogpost/volumes-postgres.yaml </span><span style="color:#75715e;">@@ -13,6 +13,19 @@ spec: </span><span> tier: postgres </span><span> clusterIP: None </span><span> --- </span><span style="color:#a6e22e;">+apiVersion: v1 </span><span style="color:#a6e22e;">+kind: PersistentVolumeClaim </span><span style="color:#a6e22e;">+metadata: </span><span style="color:#a6e22e;">+ name: postgres-pv-claim </span><span style="color:#a6e22e;">+ labels: </span><span style="color:#a6e22e;">+ app: buildbot </span><span style="color:#a6e22e;">+spec: </span><span style="color:#a6e22e;">+ accessModes: </span><span style="color:#a6e22e;">+ - ReadWriteOnce </span><span style="color:#a6e22e;">+ resources: </span><span style="color:#a6e22e;">+ requests: </span><span style="color:#a6e22e;">+ storage: 5Gi </span><span style="color:#a6e22e;">+--- </span><span> apiVersion: extensions/v1beta1 </span><span> kind: Deployment </span><span> metadata: </span><span style="color:#75715e;">@@ -45,3 +58,10 @@ spec: </span><span> ports: </span><span> - containerPort: 5432 </span><span> name: postgres </span><span style="color:#a6e22e;">+ volumeMounts: </span><span style="color:#a6e22e;">+ - name: postgres-persistent-storage </span><span style="color:#a6e22e;">+ mountPath: /var/lib/postgresql </span><span style="color:#a6e22e;">+ volumes: </span><span style="color:#a6e22e;">+ - name: postgres-persistent-storage </span><span style="color:#a6e22e;">+ persistentVolumeClaim: </span><span style="color:#a6e22e;">+ claimName: postgres-pv-claim </span></code></pre> <p><code>worker.yaml</code>:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span style="color:#75715e;">--- a/blogpost/volumes-worker.yaml </span><span style="color:#75715e;">+++ b/blogpost/volumes-worker.yaml </span><span style="color:#75715e;">@@ -13,6 +13,19 @@ spec: </span><span> tier: worker </span><span> clusterIP: None </span><span> --- </span><span style="color:#a6e22e;">+apiVersion: v1 </span><span style="color:#a6e22e;">+kind: PersistentVolumeClaim </span><span style="color:#a6e22e;">+metadata: </span><span style="color:#a6e22e;">+ name: worker-pv-claim </span><span style="color:#a6e22e;">+ labels: </span><span style="color:#a6e22e;">+ app: buidlbot </span><span style="color:#a6e22e;">+spec: </span><span style="color:#a6e22e;">+ accessModes: </span><span style="color:#a6e22e;">+ - ReadWriteOnce </span><span style="color:#a6e22e;">+ resources: </span><span style="color:#a6e22e;">+ requests: </span><span style="color:#a6e22e;">+ storage: 5Gi </span><span style="color:#a6e22e;">+--- </span><span> apiVersion: extensions/v1beta1 </span><span> kind: Deployment </span><span> metadata: </span><span style="color:#75715e;">@@ -45,3 +58,10 @@ spec: </span><span> ports: </span><span> - containerPort: 9989 </span><span> name: worker </span><span style="color:#a6e22e;">+ volumeMounts: </span><span style="color:#a6e22e;">+ - name: worker-persistent-storage </span><span style="color:#a6e22e;">+ mountPath: /buildbot/builds </span><span style="color:#a6e22e;">+ volumes: </span><span style="color:#a6e22e;">+ - name: worker-persistent-storage </span><span style="color:#a6e22e;">+ persistentVolumeClaim: </span><span style="color:#a6e22e;">+ claimName: worker-pv-claim </span></code></pre> <p>And of lastly we edit the Buildbot <code>master.cfg</code> to use our custom paths for storing and carrying out builds:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span>diff --git a/blogpost/volumes-master.cfg b/blogpost/volumes-master.cfg </span><span>index 06ad65b..36212ea 100644 </span><span style="color:#75715e;">--- a/blogpost/volumes-master.cfg </span><span style="color:#75715e;">+++ b/blogpost/volumes-master.cfg </span><span style="color:#75715e;">@@ -78,7 +78,9 @@ c[&#39;builders&#39;] = [] </span><span> c[&#39;builders&#39;].append( </span><span> util.BuilderConfig(name=&quot;runtests&quot;, </span><span> workernames=[&quot;example-worker&quot;], </span><span style="color:#f92672;">- factory=factory)) </span><span style="color:#a6e22e;">+ factory=factory, </span><span style="color:#a6e22e;">+ builddir=&#39;builds&#39;, </span><span style="color:#a6e22e;">+ workerbuilddir=&#39;builds&#39;)) </span><span> </span><span> ####### STATUS TARGETS </span><span> </span></code></pre> <p>This is just an edited version of the original <code>master.cfg</code> you can find in the original Buildbot docker-compose example repository.</p> <p>So this <em>should</em> just work, right? We request a volume of a given size, the volume exists and is the correct size, badda bing badda boom, do the thing like this:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl delete service,deployment,pods --all </span><span>service &quot;master&quot; deleted </span><span>service &quot;postgres&quot; deleted </span><span>service &quot;worker&quot; deleted </span><span>service &quot;kubernetes&quot; deleted </span><span>deployment &quot;master&quot; deleted </span><span>deployment &quot;postgres&quot; deleted </span><span>deployment &quot;worker&quot; deleted </span><span>$ kubectl create -f volumes.yaml </span><span>persistentvolume &quot;local-pv-1&quot; created </span><span>persistentvolume &quot;local-pv-2&quot; created </span><span>persistentvolume &quot;local-pv-3&quot; created </span><span>$ kubectl create -f postgres.yaml </span><span>service &quot;postgres&quot; created </span><span>persistentvolumeclaim &quot;postgres-pv-claim&quot; created </span><span>deployment &quot;postgres&quot; created </span><span>$ kubectl create -f master.yaml </span><span>service &quot;master&quot; created </span><span>persistentvolumeclaim &quot;master-pv-claim&quot; created </span><span>deployment &quot;master&quot; created </span><span>$ kubectl create -f worker.yaml </span><span>service &quot;-worker&quot; created </span><span>persistentvolumeclaim &quot;worker-pv-claim&quot; created </span><span>deployment &quot;worker&quot; created </span><span>$ kubectl get pods -l tier=master </span><span>NAME READY STATUS RESTARTS AGE </span><span>master-3176013930-gvy2i 1/1 Running 0 1m </span><span>$ kubectl port-forward master-3176013930-gvy2i 8080 </span><span>Forwarding from 127.0.0.1:8080 -&gt; 8080 </span><span>Forwarding from [::1]:8080 -&gt; 8080 </span></code></pre> <p>That kinda works, but it kept getting weird arbitrary errors which looked suspicious.</p> <p><img src="/images/buildbot-on-k8s/permissions-build-fail.png" class="align-center" style="width:100.0%" alt="Permissions fail in buildbot" /></p> <p>Since the only thing that had really changed was the volumes I figured I might as well login to the host and investigate.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ minikube ssh </span><span>docker@minikube:~$ ls -alh /data/ </span><span>total 20 </span><span>drwxr-xr-x 5 root root 4.0K Dec 11 02:21 ./ </span><span>drwxr-xr-x 6 root root 4.0K Dec 11 01:48 ../ </span><span>drwxr-xr-x 2 root root 4.0K Dec 11 02:20 pv-1/ </span><span>drwxr-xr-x 2 root root 4.0K Dec 11 02:33 pv-2/ </span><span>drwxr-xr-x 3 root root 4.0K Dec 11 02:33 pv-3/ </span></code></pre> <p>Ahah! That's the problem, root owns everything but the builds are run by the <code>buildbot</code> user (<code>uid: 1000</code>) <em>/me shakes fist at permissions errors.</em> The last first place you think to look.</p> <p>So is there any way to change the permissions of a volume as you claim it (i.e., in <code>volumes.yaml</code>)? We can both go through this journey together or I can share with you a quote from my coworker Barak Michener, the BAMF that works on <a href="https://github.com/coreos/torus">Torus</a>.</p> <blockquote> <p><em>Barak Michener</em></p> <p>yeah, hostPath is completely hands-off from k8s perspective so you just chmod the underlying dir to whatever uid you're using inside the pod</p> </blockquote> <p>Turns out the problem was with my <code>type</code> of storage in <code>volumes.yaml</code>. As of yet there is no type-agnostic way to assign permissions for mounted volumes. Storage, why you gotta be like that?</p> <p>So after digging around and asking Barak I finally concluded that I had to had to login to my host and set the correct permissions on the directories being reserved. Here's how you do that on Minikube:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ minikube ssh </span><span>docker@minikube$ sudo chown -R 1000:1000 /data/pv* </span></code></pre> <p>In this case we could just chown the one volume that's going to be used by the <code>worker</code> pod, since the other ones run their services as <code>root</code>, but much like Debra from accounting, <code>root</code> doesn't give a fuck. Might as well.</p> <p>So if we try the build again with the correct permissions, what happens?</p> <p><img src="/images/buildbot-on-k8s/build-success-perms.png" alt="Buildbot with permissions fixed" /></p> <p>Huzah</p> <h2 id="autoscaling">Autoscaling</h2> <p>At last we have what I suspect will be the <em>hardest</em> part of the project: autoscaling. Here's what we want in a perfect world:</p> <ul> <li>When a lot of builds get requested, spin up more <code>worker</code> nodes.</li> <li>When fewer builds are requested, destroy those extra <code>worker</code> nodes.</li> </ul> <h3 id="asking-the-question">Asking the question</h3> <p>This task is actually a pretty tough cookie to crack. We want enough workers to each deal with 1/N requests in a queue of N builds, essentially scaling the building service to deal with usage spikes. It sounds straight forward enough, but how do we test it?</p> <h3 id="creating-a-test-case">Creating a test-case</h3> <p>For this we need to add a few projects to our instance of Buildbot, each of which will have a hefty workload. To do this we need to host a Buildbot <code>master.cfg</code> config file at some public location and refer to this in the <code>master.yaml</code> config file. I used the GitHub repo that accompanies this blogpost <sup class="footnote-reference"><a href="#accompanying-repo">5</a></sup>, but you could use GitHub Gists or a pastebin; just make sure you're referring to the <strong>raw</strong> file or a <code>.tar.gz</code> with the <code>master.cfg</code> file at the base of the tarball.</p> <p>So let's pull a project out of a hat that takes a while to build, how about... <a href="https://github.com/rust-lang/cargo">Cargo</a>. Any objections? Great, let's roll.</p> <p>To get started we'll need a custom <code>worker</code> container can build Cargo. A small adventure later and we have this Dockerfile which has all of the dependencies to build what we want:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>FROM buildbot/buildbot-worker:master </span><span> </span><span>RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain nightly </span><span>RUN echo &#39;PATH=$PATH:$HOME/.cargo/bin&#39; &gt;&gt; $HOME/.bashrc </span><span> </span><span>USER root </span><span>RUN apt update -y </span><span>RUN apt install -y cmake pkg-config </span><span> </span><span>USER buildbot </span></code></pre> <p>I set this to auto-build on <a href="https://quay.io/">quay.io</a>, hosted at the docker-pull url <code>quay.io/elijahcaine/buidlbot-rust-worker</code>.</p> <p>Now that we have a capable worker container, we need to create an arbitrary workload. To do that we'll use this Buildbot <code>master.cfg</code>:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span>diff --git a/blogpost/robust-master.cfg b/blogpost/robust-master.cfg </span><span>index 06ad65b..892188b 100644 </span><span style="color:#75715e;">--- a/blogpost/robust-master.cfg </span><span style="color:#75715e;">+++ b/blogpost/robust-master.cfg </span><span style="color:#75715e;">@@ -18,7 +18,7 @@ c = BuildmasterConfig = {} </span><span> # a Worker object, specifying a unique worker name and password. The same </span><span> # worker name and password must be configured on the worker. </span><span> </span><span style="color:#f92672;">-c[&#39;workers&#39;] = [worker.Worker(&quot;example-worker&quot;, &#39;pass&#39;)] </span><span style="color:#a6e22e;">+c[&#39;workers&#39;] = [worker.Worker(&quot;rust-worker&quot;, &#39;pass&#39;)] </span><span> </span><span> if &#39;BUILDBOT_MQ_URL&#39; in os.environ: </span><span> c[&#39;mq&#39;] = { </span><span style="color:#75715e;">@@ -43,7 +43,7 @@ c[&#39;protocols&#39;] = {&#39;pb&#39;: {&#39;port&#39;: os.environ.get(&quot;BUILDBOT_WORKER_PORT&quot;, 9989)}} </span><span> </span><span> c[&#39;change_source&#39;] = [] </span><span> c[&#39;change_source&#39;].append(changes.GitPoller( </span><span style="color:#f92672;">- &#39;git://github.com/buildbot/pyflakes.git&#39;, </span><span style="color:#a6e22e;">+ &#39;git://github.com/rust-lang/cargo.git&#39;, </span><span> workdir=&#39;gitpoller-workdir&#39;, branch=&#39;master&#39;, </span><span> pollinterval=300)) </span><span> </span><span style="color:#75715e;">@@ -57,10 +57,10 @@ c[&#39;schedulers&#39;].append(schedulers.SingleBranchScheduler( </span><span> name=&quot;all&quot;, </span><span> change_filter=util.ChangeFilter(branch=&#39;master&#39;), </span><span> treeStableTimer=None, </span><span style="color:#f92672;">- builderNames=[&quot;runtests&quot;])) </span><span style="color:#a6e22e;">+ builderNames=[&quot;cargo1-runtests&quot;, &quot;cargo2-runtests&quot;, &quot;cargo3-runtests&quot;])) </span><span> c[&#39;schedulers&#39;].append(schedulers.ForceScheduler( </span><span> name=&quot;force&quot;, </span><span style="color:#f92672;">- builderNames=[&quot;runtests&quot;])) </span><span style="color:#a6e22e;">+ builderNames=[&quot;cargo1-runtests&quot;, &quot;cargo2-runtests&quot;, &quot;cargo3-runtests&quot;])) </span><span> </span><span> ####### BUILDERS </span><span> </span><span style="color:#75715e;">@@ -68,17 +68,30 @@ c[&#39;schedulers&#39;].append(schedulers.ForceScheduler( </span><span> # what steps, and which workers can execute them. Note that any particular build will </span><span> # only take place on one worker. </span><span> </span><span style="color:#f92672;">-factory = util.BuildFactory() </span><span style="color:#f92672;">-# check out the source </span><span style="color:#f92672;">-factory.addStep(steps.Git(repourl=&#39;http://github.com/buildbot/pyflakes.git&#39;, mode=&#39;incremental&#39;)) </span><span style="color:#f92672;">-# run the tests (note that this will require that &#39;trial&#39; is installed) </span><span style="color:#f92672;">-factory.addStep(steps.ShellCommand(command=[&quot;trial&quot;, &quot;pyflakes&quot;])) </span><span style="color:#a6e22e;">+f = {} </span><span style="color:#a6e22e;">+f[&#39;cargo&#39;] = util.BuildFactory() </span><span style="color:#a6e22e;">+f[&#39;cargo&#39;].addStep(steps.Git(repourl=&#39;git://github.com/rust-lang/cargo.git&#39;, mode=&#39;full&#39;, method=&#39;fresh&#39;)) </span><span style="color:#a6e22e;">+f[&#39;cargo&#39;].addStep(steps.ShellCommand(command=[&quot;/home/buildbot/.cargo/bin/cargo&quot;, &quot;build&quot;, &quot;--release&quot;])) </span><span> </span><span> c[&#39;builders&#39;] = [] </span><span> c[&#39;builders&#39;].append( </span><span style="color:#f92672;">- util.BuilderConfig(name=&quot;runtests&quot;, </span><span style="color:#f92672;">- workernames=[&quot;example-worker&quot;], </span><span style="color:#f92672;">- factory=factory)) </span><span style="color:#a6e22e;">+ util.BuilderConfig(name=&quot;cargo1-runtests&quot;, </span><span style="color:#a6e22e;">+ workernames=[&quot;rust-worker&quot;], </span><span style="color:#a6e22e;">+ factory=f[&#39;cargo&#39;], </span><span style="color:#a6e22e;">+ builddir=&#39;builds/cargo1&#39;, </span><span style="color:#a6e22e;">+ workerbuilddir=&#39;builds/cargo1&#39;)) </span><span style="color:#a6e22e;">+c[&#39;builders&#39;].append( </span><span style="color:#a6e22e;">+ util.BuilderConfig(name=&quot;cargo2-runtests&quot;, </span><span style="color:#a6e22e;">+ workernames=[&quot;rust-worker&quot;], </span><span style="color:#a6e22e;">+ factory=f[&#39;cargo&#39;], </span><span style="color:#a6e22e;">+ builddir=&#39;builds/cargo2&#39;, </span><span style="color:#a6e22e;">+ workerbuilddir=&#39;builds/cargo2&#39;)) </span><span style="color:#a6e22e;">+c[&#39;builders&#39;].append( </span><span style="color:#a6e22e;">+ util.BuilderConfig(name=&quot;cargo3-runtests&quot;, </span><span style="color:#a6e22e;">+ workernames=[&quot;rust-worker&quot;], </span><span style="color:#a6e22e;">+ factory=f[&#39;cargo&#39;], </span><span style="color:#a6e22e;">+ builddir=&#39;builds/cargo3&#39;, </span><span style="color:#a6e22e;">+ workerbuilddir=&#39;builds/cargo3&#39;)) </span><span> </span><span> ####### STATUS TARGETS </span><span> </span><span style="color:#75715e;">@@ -93,8 +106,8 @@ c[&#39;status&#39;] = [] </span><span> # the &#39;title&#39; string will appear at the top of this buildbot installation&#39;s </span><span> # home pages (linked to the &#39;titleURL&#39;). </span><span> </span><span style="color:#f92672;">-c[&#39;title&#39;] = &quot;Pyflakes&quot; </span><span style="color:#f92672;">-c[&#39;titleURL&#39;] = &quot;https://launchpad.net/pyflakes&quot; </span><span style="color:#a6e22e;">+c[&#39;title&#39;] = &quot;Rusty Stuffy&quot; </span><span style="color:#a6e22e;">+c[&#39;titleURL&#39;] = &quot;https://www.rust-lang.org/en-US/&quot; </span><span> </span><span> # the &#39;buildbotURL&#39; string should point to the location where the buildbot&#39;s </span><span> # internal web server is visible. This typically uses the port number set in </span></code></pre> <p>And deploy it with this <code>master.yaml</code>:</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span>diff --git a/blogpost/robust-master.yaml b/blogpost/robust-master.yaml </span><span>index c7479e3..f75b4d6 100644 </span><span style="color:#75715e;">--- a/blogpost/robust-master.yaml </span><span style="color:#75715e;">+++ b/blogpost/robust-master.yaml </span><span style="color:#75715e;">@@ -50,7 +50,7 @@ spec: </span><span> - name: BUILDBOT_CONFIG_DIR </span><span> value: config </span><span> - name: BUILDBOT_CONFIG_URL </span><span style="color:#f92672;">- value: &#39;https://raw.githubusercontent.com/ElijahCaine/buildbot-on-kubernetes/master/simple/master.cfg&#39; </span><span style="color:#a6e22e;">+ value: &#39;https://raw.githubusercontent.com/ElijahCaine/buildbot-on-kubernetes/master/robust/master.cfg&#39; </span><span> - name: BUILDBOT_WORKER_PORT </span><span> value: &#39;9989&#39; </span><span> - name: BUILDBOT_WEB_URL </span></code></pre> <p>When we look at our setup in Buildbot it looks like this:</p> <p><img src="/images/buildbot-on-k8s/ample-workload.png" alt="Buildbot wih ample workload" /></p> <p>When each build is run it completely wipes away the old git clone and starts from scratch, which is <em>suuuper</em> wasteful, but exactly what we want here. The purpose of this test is to create a heavy workload for our nodes to perform, and with this each build will <em>definitely</em> be heavy.</p> <h3 id="kubectl-autoscale">kubectl autoscale</h3> <p>Now to start scaling that build. In a perfect world we'd get a request, that would start hogging our resources, and K8s would spin up a new worker instance to handle the next build in the queue. Based on the output of <code>kubectl --help</code> it looks like <code>kubectl autoscale deployment worker</code> is the command we want... so let's do that:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl autoscale deployment worker --max=5 --min=1 --cpu-percent=50 </span><span>deployment &quot;worker&quot; autoscaled </span><span>$ kubectl describe hpa </span><span>Name: worker </span><span>Namespace: default </span><span>Labels: &lt;none&gt; </span><span>Annotations: &lt;none&gt; </span><span>CreationTimestamp: Fri, 09 Dec 2016 11:13:15 -0800 </span><span>Reference: Deployment/worker </span><span>Target CPU utilization: 50% </span><span>Current CPU utilization: &lt;unset&gt; </span><span>Min replicas: 1 </span><span>Max replicas: 5 </span><span>Events: </span><span> FirstSeen LastSeen Count From SubobjectPath Type Reason Message </span><span> --------- -------- ----- ---- ------------- -------- ------ ------- </span><span> 37s 11s 6 {horizontal-pod-autoscaler } Warning FailedGetMetrics failed to get CPU consumption and request: failed to get pods metrics: the server could not find the requested resource (get services http:heapster:) </span><span> 37s 11s 6 {horizontal-pod-autoscaler } Warning FailedComputeReplicas failed to get CPU utilization: failed to get CPU consumption and request: failed to get pods metrics: the server could not find the requested resource (get services http:heapster:) </span></code></pre> <p>Hmm... that output looks like things aren't working. Let's look around the internet and see if we find anything useful. I'll go East, you go West, and we'll meet back here in 20.</p> <p>Great, what did you find? Me? Oh I found a bunch of Github issues <sup class="footnote-reference"><a href="#hpa-gh">6</a></sup> and Stack Overflow posts <sup class="footnote-reference"><a href="#hpa-so">7</a></sup>; lot of people mentioned this Heapster thing. The <a href="https://github.com/kubernetes/heapster">Heapster GitHub page</a> says it does <em>Compute Resource Usage Analysis and Monitoring of Container Clusters</em>, which sounds like what we want. I also found the <a href="http://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling/">K8s Autoscaling</a> docs that point to a <a href="http://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling/">K8s Autoscaling Tutorial</a>. That sounds super useful and confirms the suspicion that Heapster is useful. <strong>TLDR</strong> things are pointing toward setting up Heapster.</p> <p>As it turns out there is a <a href="https://github.com/kubernetes/minikube/blob/master/README.md#add-ons">Heapster addon for Minikube</a> <sup class="footnote-reference"><a href="#heapster-release">8</a></sup>, so let's get that setup.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ minikube addons enable heapster </span><span>heapster was successfully enabled </span><span>$ minikube addons open heapster </span><span>Opening kubernetes service kube-system/monitoring-grafana in default browser... </span></code></pre> <p><img src="/images/buildbot-on-k8s/hello-heapster.png" alt="Heapster working" /></p> <p>Neato!</p> <p>So now we've got Heapster running, what's next?</p> <p>Well the tutorial runs this line:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80 </span></code></pre> <p>Which we can appropriate to get some YAML to shove in our <code>worker.yaml</code> to get a good template for our Buildbot worker Deployment.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl run test --image=busybox --requests=cpu=200m -o yaml </span><span>apiVersion: extensions/v1beta1 </span><span>... </span><span>spec: </span><span> ... </span><span> template: </span><span> ... </span><span> spec: </span><span> containers: </span><span> - args: </span><span> ... </span><span> resources: </span><span> requests: </span><span> cpu: 200m </span><span>... </span></code></pre> <p>So I added that section to <code>worker.yaml</code>...</p> <pre data-lang="diff" style="background-color:#191919;color:#f8f8f2;" class="language-diff "><code class="language-diff" data-lang="diff"><span>diff --git a/blogpost/robust-worker.yaml b/blogpost/robust-worker.yaml </span><span>index e57db12..de52320 100644 </span><span style="color:#75715e;">--- a/blogpost/robust-worker.yaml </span><span style="color:#75715e;">+++ b/blogpost/robust-worker.yaml </span><span style="color:#75715e;">@@ -42,15 +42,18 @@ spec: </span><span> tier: worker </span><span> spec: </span><span> containers: </span><span style="color:#f92672;">- - image: &quot;buildbot/buildbot-worker:master&quot; </span><span style="color:#a6e22e;">+ - image: &quot;quay.io/elijahcaine/buildbot-rust-worker:master&quot; </span><span> name: worker </span><span style="color:#a6e22e;">+ resources: </span><span style="color:#a6e22e;">+ requests: </span><span style="color:#a6e22e;">+ cpu: 200m </span><span> env: </span><span> - name: BUILDMASTER </span><span> value: master </span><span> - name: BUILDMASTER_PORT </span><span> value: &#39;9989&#39; </span><span> - name: WORKERNAME </span><span style="color:#f92672;">- value: example-worker </span><span style="color:#a6e22e;">+ value: rust-worker </span><span> - name: WORKERPASS </span><span> value: pass </span><span> - name: WORKER_ENVIRONMENT_BLACKLIST </span></code></pre> <p>... and ran <code>kubectl replace -f worker.yaml</code>, but did it work? Well, if you're impatient like me you'll keep refreshing, it won't look like it's working, and then you'll pull your hair out. If that happens to you, just wait. Wait like... a minute. Just get some tea, stare out the window for a second, <em>then</em> see if it worked or not.</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>$ kubectl get hpa worker </span><span>NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE </span><span>worker Deployment/worker 50% 0% 1 5 2m </span></code></pre> <p>There. That looks good. Let's throw some work at it!</p> <p>A few builds:</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE </span><span>worker Deployment/worker 50% 536% 1 5 6m </span></code></pre> <p>Great! Let's check up on Buildbot:</p> <p><img src="/images/buildbot-on-k8s/replicas-didnt-work.png" alt="replication failing" /></p> <p>Ruh-roh that's not good...</p> <p>Let's dig in a bit.</p> <p><img src="/images/buildbot-on-k8s/failed-build.png" alt="replication failing -- zoom, enhance" /></p> <p>That's not good...</p> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>[... end of git pull ...] </span><span>75 security_updates_as_of=2016-10-07 </span><span>76 using PTY: False </span><span>77 Cloning into &#39;.&#39;... </span><span>78 </span><span>79 command interrupted, attempting to kill </span><span>80 process killed by signal 15 </span><span>81 program finished with exit code -1 </span><span>82 elapsedTime=2.051445 </span></code></pre> <pre style="background-color:#191919;color:#f8f8f2;"><code><span>[... entirety of &#39;cancelled&#39; logs ...] </span><span>0 no reason </span></code></pre> <p>Well, it looks like that plan didn't work.</p> <p>I did a bit more digging and I can say with certainty that the approach I took to scaling the workload did <em>not</em> work. I can't say <em>why</em> for sure, and I definitely can't say how to fix it, but honestly... it's late. I'm tired and right now I'm okay with throwing in the towel for now. Maybe we'll solve this in a <strong>Part 2</strong>.</p> <h2 id="debrief">Debrief</h2> <blockquote> <p>First: This blogpost has a repo associated with it: <a href="https://github.com/elijahcaine/buildbot-on-kubernetes">https://github.com/elijahcaine/buildbot-on-kubernetes</a>.</p> <p>Please check that out, see what it has to offer, and make issues/pull requests. Right now it's just a copy of the configs used in this project and a little bit of documentation.</p> <p>Happy hacking!</p> </blockquote> <p>I'm terrible at endings so I'm just gonna braindump some stuff here and let you, the beautiful and charming reader, find your own closure from all of this.</p> <h3 id="failure-its-what-s-on-the-menu">Failure: its what's on the menu</h3> <p>That's right, I didn't do what I set out to achieve. Is that a bad thing? <strong>Of course not!</strong> To quote the great Jake the Dog:</p> <p><img src="/images/buildbot-on-k8s/sucking-quote.jpg" alt="Sucking at something is the first step to being kinda good at something" /></p> <p>Sure we didn't get the <em>perfect</em> BuildBot+K8s but we learned a ton and got <em>really</em> close!</p> <h3 id="some-stuff-we-learned">Some stuff we learned</h3> <p>So what are some things we learned that we should probably write down? This is a pretty open question and if you followed along you'll <em>probably</em> get a drastically different list than I have. Here's the big things I feel like I'm walking away with:</p> <ul> <li><code>kubectl &lt;command&gt; --dry-run -o yaml</code> is very useful for file-izing your infrastructure.</li> <li>K8s is very well documented <strong>if</strong> you are a very patient individual.</li> <li>Storage in K8s is still not a perfectly solved problem [^storage].</li> <li>K8s applications share private networking.</li> <li>K8s Pods are really just containers.</li> </ul> <p>I'm sure there's a ton more I've internalized and can't recall. I tried to take thorough notes during this entire experience to capture the failures as well as the successes, which is a harder skill than I expected.</p> <h3 id="useful-resources">Useful resources</h3> <p>Beyond searching for specific answers these are <em>honestly</em> the websites I kept going back to throughout this project.</p> <ul> <li><a href="http://kubernetes.io/docs/">http://kubernetes.io/docs/</a> The main K8s docs. A bit dense and sparse in key areas, a problem I'm actively I'm working on. Includes API docs and user-guides which are useful for those patient enough to read them.</li> <li><a href="http://k8s.info/">http://k8s.info/</a> A community-run resource with some useful links and a 'cheat sheet'.</li> <li><a href="https://github.com/kubernetes/community">https://github.com/kubernetes/community</a> Includes a bunch of very useful design documentation. I'm not usually the kind of person that enjoys reading design docs, but sometimes a feature is too new to have a usable user-guide. When it's the best you've got, take it.</li> </ul> <p>As far as Buildbot-specific stuff I honestly just used the <a href="https://docs.buildbot.net/current/">official Buildbot docs</a> which were more than enough.</p> <h3 id="whats-next">Whats next?</h3> <p>I'm sure I'll be posting more about K8s going forward. It is a very useful tool because of/despite it's complexities. As I use it I'll post more about it.</p> <p>If you have any feedback on this post feel free to get in contact with me <a href="https://twitter.com/pastywhitenoise">@PastyWhiteNoise</a> on Twitter, <code>pop</code> on <a href="https://webchat.freenode.net/">irc.freenode.net</a> <sup class="footnote-reference"><a href="#irc">9</a></sup>, you can make an issue on <a href="https://github.com/elijahcaine/elijahcaine.github.io">this website's Github Repository</a>, and of course I can be reached by Carrier Pigeon..</p> <h2 id="errata">Errata</h2> <div class="footnote-definition" id="GIFEE"><sup class="footnote-definition-label">10</sup> <p>Google Infrastructure for Everyone Else</p> </div> <div class="footnote-definition" id="accompanying-repo"><sup class="footnote-definition-label">5</sup> <p><a href="https://github.com/ElijahCaine/buildbot-on-kubernetes">https://github.com/ElijahCaine/buildbot-on-kubernetes</a></p> </div> <div class="footnote-definition" id="docs-should"><sup class="footnote-definition-label">1</sup> <p>My job, at least a big part of it, is writing documentation. Docs are a notable part of my identify, and I have a belief in the power of good documentation, but at the end of the day they <strong>are</strong> just a means to an end.</p> <p>In a perfect world docs wouldn't <em>need</em> to exist, so all docs should be enough to get whoever's readin them to the point where they can do cool shit on their own. Just like a great cinematographer is so good you don't notice the camera in a movie, great docs should be so good you don't <strong>notice</strong> how good they are.</p> <p>So that's why I like <strong>doing</strong> cool shit over <strong>reading</strong> about it.</p> <p>It's also fun to play with toys! Even if those toys are software.</p> </div> <div class="footnote-definition" id="heapster-release"><sup class="footnote-definition-label">8</sup> <p>Authors Note: The heapster addon was added to Minikube <strong>during the writing of this post</strong>. How convenient!</p> </div> <div class="footnote-definition" id="hpa-gh"><sup class="footnote-definition-label">6</sup> <p><a href="https://github.com/openshift/origin/issues/6239">https://github.com/openshift/origin/issues/6239</a> - <a href="https://github.com/kubernetes/kubernetes/issues/18652">https://github.com/kubernetes/kubernetes/issues/18652</a></p> </div> <div class="footnote-definition" id="hpa-so"><sup class="footnote-definition-label">7</sup> <p><a href="http://stackoverflow.com/questions/38874145/autoscaling-hpa-failed-to-get-cpu-consumption-cannot-unmarshal-object-into-go">http://stackoverflow.com/questions/38874145/autoscaling-hpa-failed-to-get-cpu-consumption-cannot-unmarshal-object-into-go</a> - <a href="http://stackoverflow.com/questions/37631008/kubernetes-hpa-cannot-get-cpu-consumption">http://stackoverflow.com/questions/37631008/kubernetes-hpa-cannot-get-cpu-consumption</a></p> </div> <div class="footnote-definition" id="irc"><sup class="footnote-definition-label">9</sup> <p>I usually hang out in the <code>#osu-lug</code> channel. They're cool people, you should hang out with us!</p> </div> <div class="footnote-definition" id="same-app-note"><sup class="footnote-definition-label">4</sup> <p>It wasn't painfully obvious to me so it's worth elaborating that for K8s services to communicate with one-another they need to be part of the same <code>app</code>. In the example the <code>postgres</code>, <code>worker</code>, and <code>master</code> containers are all part of the same <code>buildbot</code> app.</p> <p>It's not a terribly well documented <strong>key</strong> piece of information so it seemed worth mentioning.</p> </div> <div class="footnote-definition" id="storge"><sup class="footnote-definition-label">11</sup> <p>But neither is storage in general so who can blame 'em?</p> </div> <div class="footnote-definition" id="theory-vs-practice"><sup class="footnote-definition-label">2</sup> <p>If you read that and thought "You know what they say about theory versus practice!" then <em>yes</em> -- but you're getting ahead of me.</p> </div> <div class="footnote-definition" id="why"><sup class="footnote-definition-label">3</sup> <p>Might as well clarify <em>why</em> we want persistent storage and auto-scaling since these are usually desired features in an orchestrator but nobody explains <em>why</em> they're desireable.</p> <p>Autoscaling: Autoscaling is important is because we've got a whole data-center (in theory) and we want to maximize how we use that data-center's computational abilities. In a perfect world we would use exactly as much of the data-center as is demanded by users, but in practice this is super hard!</p> <p>Say your website gets linked on Hacker News, you get tons of requests pouring in, but your poor little 2004 server falls on it's face when you <em>look</em> at it wrong, and <strong>that</strong> is where the exact page was hosted. If you don't <strong>know</strong> that server fell down you'll never fix the problem or even <strong>know</strong> about the problem.</p> <p>With an orchestrator we can (in theory) specify how much we want to utilize a our data-center's resources and under what conditions we want to use more or less resources. If our service is creating a light load, only have one container up; if it's got a metric fuck-ton then spin up 20 containers across our entire cluster. You say what you want and the load-balancer + scheduler + monitor take care of the rest.</p> <p>Persistent Storage: The reason persistent storage is a useful feature is for two reasons</p> <p>The first is that containers are ephemeral. They can be created, destroyed, and upgraded <em>all</em> the time, basically making them the polar opposite of a 'Special Snowflake' service.</p> <p>This is nice, but what about state? As nice as it'd be to make an entirely stateless data-center, cat pictures and builds aren't going to store themselves!</p> <p>The solution to this problem is -- well it's not solved, but there's a clear direction things are going in. The TLDR is that we create some block device in the cloud and mount it to each identical container at the same mount-point, so each container (K8s 'Pod') shares some amount of state. Sorta like... I'm bad at examples actually. I'll think of one eventually.</p> <p>Piggy-backing off of this, not only do we preserve state between container upgrades/deletes/additions, but we can also share state between scaled services (i.e., a set of identical containers across a datacenter). Essentially we can have one worker pod carry out a build and another pod perform some action on that build (uploading it, verify it, archive it, etc). When the first worker pod gets deleted the build lives on in the shared state.</p> <p>This allows us to treat our state the same way we treat computing resources in the world of GIFFEE</p> </div> <div class="footnote-definition" id="GIFEE"><sup class="footnote-definition-label">10</sup> <p>Create some block device for the service, mount it to this point, and let your orchestrator take care of the rest.</p> </div> 2016-08-21T00:00:00+00:00 2016-08-21T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/programming-hurdles/ <p>I've been actively learning programming for almost five years now. I started by taking a C++ class at Portland Community College. That class was the best thing to happen to my engineering career because it taught me that <em>programming is very hard, and I am not naturally good at it</em>, but it was also very rewarding so I stuck with it.</p> <p>I'm not special either, most of us aren't very good at programming because it's so abstract and hard to grasp, like Math and Infinite Jest. There are of course the people that seem like <em>programming gods</em>, but nine times out of ten those people have been programming since they were eight and it's really not fair to compare yourself to them. That's like comparing your sporting abilities to <em>any olympic athlete</em>. Don't be so hard on yourself.</p> <p>In learning to program, and now teaching others how to program, I've identified a few <strong>hurdles</strong> (mountains, cliffs, obstacles, w/e) that folks tend to hit and have trouble crossing. They always get past these hurdles eventually (or they quit CS ☹), but not without a bit of unnecessary struggle. Struggling builds character, and I don't want to cheat anybody out of that, but I do want to help those that want it.</p> <p>This post will in no way guarantee your success in overcoming these obstacles but it should at least give you a head start to help you understand what you're learning and point you in the right direction when you want to ask a question.</p> <h2 id="data-structures">Data Structures</h2> <p>Understanding that <strong>a lot of programming is basically manipulating data to fit a need</strong> is pretty important. A social network can be thought of as a bunch of people's personal information plugged into a bunch of algorithms to make connecting easier. A word processor is really just a front-end for some <a href="https://en.wikipedia.org/wiki/Microsoft_Office_XML_formats">XML under the hood</a>, which is itself <a href="https://en.wikipedia.org/wiki/XML">just a way to represent structured data</a>.</p> <p>When handling data try to keep in mind what you are trying to accomplish and the best way to structure the data in the pursuit of that goal.</p> <span class="define"> <span class="define-word"> &lt;a href=&quot;https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Array_data_structure&quot;&gt;Arrays&#x2F;Lists&#x2F;Vectors&lt;&#x2F;a&gt; </p> <span class="define-body"> &lt;p&gt;&lt;strong&gt;Arrays and Lists are an ordered data type&lt;&#x2F;strong&gt;. Use them when you need to keep track of the order things happened in, like queuing an event. Try to keep them &lt;em&gt;relatively small&lt;&#x2F;em&gt; since they are fast for retrieval but not inherently space efficient.&lt;&#x2F;p&gt; </span> </span> <p>{% define(word="<a href="https://en.wikipedia.org/wiki/Hash_table">Hash-table/Key-Value/Dictionary</a>" %} <strong>Hashes are useful for storing unordered data</strong> with keys, like an address book or a small database. We use them to access data quickly and easily since data is retrieved using a key and always takes approximately the same amount of time. When you get data out of a dictionary you provide a key and get back the associated data, just like (you guessed it) <em>a real dictionary</em> where provided the word you get back a definition.</p> <p>Hash-tables are useful tools but should only be used for <em>relatively small</em> amounts of data. There is a bit of overhead in creating the table for storing your data so if your data-set gets too big you'll run out of memory. {% end %}</p> <span class="define"> <span class="define-word"> &lt;a href=&quot;https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Linked_list&quot;&gt;Linked Lists&#x2F;Sorted Trees&lt;&#x2F;a&gt; </p> <span class="define-body"> &lt;p&gt;Linked Lists and Sorted Trees aren&#x27;t exactly the same thing but I&#x27;m lumping them together because they both deal with a lot of the same concepts and can be implemented in similar ways. In learning about LL&#x2F;ST you&#x27;ll probably deal with Structs, Nodes, Pointers, and dynamic memory allocation, which if you&#x27;re like me will totally break your brain and then become second-nature.&lt;&#x2F;p&gt; &lt;p&gt;&lt;strong&gt;One advantage to these structures is that they can be implemented to take up only the space they require&lt;&#x2F;strong&gt;. Where Hash-maps and lists are hard to make the &lt;em&gt;right size&lt;&#x2F;em&gt;, LL&#x2F;ST can only take up exactly as much room as they need. The trade-off (a word you hear a lot when dealing choosing data structures) is that they&#x27;re not always the &lt;em&gt;fastest&lt;&#x2F;em&gt; way to store&#x2F;retrieve data. Just consider your use-case and think about what you need for the task at hand.&lt;&#x2F;p&gt; &lt;p&gt;If you&#x27;re interested in learning Rust though there&#x27;s a great guide on &lt;a href=&quot;http:&#x2F;&#x2F;cglab.ca&#x2F;~abeinges&#x2F;blah&#x2F;too-many-lists&#x2F;book&#x2F;README.html&quot;&gt;Learning Rust With Entirely Too Many Linked Lists&lt;&#x2F;a&gt;. I do suggest it.&lt;&#x2F;p&gt; </span> </span> <p><strong>These are all just the structures you're giving your data</strong> (a series of ones and zeros on disk and in memory). Anybody that's done work with a sufficiently complicated project (e.g., the Linux Kernel) can tell you that one glob of data can be treated as a Linked List, Array, Hash -- or all three at the same time! The structure you give your data is just so <em>you</em> can work with it, the computer doesn't really care one way or another, so choose what makes the most sense.</p> <p><em>Speaking of data structures...</em></p> <h2 id="objects-classes">Objects/Classes</h2> <p><strong>Objects are logical groupings of data (variables) and functions that act on that data</strong>. They differ from structs in that they are (usually) private by default, meaning that variables declared in an object are not accessible from the <em>outside</em> unless you explicitly say so. Variables are usually manipulated via a method called a <em>getter</em> and <em>setter</em>.</p> <p>Well... what I've actually described so far is creating a <strong>class: aka a blueprint for an object</strong>. This python is a pretty succinct way to describe classes and objects:</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#6d6d6d;">&quot;&quot;&quot; </span><span style="color:#6d6d6d;">objects_example.py </span><span style="color:#6d6d6d;"> </span><span style="color:#6d6d6d;">Declare a class named BazClass. </span><span style="color:#6d6d6d;">&quot;&quot;&quot; </span><span style="font-style:italic;color:#ff5e5e;">class </span><span style="text-decoration:underline;color:#8cdaff;">BazClass</span><span>(</span><span style="text-decoration:underline;font-style:italic;color:#8cdaff;">object</span><span>): </span><span> </span><span style="font-style:italic;color:#ff5e5e;">def </span><span style="color:#6699cc;">__init__</span><span>(</span><span style="font-style:italic;color:#fc9354;">self</span><span>, </span><span style="font-style:italic;color:#fc9354;">x</span><span style="color:#ff5e5e;">=</span><span style="color:#ff8942;">None</span><span>): </span><span> </span><span style="color:#6d6d6d;">&quot;&quot;&quot; </span><span style="color:#6d6d6d;"> The __init__ function sets class variables and sets up the object. </span><span style="color:#6d6d6d;"> &quot;&quot;&quot; </span><span> </span><span style="color:#e9fdac;">self</span><span>.var1 </span><span style="color:#ff5e5e;">= </span><span>x </span><span> </span><span style="color:#e9fdac;">self</span><span>.var2 </span><span style="color:#ff5e5e;">= </span><span style="color:#fdb082;">6 </span><span> </span><span> </span><span style="font-style:italic;color:#ff5e5e;">def </span><span style="color:#8cdaff;">a</span><span>(</span><span style="font-style:italic;color:#fc9354;">self</span><span>): </span><span> </span><span style="color:#6d6d6d;">&quot;&quot;&quot; </span><span style="color:#6d6d6d;"> a() acts on the object variables (accessed via `self`). </span><span style="color:#6d6d6d;"> &quot;&quot;&quot; </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">Variable you set </span><span style="color:#fdb082;">{}</span><span style="color:#ffffff;">&quot;</span><span>.</span><span style="color:#e9fdac;">format</span><span>(</span><span style="color:#e9fdac;">self</span><span>.var1)) </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">Variable set by class definition </span><span style="color:#fdb082;">{}</span><span style="color:#ffffff;">&quot;</span><span>.</span><span style="color:#e9fdac;">format</span><span>(</span><span style="color:#e9fdac;">self</span><span>.var2)) </span><span> </span><span style="color:#6d6d6d;">&quot;&quot;&quot; </span><span style="color:#6d6d6d;">Creating an instance of the class and call the `a()` function. </span><span style="color:#6d6d6d;">&quot;&quot;&quot; </span><span>foo_object </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">BazClass</span><span>(</span><span style="color:#fdb082;">7</span><span>) </span><span>foo_object.</span><span style="color:#e9fdac;">a</span><span>() </span></code></pre> <p>Objects are something that will <em>click</em> after you use the for a while. There's some nuances and implementation quirks depending on the language you're using, but in the end they're just logical collections of private data and functions.</p> <h2 id="functional-programming">Functional Programming</h2> <p><a href="https://en.wikipedia.org/wiki/Functional_programming">Functional Programming</a> (FP) was my first big paradigm shift in CS since functions. FP includes concepts of variable immutability (once it's set it's set), callbacks (functions calling functions and propagating results up), program state -- the list goes on. Here's an example to get us started.</p> <p>Non-functional paradigm:</p> <pre data-lang="javascript" style="background-color:#191919;color:#f8f8f2;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span style="font-style:italic;color:#fbdfb5;">function </span><span style="color:#8cdaff;">average_evens</span><span style="color:#ffffff;">(</span><span style="font-style:italic;color:#fc9354;">x</span><span style="color:#ffffff;">) </span><span>{ </span><span> </span><span style="color:#e9fdac;">avg </span><span style="color:#ff5e5e;">= </span><span style="color:#fdb082;">0</span><span>; </span><span> </span><span> </span><span style="color:#ff5e5e;">for </span><span>(</span><span style="color:#e9fdac;">i </span><span style="color:#ff5e5e;">= </span><span style="color:#fdb082;">0</span><span>; </span><span style="color:#e9fdac;">i </span><span style="color:#ff5e5e;">&lt; </span><span style="color:#e9fdac;">x</span><span>.length; </span><span style="color:#e9fdac;">i</span><span style="color:#ff5e5e;">++</span><span>) { </span><span> </span><span style="color:#ff5e5e;">if </span><span>(</span><span style="color:#e9fdac;">x</span><span>[</span><span style="color:#e9fdac;">i</span><span>] </span><span style="color:#ff5e5e;">% </span><span style="color:#fdb082;">2 </span><span style="color:#ff5e5e;">== </span><span style="color:#fdb082;">0</span><span>) { </span><span> </span><span style="color:#e9fdac;">avg </span><span style="color:#ff5e5e;">+= </span><span style="color:#e9fdac;">x</span><span>[</span><span style="color:#e9fdac;">i</span><span>]; </span><span> } </span><span> } </span><span> </span><span> </span><span style="color:#e9fdac;">avg </span><span style="color:#ff5e5e;">= </span><span style="color:#e9fdac;">avg</span><span style="color:#ff5e5e;">/</span><span style="color:#e9fdac;">x</span><span>.length; </span><span> </span><span style="color:#ff5e5e;">return </span><span style="color:#e9fdac;">avg</span><span>; </span><span>} </span></code></pre> <p>Functional Paradigm:</p> <pre data-lang="javascript" style="background-color:#191919;color:#f8f8f2;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span style="font-style:italic;color:#fbdfb5;">function </span><span style="color:#8cdaff;">average_evens</span><span style="color:#ffffff;">(</span><span style="font-style:italic;color:#fc9354;">x</span><span style="color:#ffffff;">) </span><span>{ </span><span> </span><span style="color:#ff5e5e;">return </span><span style="color:#e9fdac;">x</span><span>.</span><span style="color:#8cdaff;">reduce</span><span>(</span><span style="font-style:italic;color:#fbdfb5;">function</span><span style="color:#ffffff;">(</span><span style="font-style:italic;color:#fc9354;">x</span><span style="color:#ffffff;">) </span><span>{ </span><span> </span><span style="color:#ff5e5e;">if </span><span>(</span><span style="color:#e9fdac;">x </span><span style="color:#ff5e5e;">% </span><span style="color:#fdb082;">2 </span><span style="color:#ff5e5e;">== </span><span style="color:#fdb082;">0</span><span>) { </span><span> </span><span style="color:#ff5e5e;">return </span><span style="color:#e9fdac;">x</span><span>; </span><span> } </span><span> }) </span><span style="color:#ff5e5e;">/ </span><span style="color:#e9fdac;">x</span><span>.length; </span><span>} </span></code></pre> <p>In the non-functional example one would create a variable, iterate with a for-loop, and return a variable at the end. In the functional example you call functions which return data that you handle (callback) and that result is added to or replaces the original data. I'm honestly not doing this topic justice but it's one of those things you either learn first or you learn the hard way.</p> <p>This is just the tip of the ice-berg; many people prefer functional programming. If you are even remotely interested you should find one of those people and let them talk your ear off.</p> <p>If you're learning functional programming I suggest either <a href="http://learnyouahaskell.com/chapters">Learn You a Haskell for Great Good</a> if you're interested in learning Haskell (which is interesting an interesting language if nothing else) or <a href="http://www.oreilly.com/programming/free/functional-programming-python.csp">Functional Programming in Python</a> for <em>pythonistas</em>.</p> <h2 id="frameworks">Frameworks</h2> <p><strong>Frameworks are a collection of libraries, methods, and tools to accomplish a specific type of task</strong>. If you want to get something <em>complicated</em> done <em>fast</em>, you'll use a framework. Take for instance a web-app like Facebook or Twitter: it needs to be able to <em>send/receive HTTP requests</em>, <em>interface with a database</em> / <em>craft database queries</em>, <em>render web-pages</em>, <em>and</em> whatever the actual website is supposed to do.</p> <p>Frameworks can be very small or very large but they are always a big hurdle for those of us that have never worked with one before. A good starter framework I suggest is <a href="http://flask.pocoo.org/">Flask</a> for Python-based web-apps.</p> <h2 id="testing">Testing</h2> <p><strong>Testing is writing a program to test your program</strong>. There are many different <em>kinds</em> of tests from <em>unit tests</em> (checking single functions), to <em>integration tests</em> (checking that your functions work <em>together</em>), to <em>random tests</em> (trying to break your program by telling a computer to break your functions by using they in weird ways).</p> <p>Remember that first program you wrote for class? You wrote one that prompted the user for data and then manipulated that input in some way? Remember how you tested that? Probably in the most tedious way imaginable: <strong>by hand</strong>. Tests are much easier to write than your actual code and while they're a tedious investment up-front it's a small commitment relative to the <em>hours</em> you'd spend checking your program every time you made a change.</p> <h2 id="troubleshooting">Troubleshooting</h2> <p>The last thing isn't really about programming but about <em>fixing</em> your programming. When you're learning a new paradigm, language, or framework you're going to spend a lot of time <em>fixing</em> what doesn't currently work much more than you're going to actually be producing working code. This skill can also be summed up as 'How to Read Error Messages and Google well'.</p> <p>My troubleshooting advice is to <em>read</em> the error message, don't give up, and search for anything that looks meaningful. Once you find an answer try to <em>grok</em> what the answer means so you can learn <em>why</em> that worked instead of knowing <em>if I type this in a certain way it will not fall down</em>. Future you will appreciate the investment you put into <em>understanding</em> the problem and it's solution.</p> <p>Also don't be afraid to experiment. Make a copy of your program (or use <a href="https://git-scm.com/">git</a>) and see if some crazy idea is exactly what you need.</p> <hr /> <p>This list is far from complete but it still felt worth sharing. If you think I missed something, <a href="https://elijah.run/about/#contact">contact me</a> and I might do a follow-up post.</p> 2016-07-27T00:00:00+00:00 2016-07-27T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/git-does-a-lot-of-things/ <p>I should make a tool that extends <code>git</code> to delete all of the files matched by the <code>.gitginore</code>. I spend way too much time crafting <code>find &lt;...&gt; | xargs rm</code> commands. I know, I'll call it <code>git clean</code>.</p> <p>Well... before I get too far I'll just make sure it doesn't already exist.</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>$ man git clean </span><span> </span><span>GIT-CLEAN(1) Git Manual GIT-CLEAN(1) </span><span> </span><span>NAME </span><span> git-clean - Remove untracked files from the working tree </span><span> </span><span>SYNOPSIS </span><span> git clean [-d] [-f] [-i] [-n] [-q] [-e &lt;pattern&gt;] [-x | -X] [--] </span><span> &lt;path&gt;... </span><span> </span><span>DESCRIPTION </span><span> Cleans the working tree by recursively removing files that are not </span><span> under version control, starting from the current directory. </span><span> </span><span> ... </span></code></pre> <p>Huh... well I... that's pretty much exactly how I would have done it...</p> <p><code>git</code> <em>tab tab</em></p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>add filter-branch relink </span><span>am format-patch remote </span><span>annotate fsck repack </span><span>apply gc replace </span><span>archive get-tar-commit-id request-pull </span><span>bisect grep reset </span><span>blame help revert </span><span>branch imap-send review </span><span>bundle init rm </span><span>checkout instaweb send-email </span><span>cherry interpret-trailers shortlog </span><span>cherry-pick log show </span><span>clean merge show-branch </span><span>clone mergetool stage </span><span>column mv stash </span><span>commit name-rev status </span><span>config notes submodule </span><span>credential p4 svn </span><span>describe pull tag </span><span>diff push verify-commit </span><span>difftool rebase whatchanged </span><span>fetch reflog worktree </span></code></pre> <p>Ah, okay. I get it now. Git <em>does</em> do a lot of things.</p> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>The above output is from <code>git 2.8.2</code></p> </span> </span> 2016-07-27T00:00:00+00:00 2016-07-27T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/induction/ <p><img src="/images/comics/induction.png" alt="Classic mathematics memes" /></p> 2016-04-03T00:00:00+00:00 2016-04-03T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/python-exceptions/ <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span>var </span><span style="color:#ff5e5e;">= </span><span style="color:#6699cc;">input</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">Please enter a number not equal to 0: </span><span style="color:#ffffff;">&#39;</span><span>) </span><span style="color:#ff5e5e;">try</span><span>: </span><span> var </span><span style="color:#ff5e5e;">= </span><span style="color:#fdb082;">1</span><span style="color:#ff5e5e;">/</span><span style="color:#fbe3bf;">float</span><span>(var) </span><span style="color:#ff5e5e;">except </span><span style="color:#fbe3bf;">ZeroDivisionError </span><span style="color:#ff5e5e;">as </span><span>e: </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">Error: </span><span style="color:#ffffff;">&#39; </span><span style="color:#ff5e5e;">+ </span><span style="color:#fbe3bf;">str</span><span>(e)) </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">You had one job!</span><span style="color:#ffffff;">&#39;</span><span>) </span></code></pre> <div class="warning"> <span class="warning-title"> Warning </span> <span class="warning-body"> <p>This post does not cover <em>what</em> python exceptions are. If the above code doesn't make sense you should check out this <a href="https://wiki.python.org/moin/HandlingExceptions">Introduction to Python Exceptions</a> from wiki.python.org before reading this post.</p> </span> </div> <p>If you've fallen in love with Python you've no doubt discovered exceptions:</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#ff5e5e;">try</span><span>: </span><span> </span><span style="color:#6d6d6d;"># do a thing </span><span style="color:#ff5e5e;">except</span><span>: </span><span> </span><span style="color:#6d6d6d;"># thing did not work, </span><span> </span><span style="color:#6d6d6d;"># do something else </span></code></pre> <p>They are endlessly useful and help one to avoid writing checks upon checks upon checks before getting to the meat of your project. Just try a thing, catch the error, keep on rolling (or fail gracefully).</p> <p>As powerful as they can be, I have found a lot of folks (past me included) who don't know how to find python exceptions and don't know how to write their own exceptions! So let's do that.</p> <h2 id="finding-exceptions">Finding Exceptions</h2> <p>When writing exception handling code it's kosher to explicitly state <em>which</em> error you expect.</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#6d6d6d;"># Bad: </span><span style="color:#ff5e5e;">try</span><span>: </span><span> </span><span style="color:#6d6d6d;"># A thing that might not work </span><span style="color:#ff5e5e;">except</span><span>: </span><span> </span><span style="color:#6d6d6d;"># Something else </span><span> </span><span style="color:#6d6d6d;"># Good: </span><span style="color:#ff5e5e;">try</span><span>: </span><span> </span><span style="color:#6d6d6d;"># A thing that might not work </span><span style="color:#ff5e5e;">except </span><span>SpecificError </span><span style="color:#ff5e5e;">as </span><span>err: </span><span> </span><span style="color:#6d6d6d;"># Maybe print(err) </span><span> </span><span style="color:#6d6d6d;"># Something else </span></code></pre> <p>When I first found this out I thought <em>Golly that does sound useful; I always try to be explicit in my error handling -- but how??</em> Thankfully future me is here to answer questions like this.</p> <p>The exception you are looking for (for instance, <code>SpecificError</code> in the above pseudo-code) can be found in the python traceback:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>$ echo &quot;open(&#39;myfakefile.txt&#39;, &#39;r&#39;).close()&quot; &gt; my-unhandled-script.py </span><span>$ python my-unhandled-script.py </span><span>Traceback (most recent call last): </span><span> File &quot;/tmp/test.py&quot;, line 1, in &lt;module&gt; </span><span> open(&#39;myfakefile.txt&#39;, &#39;r&#39;).close() </span><span>IOError: [Errno 2] No such file or directory: &#39;myfakefile.txt&#39; </span></code></pre> <p>That bit on the last line <code>IOError</code> is the exception you're looking for. So when you write your code you'll say something like the following:</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#ff5e5e;">try</span><span>: </span><span> </span><span style="color:#6699cc;">open</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">myfakefile.txt</span><span style="color:#ffffff;">&#39;</span><span>, </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">r</span><span style="color:#ffffff;">&#39;</span><span>).</span><span style="color:#e9fdac;">close</span><span>() </span><span style="color:#ff5e5e;">except </span><span style="color:#fbe3bf;">IOError </span><span style="color:#ff5e5e;">as </span><span>e: </span><span> </span><span style="color:#6699cc;">print</span><span>(e) </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">File `myfakefile.txt` does not exist</span><span style="color:#ffffff;">&#39;</span><span>) </span></code></pre> <p>To recap, here is one way (and my preferred method for) 'doing' python exception handling:</p> <ol> <li>Write breakable code.</li> <li>Run breakable code, see what exceptions python spits out.</li> <li>Wrap breakable code in explicit <code>try/except</code> blocks.</li> <li>???</li> <li>Profit.</li> </ol> <h2 id="writing-your-own-exceptions">Writing Your Own Exceptions</h2> <p>You (who me?), yes <em>you</em> can write custom python exceptions. It's dead easy too.</p> <p>The long and short of it is you define an exception class which either inherits from the <code>Exception</code> class or another pre-existing exception.</p> <pre data-lang="python" style="background-color:#191919;color:#f8f8f2;" class="language-python "><code class="language-python" data-lang="python"><span style="font-style:italic;color:#ff5e5e;">class </span><span style="text-decoration:underline;color:#8cdaff;">CusssstomError</span><span>(</span><span style="text-decoration:underline;font-style:italic;color:#8cdaff;">Exception</span><span>): </span><span> </span><span style="color:#6d6d6d;">&#39;&#39;&#39;Raise when snakes&#39;&#39;&#39; </span><span> </span><span style="font-style:italic;color:#ff5e5e;">def </span><span style="color:#6699cc;">__init__</span><span>(</span><span style="font-style:italic;color:#fc9354;">self</span><span>, </span><span style="font-style:italic;color:#fc9354;">message</span><span>): </span><span> </span><span style="color:#e9fdac;">self</span><span>.message </span><span style="color:#ff5e5e;">= </span><span>message </span><span> </span><span style="font-style:italic;color:#ff5e5e;">def </span><span style="color:#8cdaff;">check_for_snakes</span><span>(</span><span style="font-style:italic;color:#fc9354;">foo</span><span>): </span><span> </span><span style="color:#ff5e5e;">if </span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">snake</span><span style="color:#ffffff;">&#39; </span><span style="color:#ff5e5e;">in </span><span>foo.</span><span style="color:#e9fdac;">lower</span><span>(): </span><span> </span><span style="color:#ff5e5e;">raise </span><span style="color:#e9fdac;">CusssstomError</span><span>(</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">Snakes! I hate snakes!</span><span style="color:#ffffff;">&#39;</span><span>) </span><span> </span><span style="color:#ff5e5e;">try</span><span>: </span><span> foo </span><span style="color:#ff5e5e;">= </span><span style="color:#6699cc;">input</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">Just don&#39;t mention snakes: </span><span style="color:#ffffff;">&quot;</span><span>) </span><span> </span><span style="color:#e9fdac;">check_for_snakes</span><span>(foo) </span><span style="color:#ff5e5e;">except </span><span>CusssstomError </span><span style="color:#ff5e5e;">as </span><span>e: </span><span> </span><span style="color:#6699cc;">print</span><span>(e.message) </span><span style="color:#ff5e5e;">else</span><span>: </span><span> </span><span style="color:#6699cc;">print</span><span>(</span><span style="color:#ffffff;">&quot;</span><span style="color:#fbe3bf;">Thanks. I appreciate it.</span><span style="color:#ffffff;">&quot;</span><span>) </span></code></pre> <p>The above code defines the <code>CusssstomError</code> exception class which you can <code>raise</code>. Very neat and <a href="https://docs.python.org/2/glossary.html#term-pythonic">pythonic</a></p> <h2 id="further-reading">Further Reading</h2> <p>Here are a few references I suggest you check out to get an even better grasp on exception handling and custom exceptions:</p> <ul> <li>This Stack Overflow post: <a href="http://stackoverflow.com/questions/1319615/proper-way-to-declare-custom-exceptions-in-modern-python">Proper way to declare custom exceptions in modern Python?</a>.</li> <li>This Python doc: <a href="https://docs.python.org/2/library/exceptions.html">Built-in Exceptions</a>.</li> </ul> <p>And as always, search engines are your friends.</p> 2016-03-25T00:00:00+00:00 2016-03-25T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/how-to-develop-a-site-for-the-web/ <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>This post does not deal with the basics of HTML, CSS, or Javascript, but rather a simple and often overlooked part of the web-dev toolchain.</p> <p>If you would like help with the basics of HTML, CSS, and Javascript I suggest <a href="https://codecademy.com/learn/make-a-website">Codecademy</a>.</p> </span> </span> <p>Web Development, especially the front-endy stuff, is a great way to understand how computers <em>think</em> and can be a wonderful foray into computer programming. Anybody that's made a website probably remembers their first time:</p> <ol> <li>Open up a text editor.</li> <li>Type something like the following in:</li> </ol> <pre data-lang="html" style="background-color:#191919;color:#f8f8f2;" class="language-html "><code class="language-html" data-lang="html"><span>&lt;</span><span style="color:#ff5e5e;">h1</span><span>&gt;Hello world!&lt;/</span><span style="color:#ff5e5e;">h1</span><span>&gt; </span><span>&lt;</span><span style="color:#ff5e5e;">p</span><span>&gt;This is so cool!&lt;/</span><span style="color:#ff5e5e;">h1</span><span>&gt; </span></code></pre> <ol start="3"> <li><code>Save As index.html</code>.</li> <li>Right click and open the file in your browser.</li> <li>Marvel at what you've created and hack away at it all night long.</li> </ol> <p>And there you go! You've got a website made and ready to roll. It's not on a server, and you can't tell your friends to go to it from <em>their</em> computer --but those are just technicalities. You can still celebrate doing a thing <em>like a boss</em>.</p> <p>Unfortunately, when you test your website by viewing local files you're missing out on a lot of advantages and quirks that you get when you host a website on a <em>real server</em> using something like <a href="https://en.wikipedia.org/wiki/Web_server">Apache or Nginx</a>.</p> <p>For instance, when you run check your website by clicking through <code>file///home/username/project/files.html</code> all of your hyperlinks that should point to <code>http://mywebsite.ext/somepage/</code> will take you to <code>file:////somepage</code> when they should take you to <code>file:///home/username/projects/myawesomesite/somepage/index.html</code>.</p> <h2 id="run-a-development-server-it-s-easy">Run A Development Server (it's easy!)</h2> <h3 id="the-short-answer">The Short Answer</h3> <p><strong>Use Python!</strong></p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>$ python2 -m SimpleHTTPServer </span><span>Serving HTTP on 0.0.0.0 port 8000 ... </span></code></pre> <h3 id="the-longer-answer">The Longer Answer</h3> <p>You can run your own local webserver to serve files <em>locally</em>. This gets you all of the developmental advantages of running a webserver without having to rent or run a server in the cloud. The best part is that it's very easy to run a development web server.</p> <p>Here's how:</p> <ol> <li><a href="https://python.org/downloads/">Install Python</a></li> <li>Navigate to <code>your-awesome-website</code> directory/folder.</li> <li>Run <code>python -m SimpleHTTPServer</code> if you installed <code>python2</code> or run <code>python -m http.server</code> if you installed <code>python3</code>.</li> </ol> <div class="note"> <span class="note-title"> Note </span> <span class="note-body"> <p>These instructions assume you are doing things in a Unix environment (OSX or Linux) thought the commandline. If you are using Windows you should check out this <a href="http://stackoverflow.com/questions/17351016/set-up-python-simplehttpserver-on-windows">stack overflow post</a> for Windows-specific help.</p> </span> </span> <p>Then in your web browser go to the address <code>http://localhost:8000</code> and bam! <strong>Your</strong> Website.</p> <p><img src="/images/gifs/high-five.gif" alt="High Five.gif" /></p> 2016-03-05T00:00:00+00:00 2016-03-05T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/running-firefox-in-docker/ <p>Turns out you can run Firefox in Docker. It's actually pretty easy:</p> <h2 id="code">Code</h2> <p><strong>From</strong> <a href="https://gist.github.com/ElijahCaine/29e7e829341d58abe370">my github gist</a>.</p> <script src="https://gist.github.com/ElijahCaine/29e7e829341d58abe370.js"></script> <p>Installation instructions (on Linux):</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span># install docker and start the docker daemon </span><span>$ git clone https://gist.github.com/29e7e829341d58abe370.git docker-firefox </span><span>$ ln docker-firefox/ff-docker /some/path/for/binaries </span><span>$ ff-docker -b # -b pulls &amp; builds container, etc used for first time startup. </span></code></pre> <h2 id="notes">Notes</h2> <p>I attempted to use <a href="http://alpinelinux.org/">Alpine Linux</a> as a proof-of-usability, since it's the hip new kid on the containerization block<sup class="footnote-reference"><a href="#1">1</a></sup>, but as it turns out Alpine's Firefox package is pretty fucked<sup class="footnote-reference"><a href="#2">2</a></sup>.</p> <p>I use <a href="https://wiki.archlinux.org/index.php/Dmenu">dmenu</a>, this means I can run <code>ff-docker</code> from my desktop without opening a terminal, effectively making it a super-private-yet-convenient drop-in for Firefox. Pretty neat right?</p> <h2 id="conclusion">Conclusion</h2> <p>This was pretty simple but effective in terms of giving me the option to have more privacy. I can see it being extended to add even more security; e.g., routing all of the traffic in the container through a VPN would be easy enough. Food for thought [citation needed].</p> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p>5mb containers sounds pretty nice to me. <a href="http://gliderlabs.viewdocs.io/docker-alpine/">http://gliderlabs.viewdocs.io/docker-alpine/</a></p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p>I get a segfault every time I try to start Firefox. More info here: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=724227#c11">https://bugzilla.mozilla.org/show_bug.cgi?id=724227#c11</a></p> </div> 2015-10-12T00:00:00+00:00 2015-10-12T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/cat-barcamp-2015/ <p>This weekend I went to <a href="http://catbarcamp.org/">Cat Barcamp 2015</a>, an unconference run by the PSU <a href="http://cat.pdx.edu/">Computer Action Team</a>. I went to some awesome talks (and gave one myself).</p> <p>I enjoyed every talk I went to. Here they are in order:</p> <ol> <li> <p><strong>History of Cryptography.</strong> This was given by one of my OSL co-workers. It didn't contain a ton of new information for me, but was delivered very well and focused on different types of cryptography and the context in which each one was developed. I enjoyed it.</p> </li> <li> <p><strong>Haskell on Steroids.</strong> I kinda checked out during this talk because I wanted to write my slides... Wooops. It looked very interesting and mathey though...</p> </li> <li> <p><strong>Introduction to Test Driven Development in Power Shell.</strong> Whoa Windows (actually though, I learned more about Power Shell than I ever expected to know).</p> </li> <li> <p><strong>MY TALK.</strong> <a href="https://elijah.run/blog/cat-barcamp-2015/#my-talk">(see below)</a></p> </li> <li> <p><strong>A Whirlwind Tour of Lojban.</strong> Lojban seems like a cool language. Corbin (previous LUG president) gave this talk, he's a cool guy... you might even say he's TheMostAwesomeDude.</p> </li> <li> <p><strong>Adventures in IT Security: Then and now.</strong> I like history. I also like computers. This was a good talk. I am biased.</p> </li> <li> <p><strong>Talking to people! (or, How I learned to stop worrying and love the user.)</strong> Spoiler: The secret is understanding social cues.</p> </li> </ol> <h2 id="my-talk">My Talk</h2> <p>I decided on the spur of the moment to give a talk 'How to Write a (very mediocre) Static Site Generator' and with only 40 minutes of prep work I was able to fill an hour pretty well.</p> <p>Here are some links from that:</p> <ul> <li>The <a href="https://gist.github.com/ElijahCaine/0ee1ef21692f3cbb6934">demo code</a>.</li> <li>The <a href="http://elijahcaine.me/catbarcamp2015-demo/">abomination of a website</a> we built between the demo and the end of the session (epilepsy warning... no seriously).</li> <li>The <a href="http://slides.elijahcaine.me/cat-barcamp-2015_how-to-an-ssg/">very sparse slides</a>.</li> </ul> <h2 id="tldr">TLDR</h2> <p>The CAT is an awesome group of people. I'm glad I was able to get a few LUG members to come to the event. If I went to PSU I would be honored to work with the CAT. They throw a hellofan event.</p> 2015-10-07T00:00:00+00:00 2015-10-07T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/covertly-install-packages-with-docker/ <p><a href="https://elijah.run/blog/covertly-install-packages-with-docker/#the-deets-dockerfile-and-commands">Jump to the codey bits</a></p> <h2 id="the-problem-workstations">The Problem: Workstations</h2> <p>At the OSL I use workstations which are provisioned by <a href="https://en.wikipedia.org/wiki/Chef_%28software%29">Chef</a> to look more or less identical (packages are the same and global config files are consistent from machine to machine). This is nice because it allows everybody at the lab to jump from workstation to workstation without dreading the inevitable setup that usually comes with going to a new computer.</p> <p>The only downside here is that to install a package you have to make a pull request to a GitHub Repository in which you modify a JSON file. This is almost always for my own good, but as a dev I am prone to avoiding things that the admins say are good for me (like broccoli, milk, and consistent workstation environments).</p> <p>For instance, yesterday I wanted to use <a href="http://rg3.github.io/youtube-dl/">youtube-dl</a> to grab a video I was watching on repeat*. I could have made a pull request to add the package, waited until 30 after for the workstations to refresh, used the package, and went about my day. This would have been the correct and bureaucratic way to do things.</p> <p>I <em>could</em> have, and probably <em>should</em> have, done that... but I didn't.</p> <p>* firefox kept crashing, I wasn't just stealing music for the sake of it.</p> <h2 id="enter-docker">Enter: Docker</h2> <p>In this case the solution to the 'problem' was <a href="https://en.wikipedia.org/wiki/Docker_%28software%29">Docker</a>. I wrote a Dockerfile which provisioned a container to have <code>youtube-dl</code> installed, spun up the container with a shared directory (<code>$PWD:/home/</code>), and executed the youtube-dl command. The .mp4 video was downloaded to my current working directory and I was able to play it with my media player of choice, all without installing <code>youtube-dl</code> locally.</p> <h2 id="the-deets-dockerfile-and-commands">The Deets: Dockerfile and Commands</h2> <p><code>Dockerfile</code></p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>FROM ubuntu:latest </span><span> </span><span>RUN apt-get -y update </span><span>RUN apt-get -y upgrade </span><span>RUN apt-get -y install python-pip </span><span>RUN pip install youtube-dl </span><span> </span><span>WORKDIR /home </span><span>RUN alias youtube-dl=&#39;/usr/local/bin/youtube-dl&#39; </span></code></pre> <p>If you are in the directory containing the above Dockerfile, build the container with:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>$ docker build -t yt-dl . </span></code></pre> <p>and run it with:</p> <pre data-lang="txt" style="background-color:#191919;color:#f8f8f2;" class="language-txt "><code class="language-txt" data-lang="txt"><span>$ docker run -v $PWD:/home/ yt-dl youtube-dl &lt;YOUTUBE-VID-URL&gt; </span></code></pre> <p>This will download any url's video into the current working directory.</p> <h2 id="pro-tip-add-an-alias">Pro-Tip: Add an <code>Alias</code></h2> <p>Add the following content to your <code>~/.bashrc</code> file and then run <code>source ~/.bashrc</code> to get rid the of long Docker bits of the above command.</p> <pre data-lang="bash" style="background-color:#191919;color:#f8f8f2;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#ff5e5e;">export </span><span style="color:#e9fdac;">$VID_DIR</span><span style="color:#ff5e5e;">=</span><span style="color:#fbe3bf;">/path/to/downloaded/videos/ </span><span style="color:#6699cc;">alias </span><span style="color:#8cdaff;">yt-dl</span><span style="color:#ff5e5e;">=</span><span style="color:#ffffff;">&#39;</span><span style="color:#fbe3bf;">docker run -v $VID_DIR:/home/ yt-dl youtube-dl</span><span style="color:#ffffff;">&#39; </span></code></pre> <h2 id="wait-this-is-overly-complicated">Wait This is Overly Complicated</h2> <p>You're probably thinking: "Hey, couldn't you have just run <code>pip install --user youtube-dl</code>?</p> <p>Yes. I could have. But that wouldn't be fun and wouldn't have given me an excuse to write a blog, post now would it Ms. Smarty Pantz.</p> <hr /> <p><strong>drink the coolaid. come to the docker side.</strong></p> 2015-09-03T00:00:00+00:00 2015-09-03T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-09-03/ <h2 id="travel-travel-travel">Travel Travel Travel</h2> <p>Travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel. travel. Travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel.</p> <p>Travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel. travel. Travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel. travel. Travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel travel. travel travel travel travel travel travel travel travel travel travel travel.</p> <p>Travel travel travel travel travel travel. Travel travel travel travel travel travel travel. Travel travel travel travel travel travel travel travel travel.</p> <p>Sleep.</p> 2015-09-02T00:00:00+00:00 2015-09-02T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-09-02/ <p><img src="/images/europe-2015/prague-charles-bridge.jpg" alt="Charles Bridge." /></p> <p><img src="/images/europe-2015/prague-pilot-statute.jpg" alt="Pilot Statue" /></p> <p><img src="/images/europe-2015/prague-view.jpg" alt="View of Prague with some trees in the way." /></p> <p>I went on a hike, found a great view at the top of a hill.</p> <p><img src="/images/europe-2015/prague-oldtown.jpg" alt="Oldtown Prague" /></p> <p><img src="/images/europe-2015/prague-panorama-with-bridge.jpg" alt="Panorama with bridge" /></p> <p><img src="/images/europe-2015/prague-creative-spelling.jpg" alt="Starbucks Coffee" /></p> <blockquote> <p>'Eli'</p> </blockquote> <p><img src="/images/europe-2015/prague-stunning-sunset.jpg" alt="Stunning sunset" /></p> <blockquote> <p>I swear it's not because I'm lazy and don't want to write. The pictures just capture it better.</p> </blockquote> 2015-09-01T00:00:00+00:00 2015-09-01T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-09-01/ <p>Just like yesterday, I spent most of today enjoying some wonderful talks as well as stressing out -- I mean totally not stressing out -- about my own talk.</p> <p><img src="/images/europe-2015/wtd-day-2.jpg" alt="write the Docs Staff Picture" /></p> <p>The Write the Docs Staff saying thanks.</p> <ul> <li>The first talk of the day was titled "Judas Priest Ate My Scrum Master", which has almost nothing to do with Judas Priest or eating or scrum masters; instead it talked about things that are well known issues and a lot of laws (<a href="https://en.wikipedia.org/wiki/Brooks%E2%80%99_law">Brooks' Law</a>, <a href="https://en.wikipedia.org/wiki/Lehman&#x27;s_laws_of_software_evolution">Lehmann's Laws</a>, <a href="https://en.wikipedia.org/wiki/Pareto_principle">Pareto Principle</a>, <a href="https://en.wikipedia.org/wiki/Conway&#x27;s_law">Conway's Law</a>, etc) that nobody talks about that much but can be super helpful to consider when working in an industry.</li> <li>The second talk covered different types of user interfaces and what to consider when writing error messages, troubleshooting pages, product tours. It was something I hadn't considered having primarily written command line tools with pretty limited and well established interface guidelines.</li> <li>Then I gave a talk. It was a talk. I'll post a video of it once those get put up online and you can decide if it was good or bad.</li> <li>The talk after mine was a nice history lesson and wanted the audience to reflect on what documentations can learn from documentation practices of the past. The TLDR on that one is that it used to suck to write technical docs even more than it sucks now.</li> <li>Then there were a bunch of lightning talks. One talk from a twitter employee (there's a fun name for them I'm sure) mentioned that gathering metrics are is important because without metrics everything you do is just guess and check.</li> <li>The next talk "MacGyvering your docs" brought up the fact that the best documentation toolchain will (should?) only help existing developers. A better toolchain won't attract new devs. That was something I think a lot of projects can lose sight of.</li> <li>The talk after that covered a team of scientists studying clouds (water clouds not internet clouds) using computer simulations. They wrote and documented a piece of software and they ended up publishing their program's documentation in a scientific journal. It's already been sited and everything. I hadn't even considered that as a possibility (publishing docs as a scientific paper).</li> <li>The next two talks were pretty short (15 minute) talks about putting the data where the users need it and an introduction to screen-casting. The first was good for documenting things like an API and the second was good if I ever need to make a screen-cast. I doubt that will happen, but you never know!</li> <li>The last talk "All roads might not lead to docs" held too main takeaways for me: The quote <em>The Purpose of this document is ________</em> and the <a href="http://voiceandtone.com/">mailchimp voice &amp; tone guide</a>, which is a very good document I suggest you read it (the title is pretty self explanatory I think).</li> </ul> <p>For a full list of talks given at the conference, check out the <a href="http://www.writethedocs.org/conf/eu/2015/schedule/">event's schedule</a>.</p> <p>The after party ended up happening at two venues since we got pre-maturely kicked out of the first one; we made the best of it and I got to talk to Anna Jaruga (the cloud research scientist) as well as a <a href="https://developer.mozilla.org/en-US/">Mozilla Development Network</a> writer, both very interesting people.</p> 2015-08-31T00:00:00+00:00 2015-08-31T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-31/ <p>I spent most of today enjoying the Write the Docs talks and (air conditioned) venue. I took some notes.</p> <ul> <li>The keynote about the imposter syndrome was very relatable. The coping mechanism of framing goals in "I want to do..." instead of "I want to be..." seems like it could be effective. I liked the quote "Make better mistakes tomorrow".</li> <li>The second talk about naming things was very good and got me to think about the structure of documentation metadata (using labels vs taxonomy vs thesaurus, etc). This will be useful someday I'm sure.</li> <li>The third talk about [emojii] and the use of emoji was [emojii]. I never considered using them before, but I will going forward.</li> <li>The fourth talk mentioned using <a href="https://en.wikipedia.org/wiki/Selenium_%28software%29?wprov=sfia1">Selenium</a> to test docs, that just makes sense. It's so obvious I wish it was easier.</li> <li>The API Docs talk mentioned that you should give your user the data they will be working with as a top level priority instead of an after thought. This will definitely be a priority for me. I already try to give plenty of examples (since I speak example natively), but I'll be more Conscience of it.</li> <li>The Design Docs with Disability in Mind was interesting. The big takeaway for me was keeping the elderly in mind.</li> <li>The meta talk about crafting a presentation was delivered very well and mentioned adding contrast slide decks... I should have listened to this advice...</li> <li>The second to the last talk of the day covered the plethora of rules we place on documentation. Having just recently helped established some style guides for the OSL docs I think we did a good job of giving some leeway to the writers. The extreme example of 'Limited rules docs' (e.g., the Python Documentation) does not really work well for the reader in my experience.</li> <li>The final talk by some <a href="https://developer.mozilla.org/en-US/">MDN</a> writers equated docs to a garden. The metaphor worked surprisingly well.</li> </ul> <p>For a full list of the talks and their descriptions, visit the <a href="http://www.writethedocs.org/conf/eu/2015/schedule/">event's schedule</a>.</p> <p>Day 1 was pretty emojii. We ended the night with emojii at a venue called The Pub. Many conversations and laughs were had. I like this crowd.</p> 2015-08-30T00:00:00+00:00 2015-08-30T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-30/ <p><img src="/images/europe-2015/prague-city-center.jpg" alt="Prague city center" /></p> <p>I went on a wonderful <a href="http://www.writethedocs.org/conf/eu/2015/#events">city tour</a> of Prague provided by Write The Docs. We crossed some bridges, saw some castles, and experienced many wonderful parts of the city.</p> <p><img src="/images/europe-2015/prague-gelato.jpg" alt="My dumb face and a delicious gelato." /></p> <p><em>Here's me in front of a church eating some gelato.</em></p> <p>After the tour I was very hungry and tagged along with some folks from my tour group on a journey to get food. We went to a restaurant that brought us beer on a model train.</p> <p><img src="/images/europe-2015/prague-beer-train.jpg" alt="Choo choo. Here comes the beer train." /></p> <p><em>This is the train that brought us beer.</em></p> <p>Here are some picture I took of the buildings in the city. I like the architecture a lot.</p> <p><img src="/images/europe-2015/prague-city-center.jpg" alt="Classic architecture in the Prague city center." /></p> <p><img src="/images/europe-2015/prague-street.jpg" alt="Even the small regular old streets in Prague are breathtaking." /></p> <p><em>They are so cool.</em> <em>I haven't found the words to describe why, but the asthetics please me.</em></p> <p>After the tour and food there was a <em>Write the Docs Conference Reception</em> (aka meet &amp; greet) where I got to hang out with many of the attendees of the conference. There was food and fabulous people.</p> <p>I did not have a hard time falling asleep that night.</p> 2015-08-29T00:00:00+00:00 2015-08-29T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-29/ <h2 id="i-strongly-dislike-travel">I Strongly Dislike Travel</h2> <p>I would say I hate travel, but that level of hyperbole is literally the worst thing on the planet...</p> <p>My strong dislike for travel comes mostly from my personal inability to trust the systems that will get me to where I am going. If I have a flight in one week and I have to take a bus or train to the airport, I will spend the entire week running through my head "When is my flight? Umm days, yeesh hours, and ack minutes. How am I getting there? The oh God bus leaving at some time, somewhere bound. Once you get there you will go through security which will take between 5 minutes and 2 hours. Budget your time appropriately. Any of these systems can fail and based on experience they have" over and over and over. The stress is compounded when people don't always speak your language and you forgot to eat a proper meal the morning of.</p> <p><em>That</em> is why I am 3 hours early to my flight from Amsterdam to Prague. I regret nothing.</p> <h2 id="currency-is-weird">Currency is Weird</h2> <p><img src="/images/europe-2015/prague-conversion-rate.jpg" alt="1 USD = 24.17 Czech Republic Koruna" /></p> <p>It is going to take me forever to remember <em>25 here monies is about 1 home monies</em>. Everything is gonna feel really expensive I bet...</p> <h2 id="i-made-it-to-prague">I Made it to Prague</h2> <p>I made it to Prague just fine. They have a cool train system.</p> <p><img src="/images/europe-2015/prague-mustek-stop.jpg" alt="The MUSTEK stop in the Prague underground transit system. Very Soviet-era design." /></p> <p>Way cooler than the underground train systems I'm used to...</p> 2015-08-28T00:00:00+00:00 2015-08-28T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-28/ <h2 id="my-butt-hurts">My Butt Hurts...</h2> <p>Today I went on an awesome bike ride all the way to Haarlem (15km west of Amsterdam). I didn't actually do anything there; I just went, picked up snacks, and headed back. I'm 99% sure I missed out on an awesome Haarlem exclusive cultural experience, but if was really more about the journey (I originally set out to bike to the coast and Haarlem was my compromise to that).</p> <p><img src="/images/europe-2015/amsterdam-huge-park.jpg" alt="I don&#39;t recall where this park is exactly. I remember i twas east of the city center, many kilometers" /></p> <p><img src="/images/europe-2015/amsterdam-huge-park-2.jpg" alt="Another view of the park, this time with a play structure" /></p> <p><em>I found a very neat park near the Western edge of Amsterdam; I ended up spending an hour or so there having a picnic.</em></p> <p>Dilapidated</p> <p><img src="/images/europe-2015/amsterdam-dilapidated-building.jpg" alt="An old and dilapiddated building. It&#39;s hard to tell the building original purpose..." /></p> <p><em>This was a cool abandoned building I found.</em> <em>I like abandoned buildings.</em></p> <p><img src="/images/europe-2015/amsterdam-travelers-trash-can.jpg" alt="A large ice-cream cone shaped trashcan at a 45 degree angle for cyclists to dispose of trash into." /></p> <p><em>I also found this awesome trash can for people riding bikes.</em> <em>It's genius.</em></p> <p><img src="/images/europe-2015/amsterdam-ponies.jpg" alt="Ponies!" /></p> <p><em>PONIES</em></p> <p>After the bike ride I was very tired. I slept a lot. And yet not enough...</p> <h2 id="small-and-large-appreciations">Small (and Large) Appreciations</h2> <p>Here is a list of some of the traits of Amsterdam (and what I have experienced of the Netherlands at large) which I appreciate.</p> <ul> <li>They have signs that tell you you're going the wrong direction.</li> <li>You can bike to another city and be on bike paths the whole time!</li> <li>Bikes are easy to use and aren't shoe-horned in as a second class mode of transportation.</li> <li>The trains are so quiet. Why can't America have quiet trains?</li> </ul> <p>This list will probably grow as I remember things. All in all I like the city and the country. You should visit if you haven't already.</p> 2015-08-27T00:00:00+00:00 2015-08-27T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-27/ <p>TodayI visited the Van Gough museum. I tried to get in using my mom's Amsterdam Museum Pass, but it didn't fly there like it did at the Rijks.</p> <p>It was pretty fun, but the entire experience felt very artificial and plastic. The paintings were great to see in person, but everything around the pieces felt a little tacky. My opinion will likely change about it, but I did not enjoy the Van Gough museum nearly as much as I enjoyed the Rijks Museum, despite my love for Van Gough and his works.</p> <p>I spent the rest of the day walking around and cashed in early. Jetlag + lots of walking = ready for bed by 3pm.</p> 2015-08-26T00:00:00+00:00 2015-08-26T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-26/ <h2 id="the-rijks-museum">The Rijks Museum</h2> <p><img src="/images/europe-2015/amsterdam-canal.jpg" alt="A panorama of an Amsterdam canal" /></p> <p><em>I'm not sure which street I took this panorama on, but I think it looks awesome.</em> <em>It captures the Amsterdam canal street pretty well.</em></p> <p>Today I went to the Rijks Museum and had a bit of a culture overload.</p> <p><img src="/images/europe-2015/amsterdam-milk-maid.jpg" alt="Chilling with the milk maid." /></p> <p>Everybody got selfies with Van Gough, so I got a selfie The Milkmaid. I realize in retrospect that it doesn't work as well.</p> <p>In other news, this is my face when I realized the bathroom stalls go from floor to ceiling.</p> <p><img src="/images/europe-2015/amsterdam-all-the-way.jpg" alt="Surprise and joy" /></p> <p><em>Yes, this photo was taken in the bathroom stall.</em></p> <p>And this is a classy restaurant I found with classy lamps.</p> <p><img src="/images/europe-2015/amsterdam-cute-lamp.jpg" alt="A lamp with a cute switch" /></p> <p><em>They were closed, otherwise I would have gone in.</em></p> <h2 id="some-travel-tips">Some Travel Tips</h2> <p>Although I have not completed my trip, I have some tips to share with you about travel.</p> <ul> <li>Always bring a 'Go Bag' when you go out for the day. Fill it with snacks and necessities and leave plenty of room for memorabilia you get while out and about.</li> <li>Try something new before defaulting to your comfort zone. This is true for food as well as experiences.</li> <li>Always being plenty of snacks and water. You will regret it if you don't.</li> <li>Be patient. You invested a lot to get here, make it count.</li> <li>At a museum, take a picture of the plackard after taking a picture of the piece so you don't forget what it's called.</li> <li>Don't be afraid to sit in a café and just read for a bit. Sometimes that's as culturally diversifying as a day in a museum or city tour.</li> </ul> 2015-08-25T00:00:00+00:00 2015-08-25T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-24-2015-08-25/ <p><img src="/images/europe-2015/amsterdam-view.jpg" alt="A view of amsterdam" /></p> <h2 id="pdx-ams">PDX -&gt; AMS</h2> <p>The flight was uneventful. It was the longest one I've ever been on, which is always a rough milestone to overcome, but actually one of the better flights I've taken. TIL Delta's service does not suck.</p> <p>I usually sleep the entire way to &lt;insert location here&gt;, but I managed to stay productive (coding, reading, writing) for the majority of flight. I must say, it was nice to have my own personal screen to catch up on Last Week Tonight. I've wanted to watch that show for ages...</p> <p>[<em>Lazy bliss not pictured</em>]</p> <p>So good.</p> <h2 id="day-0-part-1">Day 0: Part 1</h2> <p>So I wrote down some thoughts about Amsterdam thus far:</p> <ul> <li>People keep using fake made up words. What's up with that?</li> <li>The roads are tiny but people keep driving on them. It looks like chaos but I haven't seen an accident yet. I'm kinda impressed. People aren't even yelling at each other about almost getting hit.</li> <li>The toilet paper squares are actually rectangles. It might just be the toilet paper at the place I am staying. Either way, I can't deal with this aspect ratio change. I might not be able to poop in Amsterdam.</li> <li><strong>COFFEE COFFEE COFFEE COFFEE COFFEE COFFEE COFFEE COFFEE COFFEE</strong></li> <li>The trains are so quiet. I didn't want to get off...</li> <li><strong>BIKES BIKES BIKES BIKES BIKES BIKES BIKES BIKES BIKES BIKES</strong></li> </ul> <p>Here's some pictures I took:</p> <p><img src="/images/europe-2015/amsterdam-view.jpg" alt="Mostly trees and sky." /></p> <p><em>The view from the location I am staying at.</em></p> <p><img src="/images/europe-2015/amsterdam-park-panorama.jpg" alt="Panorama of an amsterdam park" /></p> <p><em>A panorama from the park outside of where I am staying.</em></p> <p><img src="/images/europe-2015/amsterdam-goats.jpg" alt="Goats!" /></p> <p><em>Some goats I found.</em></p> <p>Aaaaaaaand naptime.</p> <h2 id="day-0-part-2">Day 0: Part 2</h2> <p>I finished today by going to the Red Light District. There were a lot of half naked ladies in windows. They kept winking at me. I'm pretty sure they liked me; it's not like they <em>definitely</em> have a monetary incentive to get my attention or anything.</p> <p>I ended up finding a bar and enjoying a Belgian White Beer. It tasted <a href="https://xkcd.com/1534/">surprisingly good</a></p> <p><img src="/images/europe-2015/amsterdam-beer.jpg" alt="A belgium white beer" /></p> <p><em>So this is what it feels like to drink in a bar... neat.</em></p> 2015-08-23T00:00:00+00:00 2015-08-23T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-23/ <h2 id="all-packed">All Packed</h2> <p>So I'm all packed up and ready to go.</p> <p><img src="/images/europe-2015/packed.jpg" alt="Three weeks of supplies neatly displayed on a pink bed" /></p> <p><strong>Left:</strong> Chargers, electronics, entertainment, toiletries, water bottle, wallet, etc.</p> <p><strong>Center:</strong> The bag, the <em>Write the Docs</em> shirt I'll be wearing.</p> <p><strong>Right:</strong> Clothing. Gameboy SP at the top.</p> <p>Not the <em>perfect</em> bag of things to bring to Europe, but certainly solid and most of it practical. I'm packing an extra bag for all of the trinkets I'll bring back.</p> <p>Let me know if you want anything (if you know how to contact me we're close enough for you to ask.)</p> 2015-08-02T00:00:00+00:00 2015-08-02T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-02/ <p>All of my travel is booked, plane tickets are bought, and vacation time gotten.</p> <p>The only thing left to do now is to write the talk. Easy-peasy.</p> 2015-08-01T00:00:00+00:00 2015-08-01T00:00:00+00:00 Elijah Voigt https://elijah.run/blog/2015-08-01/ <p>This blog was created in preparation for my trip to Europe in about 3 weeks. I wanted to see how difficult it would be to make a blog that could be updated on the go, version controlled with git, and would not require me to log into a laptop or server to make changes.</p> <p>I ended up using PageUp (link in the footer), along with Github Gists (also linked in the footer) to make a mostly useful platform. New changes are pushed to Github, my personal server pulls those changes every 10 minutes, and this page is rebuilt pretty painlessly.</p> <p>Best of all: with a git client on my phone I can make updates anywhere that has a signal and I don't need to bust my laptop out. I wrote this post while I was on the toilet!</p> <p>It wasn't as easy as using an established platform like Twitter, but just like with everything I try to make, I put something together I would want to use and interact with. If a friend started a liveblog for a specific event I would follow it over their Twitter feed any day. Plus I don't have to deal with the character limit of Twitter.</p>