Jeff Bradberry//jeffbradberry.com/2023-08-26T19:44:00-04:00Definitely a madman with a boxBuilding an Org-mode Workflow: Epic Organization2023-08-26T19:44:00-04:002023-08-26T19:44:00-04:00Jeff Bradberrytag:jeffbradberry.com,2023-08-26:/posts/2023/08/orgmode-epic-organization/<p>Not long after my <a class="reference external" href="/posts/2022/07/orgmode-lists-and-checklists/">last Org-mode post</a>, we started planning
for a new release cycle at work. Previously, I was tracking
individual issues or small features as level-1 headlines with action
items immediately underneath as level-2 headlines with the <tt class="docutils literal">TODO</tt>
keyword. Now though, I was going to need to keep track of an entire
collection of interrelated features (an 'epic' in the Agile
terminology), not all of which would be things I would work on.
Clearly I needed to adjust some things to keep a handle on it all.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a><ul>
<li><a class="reference internal" href="#the-new-structure">The New Structure</a></li>
<li><a class="reference internal" href="#hiding-things">Hiding Things</a></li>
<li><a class="reference external" href="https://orgmode.org/manual/Tags.html">Tags</a></li>
</ul>
</li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<div class="section" id="the-new-structure">
<h3>The New Structure</h3>
<p>This process added a couple of layers of complexity, so I started with
deeper nested headlines.</p>
<p>Roughly speaking, level-1 headlines now each correspond to a
high-level epic, which would then contain multiple level-2 story or
individual feature headlines. For a while I also created level-2
headlines that would capture notes and so forth, but as things
proceeded I moved away from that (more on this in the <a class="reference external" href="#hiding-things">next
sub-section</a>).</p>
<pre class="literal-block">
* Generalized solution for adding/removing execution capacity
- https://issues.redhat.com/browse/AAP-2238
- https://issues.redhat.com/browse/AAP-493 (pushed to 4.4)
- https://issues …</pre></div></div><p>Not long after my <a class="reference external" href="/posts/2022/07/orgmode-lists-and-checklists/">last Org-mode post</a>, we started planning
for a new release cycle at work. Previously, I was tracking
individual issues or small features as level-1 headlines with action
items immediately underneath as level-2 headlines with the <tt class="docutils literal">TODO</tt>
keyword. Now though, I was going to need to keep track of an entire
collection of interrelated features (an 'epic' in the Agile
terminology), not all of which would be things I would work on.
Clearly I needed to adjust some things to keep a handle on it all.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a><ul>
<li><a class="reference internal" href="#the-new-structure">The New Structure</a></li>
<li><a class="reference internal" href="#hiding-things">Hiding Things</a></li>
<li><a class="reference external" href="https://orgmode.org/manual/Tags.html">Tags</a></li>
</ul>
</li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<div class="section" id="the-new-structure">
<h3>The New Structure</h3>
<p>This process added a couple of layers of complexity, so I started with
deeper nested headlines.</p>
<p>Roughly speaking, level-1 headlines now each correspond to a
high-level epic, which would then contain multiple level-2 story or
individual feature headlines. For a while I also created level-2
headlines that would capture notes and so forth, but as things
proceeded I moved away from that (more on this in the <a class="reference external" href="#hiding-things">next
sub-section</a>).</p>
<pre class="literal-block">
* Generalized solution for adding/removing execution capacity
- https://issues.redhat.com/browse/AAP-2238
- https://issues.redhat.com/browse/AAP-493 (pushed to 4.4)
- https://issues.redhat.com/browse/AAP-3717 parent feature
** Add a state field to the Instance and InstanceLink models...
** Add POST access to /instances endpoint...
** Create new Receptor Ansible collection to provision Receptor node(s)...
</pre>
<p>Each level-2 story then might contain multiple level-3 headlines,
sometimes a feature, sometimes just an action item. These would
usually be something assigned to an individual engineer, though not
always me. These would also only get the <tt class="docutils literal">TODO</tt> keyword if they
were a simple enough action item. I still wanted to track these
features even if I wasn't assigned, since work that I <em>was</em> doing
would often depend on them. Similarly, it was important that they be
longer-lived than I would normal treat things, so I avoided archiving
items that were technically complete.</p>
<pre class="literal-block">
** DONE Add the ability to add/remove nodes
CLOSED: [2022-09-13 Tue 11:33]
- https://github.com/ansible/tower/issues/5972
- https://github.com/ansible/awx/issues/12847
- https://github.com/ansible/awx/pull/12623 (Alex; merged)
- https://github.com/ansible/awx/pull/12655 (Alex; merged)
*** DONE Instances List: Add a new Instance
CLOSED: [2022-08-08 Mon 17:13]
- https://github.com/ansible/tower/issues/6036
*** DONE Instance Detail: Add new fields and Peers tab
CLOSED: [2022-08-26 Fri 10:09]
- https://github.com/ansible/tower/issues/6038
*** DONE review 12623
CLOSED: [2022-08-09 Tue 13:26]
*** DONE review 12655
CLOSED: [2022-08-23 Tue 09:59]
*** fail to put node in deprovisioning state when removing from UI
- https://github.com/ansible/awx/issues/12847
- https://github.com/ansible/awx/pull/12855 (Alex; merged)
*** DONE Update Instances list view "Status" column to reflect new states
CLOSED: [2022-08-31 Wed 11:00]
- https://github.com/ansible/tower/issues/6035
- https://github.com/ansible/awx/pull/12773 (Kia; merged)
*** DONE Remove hop nodes from Add Instance form
CLOSED: [2022-09-09 Fri 14:51]
- https://github.com/ansible/awx/pull/12774 (Kia; merged)
*** DONE Allow k8s to create Instance Groups
CLOSED: [2022-09-09 Fri 14:52]
- https://github.com/ansible/awx/pull/12781 (Kia; merged)
*** Instances List: Edit, Sync, Remove Instance
- https://github.com/ansible/tower/issues/6041
- https://github.com/ansible/awx/pull/12784 (Alex; merged)
</pre>
<p>Note that while I wasn't typically giving level-2 headlines a <tt class="docutils literal">TODO</tt>
keyword, I <em>did</em> give it a <tt class="docutils literal">DONE</tt> keyword when everything underneath
of it was also done.</p>
<p>Finally, each level-3 feature (if I didn't already consider it to be a
<tt class="docutils literal">TODO</tt> item) might then contain level-4 action items. These
corresponded to the same things I was previously marking as <tt class="docutils literal">TODO</tt>
items, such as "Experiment with some ideas", "Write a PR", or
"Schedule a meeting", if it was a feature that was assigned to me, or
things like "Review PR 1234" if it was someone else's work.</p>
<pre class="literal-block">
*** Instances List: Edit, Sync, Remove Instance
- https://github.com/ansible/tower/issues/6041
- https://github.com/ansible/awx/pull/12784 (Alex; merged)
**** DONE review 12784
CLOSED: [2022-09-09 Fri 14:52]
</pre>
<p>Most of these headlines would have links associated with them.
Level-1 and -2 would often have Jira tickets, and level-2 and -3 might
have Github issues or pull requests. These wind up as a bulleted list
immediately under the headline (as in my examples above).</p>
<p>For a while I would include <em>every</em> link associated with a given item
in the same bullet list, but that grew to be unwieldy as we
accumulated dozens of planning docs and meeting recordings and digital
whiteboard drawings and other such miscellany. So in addition to
capturing the new layers of complexity, I also needed a way to better
control what was visible.</p>
</div>
<div class="section" id="hiding-things">
<h3>Hiding Things</h3>
<p>Key amongst the Org tools for controlling visibility of things are
<a class="reference external" href="https://orgmode.org/manual/Drawers.html">drawers</a>. These are
blocks that can easily be folded and unfolded without having to be
their own headline item, allowing me to hide things out of the way
until I need them.</p>
<p>I have come to use these for lists of links that aren't the ticket or
PR links that ought to remain visible. I also use them for long-lived
notes around planning each feature, and also org checklists to keep
track that all of the acceptance criteria are met.</p>
<pre class="literal-block">
** DONE Add a state field to the Instance and InstanceLink models
CLOSED: [2022-08-25 Thu 13:20]
- https://github.com/ansible/tower/issues/5966
- https://github.com/ansible/awx/pull/12587 (me; unified PR; merged)
:notes:
- [X] update the management commands
- [X] pack the link info into summary_fields or whatever, to avoid
a call to a related endpoint
- [X] the UI needs to be made aware of the healthy -> ready change
- [X] open question: is the enabled toggle orthogonal to the state?
- answer: yes
- disabled nodes shouldn't transition into unavailable from a check
- disabled being a state implies that the user can set disabled
and ready states under some circumstances
:END:
</pre>
<p>Another thing that I wanted to control the visibility of was those
items of work that had already been completed. It didn't make sense
to tuck them into a drawer since they were headlines, but it was still
desirable to reduce the amount of lines that they typically took up in
the Emacs buffer. For this case, I started using the <a class="reference external" href="https://orgmode.org/manual/Property-Syntax.html">property</a> <tt class="docutils literal">:VISIBILITY:
folded</tt>.</p>
<p>To add a new property to a headline, use <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-x</span> p</tt>. You will then
be prompted for the name of the property, which can be tab-completed
if it is one of the built-in properties, and then you will be prompted
for the value. The result will be added to a <tt class="docutils literal">:PROPERTIES:</tt> drawer,
which will be created if one didn't already exist.</p>
<pre class="literal-block">
*** DONE develop implementation plan (sprint 1) [11/11]
CLOSED: [2022-06-30 Thu 11:48] DEADLINE: <2022-07-01 Fri 17:00>
:PROPERTIES:
:VISIBILITY: folded
:END:
</pre>
<p>In order to make sure that both drawers and items with <tt class="docutils literal">:VISIBILITY:
folded</tt> are folded by default when the file is first opened, I had to
configure the <tt class="docutils literal"><span class="pre">org-startup-folded</span></tt> setting to be <tt class="docutils literal">'showall</tt>
instead of its default value of <tt class="docutils literal">'showeverything</tt>.</p>
</div>
<div class="section" id="tags">
<h3>Tags</h3>
<p>The final piece of the puzzle is that I started to use <a class="reference external" href="https://orgmode.org/manual/Tags.html">tags</a> to mark up items in my Org
files. Mainly, I used these to mark the release (<tt class="docutils literal">:release4_2:</tt>,
<tt class="docutils literal">:release4_3:</tt>), the component or components the work affects
(<tt class="docutils literal">:api:</tt>, <tt class="docutils literal">:ui:</tt>, <tt class="docutils literal">:cli:</tt>, <tt class="docutils literal">:collection:</tt>, <tt class="docutils literal">:operator:</tt>,
<tt class="docutils literal">:qe:</tt>, <tt class="docutils literal">:docs:</tt>), or the expected "sprint" that the work was
going to be done in (<tt class="docutils literal">:sprint2:</tt>, etc.).</p>
<pre class="literal-block">
** DONE Add a state field to the Instance and InstanceLink models :api:sprint2:
</pre>
<p>Tags are added by using <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-q</span></tt>, and then filling in the name in
the prompt. This can be tab-completed if the tag exists elsewhere in
the file, or somewhere in the Org configuration.</p>
<p>Once an items are tagged, I can then use <tt class="docutils literal"><span class="pre">C-c</span> a m</tt> to search for
agenda items with a given tag, or <tt class="docutils literal"><span class="pre">C-c</span> a M</tt> for those specifically
that are also <tt class="docutils literal">TODO</tt> items, though it was often enough that I could
take in at a glance what the tags were for an item. Tags do get
inherited by child headlines, so the search will pick them up.</p>
<p>To make it easier to use tags, allowing them to be tab-completed, I
added this configuration to the top of my <tt class="docutils literal">devel.org</tt> file:</p>
<pre class="literal-block">
#+TAGS: api ui cli collection operator qe docs sprint2 sprint3 sprint4 release4_2 release4_3
</pre>
</div>
</div>
<div class="section" id="settings">
<h2>Settings</h2>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">org-startup-folded</span></tt>: Determines the visibility of the contents of
org files when you first visit them. The default value of this
setting is <tt class="docutils literal">showeverything</tt>, which shows the entire contents of
the file unfolded. I want drawers and things marked as folded to
not be visible initially, so I've changed this to <tt class="docutils literal">showall</tt>.</li>
</ul>
<p>After this change, my org-mode section in <tt class="docutils literal">.emacs</tt> looks like this:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c l"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-store-link</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c a"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-agenda</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c c"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-capture</span><span class="p">)</span>
<span class="p">(</span><span class="k">setq</span><span class="w"> </span><span class="nv">org-startup-folded</span><span class="w"> </span><span class="ss">'showall</span>
<span class="w"> </span><span class="nv">org-agenda-files</span><span class="w"> </span><span class="p">(</span><span class="nf">list</span><span class="w"> </span><span class="s">"~/org/"</span><span class="p">)</span>
<span class="w"> </span><span class="nv">org-refile-targets</span><span class="w"> </span><span class="o">'</span><span class="p">((</span><span class="nv">org-agenda-files</span><span class="w"> </span><span class="nb">:maxlevel</span><span class="w"> </span><span class="o">.</span><span class="w"> </span><span class="mi">5</span><span class="p">))</span>
<span class="w"> </span><span class="nv">org-refile-use-outline-path</span><span class="w"> </span><span class="ss">'file</span>
<span class="w"> </span><span class="nv">org-log-into-drawer</span><span class="w"> </span><span class="no">t</span>
<span class="w"> </span><span class="nv">org-log-done</span><span class="w"> </span><span class="ss">'time</span>
<span class="w"> </span><span class="nv">org-log-redeadline</span><span class="w"> </span><span class="ss">'time</span>
<span class="w"> </span><span class="nv">org-log-reschedule</span><span class="w"> </span><span class="ss">'time</span>
<span class="p">)</span>
</pre></div>
</div>
<div class="section" id="key-bindings-learned">
<h2>Key Bindings Learned</h2>
<ul class="simple">
<li><tt class="docutils literal">TAB</tt>: cycle through the visibility options for any item, even if
it is folded by default (i.e. drawers or subtrees with
<tt class="docutils literal">:VISIBILITY: folded</tt> set)<ul>
<li><tt class="docutils literal"><span class="pre">C-u</span> <span class="pre">C-u</span> TAB</tt>: switch back to the default visibility for
everything in the buffer.</li>
</ul>
</li>
</ul>
<p>Tags:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-q</span></tt>: set the tags for the headline.</li>
<li><tt class="docutils literal"><span class="pre">C-c</span> a m</tt>: search for headlines with a given tag</li>
<li><tt class="docutils literal"><span class="pre">C-c</span> a M</tt>: search for <tt class="docutils literal">TODO</tt> headlines with a given tag</li>
</ul>
<p>Drawers:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-x</span> d</tt>: create a new drawer, prompting for the name.</li>
</ul>
<p>Properties:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-x</span> p</tt>: create a new property, prompting for the name and
then the value. So far this was always <tt class="docutils literal">VISIBILITY</tt> and then
<tt class="docutils literal">folded</tt> for me.</li>
</ul>
<p>Checkboxes:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-c</span> #</tt>: updates the progress cookie for a checklist when you
are in the checklist but not directly on the cookie.</li>
<li><tt class="docutils literal"><span class="pre">C-u</span> <span class="pre">C-c</span> #</tt>: updates all of the progress cookies within the file.</li>
</ul>
</div>
<div class="section" id="next-steps">
<h2>Next Steps</h2>
<p>With the increase in the number of things being tracked, I also wanted
a way to let more important tasks rise to the top of my agenda. So
next time, I'll talk about <a class="reference external" href="https://orgmode.org/manual/Priorities.html">priority cookies</a>.</p>
</div>
Building an Org-mode Workflow: Lists and Checklists2022-07-25T21:02:00-04:002022-07-25T21:02:00-04:00Jeff Bradberrytag:jeffbradberry.com,2022-07-25:/posts/2022/07/orgmode-lists-and-checklists/<p>I've reached a point where I need to organize things at a level lower
than individual tasks. So for that, I've started making use of the
<a class="reference external" href="https://orgmode.org/manual/Plain-Lists.html">lists</a> and <a class="reference external" href="https://orgmode.org/manual/Checkboxes.html">checklists</a> features of Org mode.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a></li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<p>As mentioned in the part about notes in <a class="reference external" href="/posts/2022/06/orgmode-basic-todos/#usage">Part 1</a>, I've been using
unordered lists from the beginning. There are other options for
bullets to use with these, but so far I've only been using <tt class="docutils literal">-</tt>.</p>
<p>While it is easy enough to just manually type these out (and I still
do some of the time), the thing I have started to do recently is to
make use of the keybindings to insert them instead. <tt class="docutils literal"><span class="pre">M-RET</span></tt> will
add a new list item if your cursor is already in a list. A typical
thing I might then combine with that is <tt class="docutils literal"><span class="pre">M-RIGHT</span></tt> to increase the
indent of the new item, to turn it into the start of a sub-list. The
only awkwardness then is starting the list, since <tt class="docutils literal"><span class="pre">M-RET</span></tt> will just
create a new headline if you aren't already in a list. Technically
you could turn that headline into a list item with <tt class="docutils literal"><span class="pre">C-c</span> -</tt>, but I
always just manually type the dash …</p></div><p>I've reached a point where I need to organize things at a level lower
than individual tasks. So for that, I've started making use of the
<a class="reference external" href="https://orgmode.org/manual/Plain-Lists.html">lists</a> and <a class="reference external" href="https://orgmode.org/manual/Checkboxes.html">checklists</a> features of Org mode.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a></li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<p>As mentioned in the part about notes in <a class="reference external" href="/posts/2022/06/orgmode-basic-todos/#usage">Part 1</a>, I've been using
unordered lists from the beginning. There are other options for
bullets to use with these, but so far I've only been using <tt class="docutils literal">-</tt>.</p>
<p>While it is easy enough to just manually type these out (and I still
do some of the time), the thing I have started to do recently is to
make use of the keybindings to insert them instead. <tt class="docutils literal"><span class="pre">M-RET</span></tt> will
add a new list item if your cursor is already in a list. A typical
thing I might then combine with that is <tt class="docutils literal"><span class="pre">M-RIGHT</span></tt> to increase the
indent of the new item, to turn it into the start of a sub-list. The
only awkwardness then is starting the list, since <tt class="docutils literal"><span class="pre">M-RET</span></tt> will just
create a new headline if you aren't already in a list. Technically
you could turn that headline into a list item with <tt class="docutils literal"><span class="pre">C-c</span> -</tt>, but I
always just manually type the dash.</p>
<p>Many of my notes turn into these lists.</p>
<pre class="literal-block">
** add-ons and supplementary
- org-ref
- org-cite / org-ref-cite
- Zotero
- org-roam
- org-roam-bibtex
- org-roam-ui
- org-babel (computational notebook)
- org-tree-slide (slideshow)
- org-present (slideshow)
- Beamer (export to pdf)
- ox-extra (ignore specified headlines during export)
</pre>
<p>The actual new thing that I've started to do, though, is to make use
of checkbox items in my lists. These are represented by a prefix of
<tt class="docutils literal">- [ ]</tt> for your list items, and can be automatically added using
<tt class="docutils literal"><span class="pre">M-S-RET</span></tt>. Though again like with ordinary list items, this will
only work if your cursor is already in a list. Otherwise, you will
wind up with a <tt class="docutils literal">TODO</tt> headline.</p>
<p>For further tracking, you can add a progress "cookie" to your
checklists. To start these out, type out a <tt class="docutils literal">[/]</tt> or a <tt class="docutils literal">[%]</tt> at
the end of the parent of your checklist. These will then fill out
with the completed fraction or percentage as you check off the
checkboxes. As far as I'm aware there is no keybinding or function
that will drop this cookie into place, but it is easy enough to just
manually type it out.</p>
<pre class="literal-block">
*** TODO PR for model change [7/10]
- [X] Instance.node_state field with choices
- [X] InstanceLink.link_state field with choices
- [X] Django migration, with data migration
- [ ] update the provision_instance command to set the node_state to
Installed
- Originally this was going to be 'update the provision_instance and
register_peers commands to optionally take the states', but I
currently think this is unnecessary, these commands should only be
used to register established nodes and links
- [X] add the peers/links to the appropriate Instance serializer
- [X] add the states to the serializers
- [X] represent enabled/disabled in the serializers
- I decided to keep the orthogonal .enabled field
- [X] figure out how to represent de/provisioning failures
- [ ] update the heartbeat and healthcheck tasks to do proper
state transitions
- [ ] make sure that job tasks only use ready nodes
- provide via the API a link to the de/provisioning job
- this is punted to the issue about creating the job
</pre>
<p>As shown above, ordinary list items can be mixed freely with checkbox
list items.</p>
<p>In theory checkboxes can be completed by typing in a capital <tt class="docutils literal">X</tt> in
the box, but that will fail to update the cookie (if you have one in
place). Instead, you can use the keybinding <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-c</span></tt> which will
both check off the item and update the cookie. If it happens that you
have made some edits that have left the cookie out of date, you can
instead place your cursor and do <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-c</span></tt> there to update it.</p>
<p>Once created, list items and checklist items can be rearranged using
<tt class="docutils literal"><span class="pre">M-UP</span></tt> and <tt class="docutils literal"><span class="pre">M-DOWN</span></tt>, just like headlines.</p>
</div>
<div class="section" id="settings">
<h2>Settings</h2>
<p>No settings were changed to accommodate these changes to my workflow.</p>
<p>The org-mode section in my <tt class="docutils literal">.emacs</tt> file still looks like this:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c l"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-store-link</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c a"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-agenda</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c c"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-capture</span><span class="p">)</span>
<span class="p">(</span><span class="k">setq</span><span class="w"> </span><span class="nv">org-agenda-files</span><span class="w"> </span><span class="p">(</span><span class="nf">list</span><span class="w"> </span><span class="s">"~/org/"</span><span class="p">)</span>
<span class="w"> </span><span class="nv">org-refile-targets</span><span class="w"> </span><span class="o">'</span><span class="p">((</span><span class="nv">org-agenda-files</span><span class="w"> </span><span class="nb">:maxlevel</span><span class="w"> </span><span class="o">.</span><span class="w"> </span><span class="mi">5</span><span class="p">))</span>
<span class="w"> </span><span class="nv">org-refile-use-outline-path</span><span class="w"> </span><span class="ss">'file</span>
<span class="w"> </span><span class="nv">org-log-into-drawer</span><span class="w"> </span><span class="no">t</span>
<span class="w"> </span><span class="nv">org-log-done</span><span class="w"> </span><span class="ss">'time</span>
<span class="w"> </span><span class="nv">org-log-redeadline</span><span class="w"> </span><span class="ss">'time</span>
<span class="w"> </span><span class="nv">org-log-reschedule</span><span class="w"> </span><span class="ss">'time</span>
<span class="p">)</span>
</pre></div>
</div>
<div class="section" id="key-bindings-learned">
<h2>Key Bindings Learned</h2>
<p>Within lists and/or checklists:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">M-UP</span></tt> / <tt class="docutils literal"><span class="pre">M-DOWN</span></tt>: these keybindings work for moving list item
subtrees up and down within lists/checklists, as well as regular
headline subtrees.</li>
<li><tt class="docutils literal"><span class="pre">M-RIGHT</span></tt> / <tt class="docutils literal"><span class="pre">M-LEFT</span></tt>: promote or demote the current list item,
but not its children.</li>
<li><tt class="docutils literal"><span class="pre">M-S-RIGHT</span></tt> / <tt class="docutils literal"><span class="pre">M-S-LEFT</span></tt>: promote or demote the subtree of the
current list item.</li>
<li><tt class="docutils literal"><span class="pre">M-RET</span></tt>: add a new list item at the same level as the current
one.<ul>
<li><strong>Note</strong>: doing this in the middle of the text of some existing
item will break the line, putting the right side of the text into
the new item entry.</li>
<li><strong>Note</strong>: this binding does not seem to be subtree aware and will
put the new item entry immediately under the current one, even if
there are child items.</li>
<li>if you do this keybinding at the beginning of the existing line,
it will put the new item above the current one, instead of below.</li>
</ul>
</li>
<li><tt class="docutils literal"><span class="pre">M-S-RET</span></tt>: add a new checkbox item at the same level as the
current list item.<ul>
<li><strong>Note</strong>: this has all of the same downsides as <tt class="docutils literal"><span class="pre">M-RET</span></tt> above.</li>
</ul>
</li>
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-c</span></tt>: toggle a checkbox. This does have the side effect of
updating a progress cookie in the parent of the list.</li>
</ul>
<p>On a line with a progress cookie:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-c</span></tt>: update the progress cookie.</li>
</ul>
<p>Outside of lists:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">M-UP</span></tt> / <tt class="docutils literal"><span class="pre">M-DOWN</span></tt>: move a headline subtree up or down.</li>
<li><tt class="docutils literal"><span class="pre">M-RIGHT</span></tt> / <tt class="docutils literal"><span class="pre">M-LEFT</span></tt>: promote or demote the current headline
item, but not its children.</li>
<li><tt class="docutils literal"><span class="pre">M-S-RIGHT</span></tt> / <tt class="docutils literal"><span class="pre">M-S-LEFT</span></tt>: promote or demote the current subtree.</li>
<li><tt class="docutils literal"><span class="pre">M-RET</span></tt>: creates a new headline at the current level. Has all of
the same caveats as the within-list version.</li>
<li><tt class="docutils literal"><span class="pre">M-S-RET</span></tt>: creates a new TODO headline at the current level. Also
has the same caveats.</li>
</ul>
<p>Completely unrelated, I've recently learned that clicking on a link
isn't the only way to open it:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-o</span></tt>: open a link in your browser</li>
</ul>
</div>
<div class="section" id="next-steps">
<h2>Next Steps</h2>
<p>We've started ramping up a cycle of feature development at work
recently, and I'm finding that my original style of organizing tasks
plus notes is a bit awkward to deal with the larger features I need to
work on. Next time I'll attempt to address that.</p>
</div>
Building an Org-mode Workflow: Scheduling and Deadlines2022-06-20T14:24:00-04:002022-06-20T14:24:00-04:00Jeff Bradberrytag:jeffbradberry.com,2022-06-20:/posts/2022/06/orgmode-scheduling-and-deadlines/<p>As I mentioned at the end of part 2, I want to schedule things I need
to do on the calendar. So, this week I explored the <a class="reference external" href="https://orgmode.org/manual/Deadlines-and-Scheduling.html">deadlines and
scheduling</a> features.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a></li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<p>There are two types of these timestamps that you can give to your
agenda items — schedules and deadlines. Schedules are for events that
happen at a particular time, or for when you want to set aside time to
work on something particular. Deadlines are exactly what they sound
like — a date and/or time something is due by. Both of these seem to
behave nearly identically, and you can apply either type or both at
the same time to an item.</p>
<p>In order to apply one of these timestamps to an item, use <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-s</span></tt>
for a schedule, or <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-d</span></tt> for a deadline. This will open up a
little calendar frame, and will request input in the mini-buffer.</p>
<img alt="Org-mode schedule input calendar frame and mini-buffer" src="//jeffbradberry.com/images/orgmode-schedule-input.png" style="width: 707px; height: auto; max-width: 100%;"/>
<p>This prompt is fairly intelligent about what <a class="reference external" href="https://orgmode.org/manual/The-date_002ftime-prompt.html">date and time
specifications</a> it will
take.</p>
<p>When complete, the result will show up as an all-caps keyword
immediately under the headline. It must be the next line, with no
other lines between it and …</p></div><p>As I mentioned at the end of part 2, I want to schedule things I need
to do on the calendar. So, this week I explored the <a class="reference external" href="https://orgmode.org/manual/Deadlines-and-Scheduling.html">deadlines and
scheduling</a> features.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a></li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<p>There are two types of these timestamps that you can give to your
agenda items — schedules and deadlines. Schedules are for events that
happen at a particular time, or for when you want to set aside time to
work on something particular. Deadlines are exactly what they sound
like — a date and/or time something is due by. Both of these seem to
behave nearly identically, and you can apply either type or both at
the same time to an item.</p>
<p>In order to apply one of these timestamps to an item, use <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-s</span></tt>
for a schedule, or <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-d</span></tt> for a deadline. This will open up a
little calendar frame, and will request input in the mini-buffer.</p>
<img alt="Org-mode schedule input calendar frame and mini-buffer" src="//jeffbradberry.com/images/orgmode-schedule-input.png" style="width: 707px; height: auto; max-width: 100%;"/>
<p>This prompt is fairly intelligent about what <a class="reference external" href="https://orgmode.org/manual/The-date_002ftime-prompt.html">date and time
specifications</a> it will
take.</p>
<p>When complete, the result will show up as an all-caps keyword
immediately under the headline. It must be the next line, with no
other lines between it and the task it applies to.</p>
<pre class="literal-block">
* TODO Thing
SCHEDULED: <2022-06-21 Tue>
* TODO Thing with a deadline
DEADLINE: <2022-11-09 Wed>
* TODO Thing with both
SCHEDULED: <2022-06-21 Tue> DEADLINE: <2022-11-09 Wed>
</pre>
<p>Once we start having tasks marked with timestamps, we can start using
the weekly agenda view (<tt class="docutils literal"><span class="pre">C-c</span> a a</tt>). This takes you into a view of
the current week, and from there you can navigate to the previous or
following weeks by pressing <tt class="docutils literal">b</tt> or <tt class="docutils literal">f</tt>. Each day of the week is
shown, and scheduled items are shown under the day they are scheduled
for. Deadlines on the other hand get a warning about how soon they
are due, in addition to the actual date they fall under. Past due
deadlines will persist on the calendar until they are done or
otherwise removed.</p>
<pre class="literal-block">
Week-agenda (W25):
Monday 20 June 2022 W25
home: In 10 d.: TODO Replace water filter
home: In 11 d.: TODO Pay bills
Tuesday 21 June 2022
home: 14:00-15:00 Scheduled: TODO foo bar baz
Wednesday 22 June 2022
Thursday 23 June 2022
Friday 24 June 2022
Saturday 25 June 2022
Sunday 26 June 2022
</pre>
<p>Another thing I was interested in was to have <a class="reference external" href="https://orgmode.org/manual/Repeated-tasks.html">recurring tasks</a>. This can be done
by adding a 'cookie' to the end of your timestamp. The prompt doesn't
seem to allow for including it in your original specification of the
timestamp, but it is easy enough to manually edit it in afterwards.</p>
<p>The most basic of these repeater cookies uses a single <tt class="docutils literal">+</tt> with an
amount of time, e.g. <tt class="docutils literal">+1m</tt>. This repeater will advance by exactly
the specified time no matter the circumstances, past due or not.</p>
<p>before:</p>
<pre class="literal-block">
* TODO foo
DEADLINE: <2022-07-01 Fri +1m>
* TODO bar
DEADLINE: <2022-06-01 Fri +1m>
* TODO baz
DEADLINE: <2022-05-01 Fri +1m>
</pre>
<p>after:</p>
<pre class="literal-block">
* TODO foo
DEADLINE: <2022-08-01 Mon +1m>
:PROPERTIES:
:LAST_REPEAT: [2022-06-20 Mon 12:37]
:END:
- State "DONE" from "TODO" [2022-06-20 Mon 12:37]
* TODO bar
DEADLINE: <2022-07-01 Fri +1m>
:PROPERTIES:...
- State "DONE" from "TODO" [2022-06-20 Mon 12:37]
* TODO baz
DEADLINE: <2022-06-01 Wed +1m>
:PROPERTIES:...
- State "DONE" from "TODO" [2022-06-20 Mon 12:37]
</pre>
<p>Notice that overdue item <tt class="docutils literal">baz</tt> is still overdue after marking it as
complete. Under most circumstances I would think this isn't what you
would want, so I'm not sure when I will actually use this version
instead of the ones I'm about to talk about.</p>
<p>Each time you mark a recurring task as done it gets an entry under
<tt class="docutils literal">:PROPERTIES:</tt> and a timestamped note. This could potentially get
bulky, but we'll deal with that presently. For the remaining examples
I will show them with the properties and notes trimmed off to save
space.</p>
<p>The next repeater cookie type uses a <tt class="docutils literal">++</tt> in front of the time spec,
e.g. <tt class="docutils literal">++2w</tt>. This will potentially cause the recurrence to advance
multiple times until the result is in the future. This is so that if
you complete something overdue enough that its next iteration would
also be overdue, it will just skip ahead.</p>
<p>before:</p>
<pre class="literal-block">
* TODO foo
DEADLINE: <2022-07-01 Fri ++1m>
* TODO bar
DEADLINE: <2022-06-01 Fri ++1m>
* TODO baz
DEADLINE: <2022-05-01 Fri ++1m>
</pre>
<p>after:</p>
<pre class="literal-block">
* TODO foo
DEADLINE: <2022-08-01 Mon ++1m>
* TODO bar
DEADLINE: <2022-07-01 Fri ++1m>
* TODO baz
DEADLINE: <2022-07-01 Fri ++1m>
</pre>
<p>Seriously overdue item <tt class="docutils literal">baz</tt> is now brought fully up to date,
without needing to deal explicitly with the June 1st iteration.
However, it still conforms to the day of month or weekday or whatever
specification you originally gave the task.</p>
<p>Finally, there is the <tt class="docutils literal">.+</tt> repeater. This is for tasks where there
isn't some hard external constraint on when each deadline is, and you
want the result to be based on when you did the last iteration. I'm
only going to give two examples here because the results will be the
same regardless.</p>
<p>before:</p>
<pre class="literal-block">
* TODO foo
DEADLINE: <2022-07-01 Fri .+1m>
* TODO bar
DEADLINE: <2022-06-01 Fri .+1m>
</pre>
<p>after:</p>
<pre class="literal-block">
* TODO foo
DEADLINE: <2022-07-20 Wed .+1m>
* TODO bar
DEADLINE: <2022-07-20 Wed .+1m>
</pre>
<p>This is great for chores where there is some leeway on when they get
done, e.g. replacing the A/C air filter.</p>
<p>I tend to like having a record of things, so I have enabled some
settings that Org-mode provides to make sure changes to schedules or
deadlines get a timestamped entry showing when the change happened:
<tt class="docutils literal"><span class="pre">org-log-redeadline</span></tt> for changes to deadlines, and
<tt class="docutils literal"><span class="pre">org-log-reschedule</span></tt> for changes to schedules. These seem like they
would only rarely need to be examined, though, so I additionally
enabled a setting (<tt class="docutils literal"><span class="pre">org-log-into-drawer</span></tt>) that causes these logs to
be stashed into a <a class="reference external" href="https://orgmode.org/manual/Drawers.html">drawer</a>,
named <tt class="docutils literal">LOGBOOK</tt> by default.</p>
<p>This is my first use of Org-mode drawers, and I suspect that I'm going
to want to use them more later on.</p>
<p>In addition to scheduling changes, this setting will also cause
<tt class="docutils literal">TODO</tt> state changes to be logged, but only if that state change
would result in a note. See the entry in <a class="reference internal" href="#settings">Settings</a> for details.</p>
<p>I should note, however, that these settings will only cause an entry
to be logged if you use the command or key binding to issue a new
schedule or deadline, not if you edit the timestamp as text.</p>
<p>Before I wrap up, I have a couple of new usage patterns this week that
do not fit the theme of schedules and deadlines:</p>
<p>I have discovered a more streamlined way of accessing my org files:
<tt class="docutils literal"><span class="pre">C-'</span></tt> will cycle to the next org file, even if it isn't actually an
open buffer. I'm finding this considerably handier than using <tt class="docutils literal"><span class="pre">C-x</span>
<span class="pre">C-f</span></tt> or <tt class="docutils literal"><span class="pre">C-x</span> b</tt> to pick out the file that I want next.</p>
<p>Also, I have discovered that it is possible to open an agenda view
based on a search for an arbitrary string: <tt class="docutils literal"><span class="pre">C-c</span> a s</tt>. This does
supposedly also include regex searches, though I haven't tried that
part out yet. Mostly I've been using this to quickly find tasks by
ticket number. I can quickly get to the task at hand by pressing
<tt class="docutils literal">SPC</tt> (space) with my cursor on the item in the agenda view list to
cause the upper panel to move to it, then <tt class="docutils literal">q</tt> to quit out of the
agenda view.</p>
</div>
<div class="section" id="settings">
<h2>Settings</h2>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">org-log-into-drawer</span></tt>: Folds all logged timestamp items into a
drawer, by default <tt class="docutils literal">LOGBOOK</tt>. In addition to rescheduling or
changing deadlines, this will also include changes to the <tt class="docutils literal">TODO</tt>
state if<ul>
<li>you explicitly use a key binding or command that makes you write a
note</li>
<li>you have per-keyword configuration set up with the special note or
timestamp markers (see
<a class="reference external" href="https://orgmode.org/manual/Tracking-TODO-state-changes.html">https://orgmode.org/manual/Tracking-TODO-state-changes.html</a>)</li>
<li>you mark a recurring <tt class="docutils literal">TODO</tt> item complete</li>
</ul>
</li>
<li><tt class="docutils literal"><span class="pre">org-log-redeadline</span></tt>: Adds a timestamp to the log for this item
when you change an existing deadline.</li>
<li><tt class="docutils literal"><span class="pre">org-log-reschedule</span></tt>: Adds a timestamp to the log for this item
when you change an existing schedule.</li>
</ul>
<p>After these changes, my org-mode section in <tt class="docutils literal">.emacs</tt> looks like this:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c l"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-store-link</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c a"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-agenda</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c c"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-capture</span><span class="p">)</span>
<span class="p">(</span><span class="k">setq</span><span class="w"> </span><span class="nv">org-agenda-files</span><span class="w"> </span><span class="p">(</span><span class="nf">list</span><span class="w"> </span><span class="s">"~/org/"</span><span class="p">)</span>
<span class="w"> </span><span class="nv">org-refile-targets</span><span class="w"> </span><span class="o">'</span><span class="p">((</span><span class="nv">org-agenda-files</span><span class="w"> </span><span class="nb">:maxlevel</span><span class="w"> </span><span class="o">.</span><span class="w"> </span><span class="mi">5</span><span class="p">))</span>
<span class="w"> </span><span class="nv">org-refile-use-outline-path</span><span class="w"> </span><span class="ss">'file</span>
<span class="w"> </span><span class="nv">org-log-into-drawer</span><span class="w"> </span><span class="no">t</span>
<span class="w"> </span><span class="nv">org-log-done</span><span class="w"> </span><span class="ss">'time</span>
<span class="w"> </span><span class="nv">org-log-redeadline</span><span class="w"> </span><span class="ss">'time</span>
<span class="w"> </span><span class="nv">org-log-reschedule</span><span class="w"> </span><span class="ss">'time</span>
<span class="p">)</span>
</pre></div>
<p>Note that I've moved <tt class="docutils literal"><span class="pre">org-log-done</span></tt> down to group it with these new
settings, since its purpose is similar.</p>
</div>
<div class="section" id="key-bindings-learned">
<h2>Key Bindings Learned</h2>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-s</span></tt>: Set a scheduled time for an entry.</li>
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-d</span></tt>: Set a deadline for an entry.</li>
<li><tt class="docutils literal"><span class="pre">C-c</span> a a</tt>: Show the agenda view for the current week.<ul>
<li><tt class="docutils literal">f</tt>, <tt class="docutils literal">b</tt>: Change the display to the next / previous week.</li>
</ul>
</li>
<li><tt class="docutils literal"><span class="pre">C-c</span> a s</tt>: Put together an agenda view using a search.</li>
<li>Within agenda views<ul>
<li><tt class="docutils literal">SPC</tt>: Immediately takes you to the item your cursor is on,
without changing focus away from the agenda view panel.</li>
<li><tt class="docutils literal">q</tt>: Quit out of agenda views. Not really new, but wanted to
reiterate it since this is a new agenda view.</li>
</ul>
</li>
<li><tt class="docutils literal"><span class="pre">C-'</span></tt>: Cycle through your agenda files.</li>
<li><tt class="docutils literal">TAB</tt>: Can also be used to toggle the visibility of a drawer such
as <tt class="docutils literal">:LOGBOOK:</tt>. However, for drawers you have to place your
cursor directly on the drawer when you press <tt class="docutils literal">TAB</tt>.</li>
</ul>
<p>Not exactly new to me, but the process of learning Org-mode these past
few weeks has caused me to use them considerably more:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-h</span> v</tt>: Look up an Emacs variable in the manual.</li>
<li><tt class="docutils literal"><span class="pre">C-h</span> k</tt>: Look up an Emacs key binding in the manual.</li>
<li><tt class="docutils literal"><span class="pre">C-h</span> f</tt>: Look up an Emacs function in the manual.</li>
</ul>
<p>And a not particularly related one that I needed this week:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-x</span> 8</tt>: Prefix for composing an accented character, e.g. <tt class="docutils literal"><span class="pre">C-x</span> 8
' e</tt> resulting in a 'é' character. You can see a list of these
with <tt class="docutils literal"><span class="pre">C-x</span> 8 <span class="pre">C-h</span></tt>.</li>
</ul>
</div>
<div class="section" id="next-steps">
<h2>Next Steps</h2>
<p>After three weeks of using Org-mode, it's becoming less clear what the
next obvious need is that can easily be addressed. What I'm using now
definitely feels like the low-hanging fruit. Having said that,
though, there are a few things I could explore:</p>
<p>I have been writing quite large and complex notes in some of my
subtrees for tasks at work. This is getting a bit bulky, and it might
be worthwhile to explore whether stashing them in drawers is a good
solution.</p>
<p>As our new release cycle starts up at work, I am going to start
running into the need to break down tasks into a bunch of sub-tasks
and check them off as well. This would likely be quite a short blog
post, though.</p>
<p>Another possibility would be to double-down on logging things. There
are a number of Org-mode features around doing timestamped notes that
might be useful for me.</p>
</div>
Building an Org-mode Workflow: Multiple Files and Archiving2022-06-11T11:57:00-04:002022-06-11T11:57:00-04:00Jeff Bradberrytag:jeffbradberry.com,2022-06-11:/posts/2022/06/orgmode-multiple-files/<p>As I mentioned at the end of part 1, I quickly began to feel the need
to break things up into multiple files to make things more manageable.
So, this week I explored the <a class="reference external" href="https://orgmode.org/manual/Refiling-and-Archiving.html">refile and archiving</a> features.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a></li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<p>To begin with I wanted to break up my items into categories, one
category per file. After some waffling, the categories I settled upon
for work were:</p>
<ul class="simple">
<li><tt class="docutils literal">devel.org</tt>: Every item that is going to involve me personally
writing code.</li>
<li><tt class="docutils literal">reviews.org</tt>: Coding work owned by other developers, that I need
to review or pair with them on.</li>
<li><tt class="docutils literal">watching.org</tt>: Items that I want to keep an eye on, but that
aren't yet ready to be worked on or prioritized</li>
<li><tt class="docutils literal">professional.org</tt>: Professional development, giving talks,
mentoring, or other things of that nature.</li>
<li><tt class="docutils literal">other.org</tt>: Procedural things that don't comfortably fit in any
of the other categories, e.g. talking to IT about problems with one
of my accounts.</li>
</ul>
<p>While I could have just cut and pasted items from my original
<tt class="docutils literal">todo.org</tt> file into their respective new files, I wanted something
a bit more streamlined. It seemed like the <tt class="docutils literal"><span class="pre">org-refile</span></tt> command
(<tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-w …</span></tt></p></div><p>As I mentioned at the end of part 1, I quickly began to feel the need
to break things up into multiple files to make things more manageable.
So, this week I explored the <a class="reference external" href="https://orgmode.org/manual/Refiling-and-Archiving.html">refile and archiving</a> features.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a></li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<p>To begin with I wanted to break up my items into categories, one
category per file. After some waffling, the categories I settled upon
for work were:</p>
<ul class="simple">
<li><tt class="docutils literal">devel.org</tt>: Every item that is going to involve me personally
writing code.</li>
<li><tt class="docutils literal">reviews.org</tt>: Coding work owned by other developers, that I need
to review or pair with them on.</li>
<li><tt class="docutils literal">watching.org</tt>: Items that I want to keep an eye on, but that
aren't yet ready to be worked on or prioritized</li>
<li><tt class="docutils literal">professional.org</tt>: Professional development, giving talks,
mentoring, or other things of that nature.</li>
<li><tt class="docutils literal">other.org</tt>: Procedural things that don't comfortably fit in any
of the other categories, e.g. talking to IT about problems with one
of my accounts.</li>
</ul>
<p>While I could have just cut and pasted items from my original
<tt class="docutils literal">todo.org</tt> file into their respective new files, I wanted something
a bit more streamlined. It seemed like the <tt class="docutils literal"><span class="pre">org-refile</span></tt> command
(<tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-w</span></tt>) fit what I wanted, but by default it would only target
existing headlines within the same file. I couldn't target other
files and I couldn't keep items at the same headline level without
doing some extra work.</p>
<p>It turns out that there is a pair of settings (as noted below in the
<a class="reference internal" href="#settings">Settings</a> section) that allows you to both target all of your org
files and build up a slash-separated target path that includes the
filename. This can be just the file name if you want to move the
subtree to the top level of the target file, which is what I did to
move items out of <tt class="docutils literal">todo.org</tt> into their desired category files. You
can even target the same file it already lives in, which will move it
to the bottom of the file. I started using that particular feature
this week to move items that I had taken some action on but wasn't yet
ready to archive to the bottom of the file, in order to keep things
that I should look at next towards the top. Finally, you can target
some headline if you want the subtree to become a child of that
headline. In my case, I used this for a closely related issue that I
felt should be addressed in the same pull request as another.</p>
<p>Examples:</p>
<ul class="simple">
<li>From <tt class="docutils literal">todo.org</tt>, target <tt class="docutils literal">devel.org/</tt>: Moves the subtree to the
bottom of <tt class="docutils literal">devel.org</tt>.</li>
<li>From <tt class="docutils literal">devel.org</tt>, target <tt class="docutils literal">devel.org/</tt>: Moves the subtree to the
bottom of the file.</li>
<li>From <tt class="docutils literal">devel.org</tt>, target <tt class="docutils literal">reviews.org/Some <span class="pre">top-level</span> headline/</tt>:
Moves the subtree to become a child subtree below this headline.</li>
</ul>
<p>In all of these cases, tab completion works and <tt class="docutils literal">TAB</tt> can be pressed
twice to pull up a menu of potential options.</p>
<p>I should note however, that the files have to already exist in order
to be a valid target for the refile command. So to get the new files
started, I used Emacs' normal <tt class="docutils literal"><span class="pre">find-file</span></tt> command (<tt class="docutils literal"><span class="pre">C-x</span> <span class="pre">C-f</span></tt>) and
then saved it empty.</p>
<p>Finally, this week I reached the point where some items had all of
their actions set to <tt class="docutils literal">DONE</tt>, and I was reasonably sure I wasn't
going to need to revisit them. To deal with these, I used the
<tt class="docutils literal"><span class="pre">org-archive-subtree</span></tt> command (<tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-x</span> <span class="pre">C-s</span></tt>, though possibly I
should have been using <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-x</span> <span class="pre">C-a</span></tt> instead?). This moves the
subtree to an archive file, creating it if it does not exist, based on
whatever pattern is in the settings. I haven't changed this setting
and probably won't care for a while, so this has been the default,
e.g. items from <tt class="docutils literal">devel.org</tt> get sent to <tt class="docutils literal">devel.org_archive</tt>. I
could see those files eventually becoming large, though, so moving
them occasionally or making the pattern include parts of the date
might be nice.</p>
</div>
<div class="section" id="settings">
<h2>Settings</h2>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">org-refile-targets</span></tt>: By default, org-mode only allows refiling to
the current agenda file. Changing this to make use of the
<tt class="docutils literal"><span class="pre">org-agenda-files</span></tt> setting allowed me to target any of my other
org files, which works well in conjunction with the next setting.</li>
<li><tt class="docutils literal"><span class="pre">org-refile-use-outline-path</span></tt>: This allowed me to include the
filename as part of the path to the location I want the subtree to
be moved to.</li>
</ul>
<p>After these changes, the org-mode section in my <tt class="docutils literal">.emacs</tt> file looks
like this:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c l"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-store-link</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c a"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-agenda</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c c"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-capture</span><span class="p">)</span>
<span class="p">(</span><span class="k">setq</span><span class="w"> </span><span class="nv">org-log-done</span><span class="w"> </span><span class="ss">'time</span>
<span class="w"> </span><span class="nv">org-agenda-files</span><span class="w"> </span><span class="p">(</span><span class="nf">list</span><span class="w"> </span><span class="s">"~/org/"</span><span class="p">)</span>
<span class="w"> </span><span class="nv">org-refile-targets</span><span class="w"> </span><span class="o">'</span><span class="p">((</span><span class="nv">org-agenda-files</span><span class="w"> </span><span class="nb">:maxlevel</span><span class="w"> </span><span class="o">.</span><span class="w"> </span><span class="mi">5</span><span class="p">))</span>
<span class="w"> </span><span class="nv">org-refile-use-outline-path</span><span class="w"> </span><span class="ss">'file</span>
<span class="p">)</span>
</pre></div>
</div>
<div class="section" id="key-bindings-learned">
<h2>Key Bindings Learned</h2>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-w</span></tt>: Refile the current subtree to a file or as a child of
some other subtree.</li>
<li><tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-x</span> <span class="pre">C-s</span></tt>: Archive the current subtree into an archive file.
I'm not clear what the difference is between this and <tt class="docutils literal"><span class="pre">C-c</span> <span class="pre">C-x</span>
<span class="pre">C-a</span></tt> (archive subtree using the default command). I should
definitely look into that. The manual also shows a "short" command,
<tt class="docutils literal"><span class="pre">C-c</span> $</tt>. I should try that out, but haven't done so yet.</li>
</ul>
</div>
<div class="section" id="next-steps">
<h2>Next Steps</h2>
<p>This week I started to want the ability to set deadlines and scheduled
events, so that will be what I tackle next week.</p>
</div>
Building an Org-mode Workflow: Basic TODOs2022-06-09T18:47:00-04:002022-06-09T18:47:00-04:00Jeff Bradberrytag:jeffbradberry.com,2022-06-09:/posts/2022/06/orgmode-basic-todos/<p>I've been thinking recently that I need to adopt a task management
system. As a long-time Emacs user, it should be natural for me to
also be a user of <a class="reference external" href="https://orgmode.org/">Org-mode</a>, an Emacs mode
for <a class="reference external" href="https://en.wikipedia.org/wiki/Org-mode">outlining, note-taking, task management, and more</a>. Building a system around
an add-on of a program I already have open every day is a major plus
and ought to encourage me to keep up with it. It will also be not too
far different from the simple ad-hoc text files that I have been
using, but with the addition of some programmed functionality that
will improve my ability to quickly take an action and then move on
without having to think about it too much.</p>
<p>I'm starting here with a fairly small workflow that does the bare
minimum of what I need, then I will incrementally build up new pieces
(and write a blog post about each as I go) as I discover new things
that I want to be able to do.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a></li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<p>For this first phase, I threw everything together into a single file,
called simply <tt class="docutils literal">todo.org</tt>. But since I need to configure
autodetection of org …</p></div><p>I've been thinking recently that I need to adopt a task management
system. As a long-time Emacs user, it should be natural for me to
also be a user of <a class="reference external" href="https://orgmode.org/">Org-mode</a>, an Emacs mode
for <a class="reference external" href="https://en.wikipedia.org/wiki/Org-mode">outlining, note-taking, task management, and more</a>. Building a system around
an add-on of a program I already have open every day is a major plus
and ought to encourage me to keep up with it. It will also be not too
far different from the simple ad-hoc text files that I have been
using, but with the addition of some programmed functionality that
will improve my ability to quickly take an action and then move on
without having to think about it too much.</p>
<p>I'm starting here with a fairly small workflow that does the bare
minimum of what I need, then I will incrementally build up new pieces
(and write a blog post about each as I go) as I discover new things
that I want to be able to do.</p>
<ul class="simple">
<li><a class="reference internal" href="#usage">Usage</a></li>
<li><a class="reference internal" href="#settings">Settings</a></li>
<li><a class="reference internal" href="#key-bindings-learned">Key Bindings Learned</a></li>
<li><a class="reference internal" href="#next-steps">Next Steps</a></li>
</ul>
<div class="section" id="usage">
<h2>Usage</h2>
<p>For this first phase, I threw everything together into a single file,
called simply <tt class="docutils literal">todo.org</tt>. But since I need to configure
autodetection of org files anyway, I'll allow for easy future
expansion and avoid clutter in my home directory by setting the agenda
files to be in a subdirectory, <tt class="docutils literal">~/org</tt>. The <tt class="docutils literal">todo.org</tt> file will
go there.</p>
<p>Each conceptual thing I need to track (I'll go ahead and call these
'items' from here on out) will get a level 1 header. Most, but
certainly not all, of the items I need to keep track of correspond to
an issue in one of our ticket tracking systems. However, each item
might possibly be tracked across more than one ticket in multiple
tracking systems, and they almost certainly will have some additional
thoughts and context that I want to remember; additionally, they might
have artifacts such as pull requests or documents that I will want to
reference. It would be good to have these all in one place where I
can see them at a glance without having to chase a bunch of links.</p>
<p>If an item is sufficiently simple or not expected to need more than
one action, it can be marked with <tt class="docutils literal">TODO</tt> as part of the level 1
header.</p>
<pre class="literal-block">
* TODO requisition a new laptop
</pre>
<p>Otherwise, since I expect many items will wind up needing multiple
actions over their lifetime, <tt class="docutils literal">TODO</tt> actions will mainly become level
2 headers underneath their items. This will allow me to keep state
tracking simple, with only <tt class="docutils literal">TODO</tt> and <tt class="docutils literal">DONE</tt> (and possibly
eventually <tt class="docutils literal">BLOCKED</tt> and <tt class="docutils literal">CANCELED</tt>) states needed instead of the
many columns that issues get on our project board.</p>
<p>While there are some key bindings around inserting new headlines or
<tt class="docutils literal">TODO</tt> entries, I found it simpler this first week to just manually
type them out. The exception is if I already have a headline item
typed out that I then want to turn into a <tt class="docutils literal">TODO</tt> entry, in which
case <tt class="docutils literal"><span class="pre">S-RIGHT</span></tt> will take it into the first state (<tt class="docutils literal">TODO</tt> by
default).</p>
<p>Level 1 items will get a short but meaningful title, but also likely
some simple notes text immediately below it, including any relevant
links. Org-mode headers can get arbitrary amounts of text underneath
by indenting at the same level as the title text of the headline.</p>
<pre class="literal-block">
* Manually gathering analytics from CLI results in a unicode error
https://issues.redhat.com/browse/AAP-3132
https://github.com/ansible/tower/issues/5862
per Phil, only fix in the latest version
** TODO merge if everything is cool
https://github.com/ansible/awx/pull/12252
</pre>
<p>It turns out that URLs automatically get treated as links, opening up
a new tab in your browser when clicked, without the need to do
anything special.</p>
<p>If the size of the notes and links block gets to be more than about
half a dozen lines or so, I will probably break it out into one or
more level 2 items that can then have their visibility toggled using
<tt class="docutils literal">TAB</tt>. Org-mode also supports plain list items by prefixing with a
<tt class="docutils literal">-</tt> instead of asterisks, which I'll likely use for these notes.</p>
<pre class="literal-block">
* Instance Groups fallback / ownership / admin rights
** issues
Remove fallback behavior when an IG is defined on JT or inventory
https://github.com/ansible/awx/issues/6134
Org admin can not operate on instance group associated with their
org
https://github.com/ansible/awx/issues/11910
RFE: Allow assigning an organization IG to a job template without
requiring org admin
https://github.com/ansible/awx/issues/4292
Allow Promptable Instance Group from Predetermined List
https://github.com/ansible/awx/issues/5711 (mentioned by Bill
in 6134)
** notes
- 4292, it seems like what we want is an Org IG Admin role, or make
the existing Org JT Admin role allow attaching IGs
- 6134,
- wants there to be a distinction between fallback and ownership
- Alan proposed a special Terminate group that would not allow
any instances to be added, and could be placed in the ordering
of IGs for an Org such that anything after that point in the
list doesn't get used for fallback
- my counter-proposal is to add a boolean field to IGs that
prevent their use as a fallback
- we need to capture the opposite problem, of JTs that need a
special environment not allowing ordinary IGs to be used as
fallbacks
</pre>
<p>For now, I'm keeping track of the rough order I'll work on items by
raising or lowering their position in the file. Aside from the
obvious method of cutting a region and pasting it somewhere else, the
method I've been using for this first week has been to place the
cursor on the headline and use <tt class="docutils literal"><span class="pre">M-UP</span></tt> and <tt class="docutils literal"><span class="pre">M-DOWN</span></tt>. These key
bindings move the entire subtree up or down. There are probably more
sophisticated key bindings to manipulate whole subtrees, but I'll
worry about that later.</p>
<p>To see an overview of my entire list of <tt class="docutils literal">TODO</tt> actions, I have
started to use the global <tt class="docutils literal">TODO</tt> list agenda view. This can be
accessed using the key binding <tt class="docutils literal"><span class="pre">C-c</span> a t</tt> (assuming you have
<tt class="docutils literal"><span class="pre">org-agenda</span></tt> configured the same way I do, as noted in the <a class="reference internal" href="#settings">Settings</a>
section below). The downside is that I've been using very simple
non-descriptive headers for the direct <tt class="docutils literal">TODO</tt> actions, so it's a bit
difficult to meaningfully read the list. I am mitigating this by
using the follow-mode (the capital <tt class="docutils literal">F</tt> key binding) while moving
around in the <tt class="docutils literal">TODO</tt> list, which causes the upper panel in the
window to move to the place in the file where the item your cursor is
on actually is. I do wonder, though, if other people do something
different to deal with shared context between multiple <tt class="docutils literal">TODO</tt> items.</p>
<p>Finally, I like keeping track of when things happened, so I configured
<tt class="docutils literal">TODO</tt> action items to get marked with a <tt class="docutils literal">CLOSED</tt> timestamped
property whenever they move into the <tt class="docutils literal">DONE</tt> state (or any future
<tt class="docutils literal">DONE</tt>-like states I wind up creating). I'm primarily using
<tt class="docutils literal"><span class="pre">S-RIGHT</span></tt> to move into the <tt class="docutils literal">DONE</tt> state.</p>
<pre class="literal-block">
* ansible-runner streaming with process isolation checks it in every stage
https://github.com/ansible/ansible-runner/issues/1085
https://github.com/ansible/ansible-runner/pull/1084 (Hao PR)
** DONE investigate
CLOSED: [2022-06-01 Wed 10:08]
</pre>
</div>
<div class="section" id="settings">
<h2>Settings</h2>
<p>In each of the posts in this series, I'll note any changes I've made
to my <tt class="docutils literal"><span class="pre">~/.emacs</span></tt> file to support the new patterns I'm using. I tend
to manually edit my config file, but many of these changes can also be
made through the Options / Customize Emacs menu.</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="k">setq</span><span class="w"> </span><span class="nv">org-agenda-files</span><span class="w"> </span><span class="p">(</span><span class="nf">list</span><span class="w"> </span><span class="s">"~/org/"</span><span class="p">)</span>
<span class="w"> </span><span class="nv">org-log-done</span><span class="w"> </span><span class="ss">'time</span>
<span class="p">)</span>
</pre></div>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">org-agenda-files</span></tt>: specifies the files and/or directories that
will automatically be included in your agenda</li>
<li><tt class="docutils literal"><span class="pre">org-log-done</span></tt>: take an action when moving a <tt class="docutils literal">TODO</tt> into a
completed state, in this case give it a timestamp</li>
</ul>
<p>I have also added the key bindings as suggested in the <a class="reference external" href="https://orgmode.org/manual/Activation.html">manual</a>:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c l"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-store-link</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c a"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-agenda</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-set-key</span><span class="w"> </span><span class="p">(</span><span class="nv">kbd</span><span class="w"> </span><span class="s">"C-c c"</span><span class="p">)</span><span class="w"> </span><span class="nf">#'</span><span class="nv">org-capture</span><span class="p">)</span>
</pre></div>
<p>However, I only used the <tt class="docutils literal"><span class="pre">org-agenda</span></tt> binding during this phase, and
that only to access the global <tt class="docutils literal">TODO</tt> list.</p>
</div>
<div class="section" id="key-bindings-learned">
<h2>Key Bindings Learned</h2>
<p>I will also briefly note any new key bindings I have started to use in
each blog post.</p>
<ul class="simple">
<li>Structure Editing:<ul>
<li><tt class="docutils literal"><span class="pre">M-UP</span></tt> / <tt class="docutils literal"><span class="pre">M-DOWN</span></tt>: move a subtree or item up or down, swapping
with the previous or next item as needed</li>
<li><tt class="docutils literal"><span class="pre">M-LEFT</span></tt> / <tt class="docutils literal"><span class="pre">M-RIGHT</span></tt>: decrease or increase the level of the heading</li>
</ul>
</li>
<li><tt class="docutils literal">TODO</tt> items:<ul>
<li><tt class="docutils literal"><span class="pre">S-LEFT</span></tt> / <tt class="docutils literal"><span class="pre">S-RIGHT</span></tt>: select the next or previous <tt class="docutils literal">TODO</tt> state</li>
</ul>
</li>
<li>Visibility:<ul>
<li><tt class="docutils literal">TAB</tt>: toggle the visibility of the current subtree</li>
<li><tt class="docutils literal"><span class="pre">S-TAB</span></tt>: toggle the visibility of the entire file</li>
</ul>
</li>
<li>Agenda views:<ul>
<li><tt class="docutils literal"><span class="pre">C-c</span> a t</tt>: put together and show the global <tt class="docutils literal">TODO</tt> list<ul>
<li><tt class="docutils literal">F</tt>: within an agenda view, toggle the ability to follow the
<tt class="docutils literal">TODO</tt> list item your cursor is on, showing the full thing in
the upper buffer pane</li>
<li><tt class="docutils literal">q</tt>: quit out of an agenda view</li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
<div class="section" id="next-steps">
<h2>Next Steps</h2>
<p>Even with only a partial work week for my first week using Org-mode, I
filled up my <tt class="docutils literal">org/todo.org</tt> file with more items than was
comfortable to look over all at once. I'm definitely going to need to
start doing some multi-file organization, especially archiving items
that I don't need to consider any more.</p>
</div>
Getting Started With Let's Encrypt2016-01-15T08:23:00-05:002016-01-15T08:23:00-05:00Jeff Bradberrytag:jeffbradberry.com,2016-01-15:/posts/2016/01/getting-started-with-lets-encrypt/<p><a class="reference external" href="https://letsencrypt.org/">Let's Encrypt</a> is a new certificate
authority, run in a cooperative effort with the goal of making it easy
for everyone to obtain and renew the certificates needed to enable
secure encrypted connections for their domain. It's free to use, uses
open standards and open source software, and provides certificates via
a fully automated process. Major sponsors of Let's Encrypt include
the <a class="reference external" href="https://www.mozilla.org/">Mozilla Foundation</a> and the <a class="reference external" href="https://www.eff.org/">EFF</a>.</p>
<p>Let's Encrypt entered public beta on December 3rd, so there was no
longer any good reason to hold back from obtaining a certificate for
my site. This blog post will go over the steps I took to get set up
with a Let's Encrypt certificate, and to get everything working.</p>
<p>The documentation for the Let's Encrypt client can be found here:
<a class="reference external" href="https://letsencrypt.readthedocs.org/en/latest/intro.html">https://letsencrypt.readthedocs.org/en/latest/intro.html</a>.</p>
<p>I prefer to run my site on Debian, with <a class="reference external" href="http://nginx.org/en/docs/">nginx</a> as the static file server and as a
reverse proxy for my <a class="reference external" href="https://docs.djangoproject.com/">Django</a> apps,
which run on top of Apache. Since the nginx plugin for Let's Encrypt
is currently 'highly experimental', I decided against using it.
Additionally, my config files are jinja templates kept in version
control, and are populated and pushed by <a class="reference external" href="https://docs.ansible.com/">Ansible …</a></p><p><a class="reference external" href="https://letsencrypt.org/">Let's Encrypt</a> is a new certificate
authority, run in a cooperative effort with the goal of making it easy
for everyone to obtain and renew the certificates needed to enable
secure encrypted connections for their domain. It's free to use, uses
open standards and open source software, and provides certificates via
a fully automated process. Major sponsors of Let's Encrypt include
the <a class="reference external" href="https://www.mozilla.org/">Mozilla Foundation</a> and the <a class="reference external" href="https://www.eff.org/">EFF</a>.</p>
<p>Let's Encrypt entered public beta on December 3rd, so there was no
longer any good reason to hold back from obtaining a certificate for
my site. This blog post will go over the steps I took to get set up
with a Let's Encrypt certificate, and to get everything working.</p>
<p>The documentation for the Let's Encrypt client can be found here:
<a class="reference external" href="https://letsencrypt.readthedocs.org/en/latest/intro.html">https://letsencrypt.readthedocs.org/en/latest/intro.html</a>.</p>
<p>I prefer to run my site on Debian, with <a class="reference external" href="http://nginx.org/en/docs/">nginx</a> as the static file server and as a
reverse proxy for my <a class="reference external" href="https://docs.djangoproject.com/">Django</a> apps,
which run on top of Apache. Since the nginx plugin for Let's Encrypt
is currently 'highly experimental', I decided against using it.
Additionally, my config files are jinja templates kept in version
control, and are populated and pushed by <a class="reference external" href="https://docs.ansible.com/">Ansible</a>. I didn't want to choose a plugin that
would make in-place changes that would then get clobbered on the next
deployment. So, ultimately I decided to go with the webroot plugin.</p>
<p>The documentation's only requirement for webroot is that you make sure
that hidden directories can be served by your webserver. However,
this is a bit problematic for my nginx Django config because I have
everything except for <tt class="docutils literal">/static</tt> and <tt class="docutils literal">/media</tt> proxied to Apache.
It would be better to explicitly expose a single directory rather than
opening up all hidden directories that happen to exist. The solution
is to add an explicit location directive for the hidden directory that
the webroot plugin will be using. This information isn't revealed
anywhere in the docs at the current time, but some Googling around
allowed me to find that it uses <tt class="docutils literal"><span class="pre">WEBROOT/.well-known/</span></tt>. After
adding the directive to expose this directory, my nginx template files
look like this:</p>
<div class="highlight"><pre><span></span><span class="x">server {</span>
<span class="x"> listen 80;</span>
<span class="x"> listen [::]:80;</span>
<span class="x"> server_name </span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">;</span>
<span class="x"> location /.well-known {</span>
<span class="x"> alias /var/www/</span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">/.well-known;</span>
<span class="x"> }</span>
<span class="x"> location /static {</span>
<span class="x"> alias /var/www/</span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">/static;</span>
<span class="x"> }</span>
<span class="x"> location /media {</span>
<span class="x"> alias /var/www/</span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">/media;</span>
<span class="x"> }</span>
<span class="x"> location / {</span>
<span class="x"> proxy_pass http://127.0.0.1:</span><span class="cp">{{</span> <span class="nv">port</span> <span class="cp">}}</span><span class="x">;</span>
<span class="x"> proxy_set_header Host $host;</span>
<span class="x"> }</span>
<span class="x">}</span>
</pre></div>
<p>Once this configuration has been deployed, you are ready to run the
letsencrypt command on the server.</p>
<div class="highlight"><pre><span></span><span class="gp"># </span>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/letsencrypt/letsencrypt/
<span class="gp"># </span><span class="nb">cd</span><span class="w"> </span>letsencrypt
</pre></div>
<p>Let's suppose that I want a single certificate that covers both my
main blog site at <tt class="docutils literal"><span class="pre">(www.)jeffbradberry.com</span></tt>, and a single subdomain
<tt class="docutils literal">foo.jeffbradberry.com</tt> that hosts a Django app. The <tt class="docutils literal"><span class="pre">-w</span>
<filepath></tt> flag specifies the path to the directory that is the root
for the following subdomains, and <tt class="docutils literal"><span class="pre">-d</span> <subdomain></tt> specifies those
subdomains. I'd also like to skip any manual input steps, so I'll add
the <tt class="docutils literal"><span class="pre">--email</span></tt> flag to specify my email address, and <tt class="docutils literal"><span class="pre">--agree-tos</span></tt>
to automatically agree to the terms of service (which can be found
<a class="reference external" href="https://letsencrypt.org/repository/">here</a>, under Subscriber
Agreement). This command, then, requests one certificate that
includes all of the requested domains:</p>
<div class="highlight"><pre><span></span><span class="gp"># </span>./letsencrypt-auto<span class="w"> </span>certonly<span class="w"> </span>--webroot<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-w<span class="w"> </span>/var/www/jeffbradberry.com/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span>jeffbradberry.com<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span>www.jeffbradberry.com<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-w<span class="w"> </span>/var/www/foo.jeffbradberry.com/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span>foo.jeffbradberry.com<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--email<span class="o">=</span>jeffbradberry@example.com<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--agree-tos
</pre></div>
<p>Once the cert is successfully issued, we can configure nginx to
actually use it. We just add <tt class="docutils literal">listen</tt> directives for port 443 with
ssl, and point to the <tt class="docutils literal">fullchain.pem</tt> and <tt class="docutils literal">privkey.pem</tt> files,
which live under the <tt class="docutils literal">/etc/letsencrypt/live/</tt> directory. After these
changes, my config templates now look like this:</p>
<div class="highlight"><pre><span></span><span class="x">server {</span>
<span class="x"> listen 80;</span>
<span class="x"> listen [::]:80;</span>
<span class="x"> listen 443 ssl;</span>
<span class="x"> listen [::]:443 ssl;</span>
<span class="x"> server_name </span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">;</span>
<span class="x"> ssl_certificate /etc/letsencrypt/live/</span><span class="cp">{{</span> <span class="nv">domain</span> <span class="cp">}}</span><span class="x">/fullchain.pem;</span>
<span class="x"> ssl_certificate_key /etc/letsencrypt/live/</span><span class="cp">{{</span> <span class="nv">domain</span> <span class="cp">}}</span><span class="x">/privkey.pem;</span>
<span class="x"> location /.well-known {</span>
<span class="x"> alias /var/www/</span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">/.well-known;</span>
<span class="x"> }</span>
<span class="x"> location /static {</span>
<span class="x"> alias /var/www/</span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">/static;</span>
<span class="x"> }</span>
<span class="x"> location /media {</span>
<span class="x"> alias /var/www/</span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">/media;</span>
<span class="x"> }</span>
<span class="x"> location / {</span>
<span class="x"> proxy_pass http://127.0.0.1:</span><span class="cp">{{</span> <span class="nv">port</span> <span class="cp">}}</span><span class="x">;</span>
<span class="x"> proxy_set_header Host $host;</span>
<span class="x"> }</span>
<span class="x">}</span>
</pre></div>
<p>I also had to modify a task in my Ansible nginx role, to open up port
443 in addition to port 80 in my firewall (ufw).</p>
<div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">allow http(s) through firewall</span>
<span class="w"> </span><span class="nt">ufw</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">rule=allow port={{ item }}</span>
<span class="w"> </span><span class="nt">with_items</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">http</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https</span>
</pre></div>
<p>One deployment later, and I was able to confirm that
<a class="reference external" href="https://jeffbradberry.com">https://jeffbradberry.com</a> was working.</p>
<p>When we are satisfied that the command works and doesn't require user
input, we can automate renewal by putting it into a cronjob. Renewal
of Let's Encrypt certificates work by running the <tt class="docutils literal"><span class="pre">letsencrypt-auto</span></tt>
command with the same parameters that you originally ran it with.
These certificates expire after 90 days, so we need to set up a
renewal to happen a bit more frequently than that. Here I'm going
with a 2 month interval, 5:30am on the first day of each even-numbered
month, followed by a reload of nginx:</p>
<div class="highlight"><pre><span></span># /etc/cron.d/letsencrypt
30 5 1 */2 * /root/letsencrypt/letsencrypt-auto certonly --webroot ... && /etc/init.d/nginx reload
</pre></div>
<p>So what's next?</p>
<p>I wanted to see how secure my setup was, so I used the <a class="reference external" href="https://www.ssllabs.com/ssltest/index.html">Qualsys SSL
Server Test</a> to rate my
domain. The result was a 'B' grade. The main complaints were that
the server was using <a class="reference external" href="https://weakdh.org/">weak Diffie-Hellman</a> (DH)
key exchange parameters, that it was using common DH primes, and that
it didn't have session resumption via caching enabled. Additionally,
though it didn't trigger warnings, OCSP stapling and String Transport
Security (HSTS) were not enabled. For more information about these,
see <a class="reference external" href="https://wiki.mozilla.org/Security/Server_Side_TLS">https://wiki.mozilla.org/Security/Server_Side_TLS</a>.</p>
<p>To address these issues, first I generated a strong DH parameter using</p>
<div class="highlight"><pre><span></span><span class="gp"># </span><span class="nb">cd</span><span class="w"> </span>/etc/ssl/certs
<span class="gp"># </span>openssl<span class="w"> </span>dhparam<span class="w"> </span>-out<span class="w"> </span>dhparam.pem<span class="w"> </span><span class="m">4096</span>
</pre></div>
<p>The full path to this file can then be added to the nginx config under
the <tt class="docutils literal">ssl_dhparam</tt> directive.</p>
<p>Next, I used the <a class="reference external" href="https://mozilla.github.io/server-side-tls/ssl-config-generator/">Mozilla SSL Configuration Generator</a> to
construct good settings. After these changes, my templates now look
like</p>
<div class="highlight"><pre><span></span><span class="x">server {</span>
<span class="x"> listen 80;</span>
<span class="x"> listen [::]:80;</span>
<span class="x"> listen 443 ssl;</span>
<span class="x"> listen [::]:443 ssl;</span>
<span class="x"> server_name </span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">;</span>
<span class="x"> ssl_session_timeout 1d;</span>
<span class="x"> ssl_session_cache shared:SSL:50m;</span>
<span class="x"> ssl_session_tickets off;</span>
<span class="x"> ssl_dhparam /etc/ssl/certs/dhparam.pem;</span>
<span class="x"> ssl_certificate /etc/letsencrypt/live/</span><span class="cp">{{</span> <span class="nv">domain</span> <span class="cp">}}</span><span class="x">/fullchain.pem;</span>
<span class="x"> ssl_certificate_key /etc/letsencrypt/live/</span><span class="cp">{{</span> <span class="nv">domain</span> <span class="cp">}}</span><span class="x">/privkey.pem;</span>
<span class="x"> ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';</span>
<span class="x"> ssl_prefer_server_ciphers on;</span>
<span class="x"> add_header Strict-Transport-Security max-age=15768000;</span>
<span class="x"> ssl_stapling on;</span>
<span class="x"> ssl_stapling_verify on;</span>
<span class="x"> location /.well-known {</span>
<span class="x"> alias /var/www/</span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">/.well-known;</span>
<span class="x"> }</span>
<span class="x"> location /static {</span>
<span class="x"> alias /var/www/</span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">/static;</span>
<span class="x"> }</span>
<span class="x"> location /media {</span>
<span class="x"> alias /var/www/</span><span class="cp">{{</span> <span class="nv">sitename</span> <span class="cp">}}</span><span class="x">/media;</span>
<span class="x"> }</span>
<span class="x"> location / {</span>
<span class="x"> proxy_pass http://127.0.0.1:</span><span class="cp">{{</span> <span class="nv">port</span> <span class="cp">}}</span><span class="x">;</span>
<span class="x"> proxy_set_header Host $host;</span>
<span class="x"> }</span>
<span class="x">}</span>
</pre></div>
<p>This configuration is sufficient to raise my site's grade to an 'A+'.</p>
<p>Everything so far has been a bunch of manual operations, so now I want
to automate it. To do so, I created the following <tt class="docutils literal">letsencrypt.yml</tt>
playbook, to run as a one-off script against brand-new deployments:</p>
<div class="highlight"><pre><span></span><span class="nn">---</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">hosts</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">all</span>
<span class="w"> </span><span class="nt">vars</span><span class="p">:</span>
<span class="w"> </span><span class="nt">domain</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jeffbradberry.com</span>
<span class="w"> </span><span class="nt">email</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jbradberry@example.com</span>
<span class="w"> </span><span class="nt">webroots_and_domains</span><span class="p">:</span><span class="w"> </span><span class="s">"-w</span><span class="nv"> </span><span class="s">/var/www/jeffbradberry.com/</span><span class="nv"> </span><span class="s">-d</span><span class="nv"> </span><span class="s">jeffbradberry.com</span><span class="nv"> </span><span class="s">-d</span><span class="nv"> </span><span class="s">www.jeffbradberry.com</span><span class="nv"> </span><span class="s">-w</span><span class="nv"> </span><span class="s">/var/www/foo.jeffbradberry.com/</span><span class="nv"> </span><span class="s">-d</span><span class="nv"> </span><span class="s">foo.jeffbradberry.com"</span>
<span class="w"> </span><span class="nt">tasks</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">checkout or update the letsencrypt repo</span>
<span class="w"> </span><span class="nt">git</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">repo=https://github.com/letsencrypt/letsencrypt</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">dest=/root/letsencrypt</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">check if we already have a letsencrypt cert</span>
<span class="w"> </span><span class="nt">stat</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">path=/etc/letsencrypt/live/{{ domain }}/fullchain.pem</span>
<span class="w"> </span><span class="nt">register</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cert</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">obtain initial letsencrypt cert</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/root/letsencrypt/letsencrypt-auto certonly --webroot</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">{{ webroots_and_domains }}</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">--email={{ email }}</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">--agree-tos</span>
<span class="w"> </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cert.stat.exists == False</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">add renewal cron job</span>
<span class="w"> </span><span class="nt">cron</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">name="renew cert"</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cron_file=letsencrypt</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">user=root</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">month="2,4,6,8,10,12" day="28"</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">hour="13" minute="40"</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">job="/root/letsencrypt/letsencrypt-auto certonly --webroot {{ webroots_and_domains }} --email={{ email }} --agree-tos --expand && /etc/init.d/nginx reload"</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">state=present</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">create strong DH params if we haven\'t already done so</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">creates=/etc/ssl/certs/dhparam.pem</span>
</pre></div>
<p>Note the <tt class="docutils literal"><span class="pre">--expand</span></tt> flag to the <tt class="docutils literal"><span class="pre">letsencrypt-auto</span></tt> command, which
automates adding more subdomains to your certificate when you renew.
Also, in the final task in this playbook the <tt class="docutils literal">creates=</tt> parameter to
<tt class="docutils literal">command</tt> ensures that the task will only fire off if this file does
not yet exist.</p>
<p>Now that we have everything working satisfactorily, we can put the
finishing touches on the site. All content should be checked for
internal links that use a hard-coded <tt class="docutils literal"><span class="pre">http://</span></tt>, and these should be
replaced with either <tt class="docutils literal">//</tt> or <tt class="docutils literal"><span class="pre">https://</span></tt>, depending on whether you
want to enforce the use of ssl or not. If you do decide that you want
to force ssl, you should also move the <tt class="docutils literal">listen 80</tt> statements into
their own server block in the nginx config and redirect:</p>
<div class="highlight"><pre><span></span><span class="k">server</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="mi">80</span><span class="p">;</span>
<span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="s">[::]:80</span><span class="p">;</span>
<span class="w"> </span><span class="kn">return</span><span class="w"> </span><span class="mi">301</span><span class="w"> </span><span class="s">https://</span><span class="nv">$server_name$request_uri</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p><strong>UPDATE 2/28:</strong> Fixed up the cron job and Ansible playbook.</p>
Minimum Paragraph Widths, Revisited2015-09-12T14:48:00-04:002015-09-12T14:48:00-04:00Jeff Bradberrytag:jeffbradberry.com,2015-09-12:/posts/2015/09/minimum-paragraph-widths-revisited/<p>There's a great little demo up on <a class="reference external" href="https://css-tricks.com/minimum-paragraph-widths/">css-tricks.com</a>, demonstrating a
technique that uses CSS pseudo-elements to prevent paragraphs from
becoming too squeezed when flowing around floated images.</p>
<p>The thing is, it didn't work when I tried to do it myself.</p>
<p>At least, not in my preferred browser, <a class="reference external" href="https://www.mozilla.org/en-US/firefox/products/">Firefox</a>. And what was
really puzzling was that the demo <em>did</em> work in the exact same
browser.</p>
<p>Then I found <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=290146">https://bugzilla.mozilla.org/show_bug.cgi?id=290146</a>.
Apparently, Firefox has had a bug until recently where elements with
<tt class="docutils literal">overflow: hidden</tt> set (perhaps <tt class="docutils literal">scroll</tt> and <tt class="docutils literal">auto</tt> as well)
would fail to establish a new block formatting context when the
element was at the top of the container. The css-tricks.com demo
works, then, because of the 1px red border. If you turn off that css
using Firebug (or whatever), you'll be able to see the problem there
as well.</p>
<p>My workaround? I've added a 1px top padding to my <tt class="docutils literal"><p></tt> elements.</p>
<p>I hope that this post spares someone else the frustration I
experienced tracking this down.</p>
Introduction to Monte Carlo Tree Search2015-09-07T10:17:00-04:002015-09-07T10:17:00-04:00Jeff Bradberrytag:jeffbradberry.com,2015-09-07:/posts/2015/09/intro-to-monte-carlo-tree-search/<p>The subject of game AI generally begins with so-called <em>perfect
information</em> games. These are turn-based games where the players have
no information hidden from each other and there is no element of
chance in the game mechanics (such as by rolling dice or drawing cards
from a shuffled deck). Tic Tac Toe, Connect 4, Checkers, Reversi,
Chess, and Go are all games of this type. Because everything in this
type of game is fully determined, a tree can, in theory, be
constructed that contains all possible outcomes, and a value assigned
corresponding to a win or a loss for one of the players. Finding the
best possible play, then, is a matter of doing a search on the tree,
with the method of choice at each level alternating between picking
the maximum value and picking the minimum value, matching the
different players' conflicting goals, as the search proceeds down the
tree. This algorithm is called <a class="reference external" href="https://en.wikipedia.org/wiki/Minimax">Minimax</a>.</p>
<p>The problem with Minimax, though, is that it can take an impractical
amount of time to do a full search of the game tree. This is
particularly true for games with a high <em>branching factor</em>, or high
average number of available moves per turn …</p><p>The subject of game AI generally begins with so-called <em>perfect
information</em> games. These are turn-based games where the players have
no information hidden from each other and there is no element of
chance in the game mechanics (such as by rolling dice or drawing cards
from a shuffled deck). Tic Tac Toe, Connect 4, Checkers, Reversi,
Chess, and Go are all games of this type. Because everything in this
type of game is fully determined, a tree can, in theory, be
constructed that contains all possible outcomes, and a value assigned
corresponding to a win or a loss for one of the players. Finding the
best possible play, then, is a matter of doing a search on the tree,
with the method of choice at each level alternating between picking
the maximum value and picking the minimum value, matching the
different players' conflicting goals, as the search proceeds down the
tree. This algorithm is called <a class="reference external" href="https://en.wikipedia.org/wiki/Minimax">Minimax</a>.</p>
<p>The problem with Minimax, though, is that it can take an impractical
amount of time to do a full search of the game tree. This is
particularly true for games with a high <em>branching factor</em>, or high
average number of available moves per turn. This is because the basic
version of Minimax needs to search all of the nodes in the tree to
find the optimal solution, and the number of nodes in the tree that
must be checked grows exponentially with the branching factor. There
are methods of mitigating this problem, such as searching only to a
limited number of moves ahead (or <em>ply</em>) and then using an <em>evaluation
function</em> to estimate the value of the position, or by <a class="reference external" href="https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning">pruning</a> branches
to be searched if they are unlikely to be worthwhile. Many of these
techniques, though, require encoding domain knowledge about the game,
which may be difficult to gather or formulate. And while such methods
have produced Chess programs capable of defeating grandmasters,
similar success in Go has been elusive, particularly for programs
playing on the full 19x19 board.</p>
<p>However, there is a game AI technique that does do well for games with
a high branching factor and has come to dominate the field of Go
playing programs. It is easy to create a basic implementation of this
algorithm that will give good results for games with a smaller
branching factor, and relatively simple adaptations can build on it
and improve it for games like Chess or Go. It can be configured to
stop after any desired amount of time, with longer times resulting in
stronger game play. Since it doesn't necessarily require
game-specific knowledge, it can be used for <a class="reference external" href="https://en.wikipedia.org/wiki/General_game_playing">general game playing</a>. It may even
be adaptable to games that incorporate randomness in the rules. This
technique is called Monte Carlo Tree Search. In this article I will
describe how <abbr title="Monte Carlo Tree Search">MCTS</abbr> works,
specifically a variant called Upper Confidence bound applied to Trees
(<abbr title="Upper Confidence bound applied to Trees">UCT</abbr>), and then will
show you how to build a basic implementation in Python.</p>
<p>Imagine, if you will, that you are faced with a row of slot machines,
each with different payout probabilities and amounts. As a rational
person (if you are going to play them at all), you would prefer to use
a strategy that will allow you to maximize your net gain. But how can
you do that? For whatever reason, there is no one nearby, so you
can't watch someone else play for a while to gain information about
which is the best machine. Clearly, your strategy is going to have to
balance playing all of the machines to gather that information
yourself, with concentrating your plays on the observed best machine.
One strategy, called <abbr title="Upper Confidence Bound 1">UCB1</abbr>, does
this by constructing statistical <em>confidence intervals</em> for each
machine</p>
<div class="math">
\begin{equation*}
\bar{x}_i \pm \sqrt{\frac{2 \ln n}{n_i}}
\end{equation*}
</div>
<p>where:</p>
<ul class="simple">
<li><span class="math">\(\bar{x}_i\)</span>: the mean payout for machine <span class="math">\(i\)</span></li>
<li><span class="math">\(n_i\)</span>: the number of plays of machine <span class="math">\(i\)</span></li>
<li><span class="math">\(n\)</span>: the total number of plays</li>
</ul>
<p>Then, your strategy is to pick the machine with the highest upper
bound each time. As you do so, the observed mean value for that
machine will shift and its confidence interval will become narrower,
but all of the other machines' intervals will widen. Eventually, one
of the other machines will have an upper bound that exceeds that of
your current one, and you will switch to that one. This strategy has
the property that your <em>regret</em>, the difference between what you would
have won by playing solely on the actual best slot machine and your
expected winnings under the strategy that you do use, grows only as
<span class="math">\(\mathcal{O}(\ln n)\)</span>. This is the same <a class="reference external" href="https://en.wikipedia.org/wiki/Big_O_notation">big-O</a> growth rate as the
theoretical best for this problem (referred to as the <em>multi-armed
bandit problem</em>), and has the additional benefit of being easy to
calculate.</p>
<p>And here's how Monte Carlo comes in. In a standard Monte Carlo
process, a large number of random simulations are run, in this case,
from the board position that you want to find the best move for.
Statistics are kept for each possible move from this starting state,
and then the move with the best overall results is returned. The
downside to this method, though, is that for any given turn in the
simulation, there may be many possible moves, but only one or two that
are good. If a random move is chosen each turn, it becomes extremely
unlikely that the simulation will hit upon the best path forward. So,
UCT has been proposed as an enhancement. The idea is this: any given
board position can be considered a multi-armed bandit problem, if
statistics are available for all of the positions that are only one
move away. So instead of doing many purely random simulations, UCT
works by doing many multi-phase <em>playouts</em>.</p>
<div class="figure align-left" style="width: 347px; height: auto; max-width: 100%;">
<img alt="Selection" src="//jeffbradberry.com/images/mcts_selection.png" style="width: 347px; height: auto; max-width: 100%;"/>
<p class="caption">Selection</p>
<div class="legend">
Here the positions and moves selected by the UCB1 algorithm at each
step are marked in bold. Note that a number of playouts have
already been run to accumulate the statistics shown. Each circle
contains the number of wins / number of times played.</div>
</div>
<p class="group">The first phase, <em>selection</em>, lasts while you have the statistics
necessary to treat each position you reach as a multi-armed bandit
problem. The move to use, then, would be chosen by the UCB1 algorithm
instead of randomly, and applied to obtain the next position to be
considered. Selection would then proceed until you reach a position
where not all of the child positions have statistics recorded.</p>
<div class="figure align-right" style="width: 347px; height: auto; max-width: 100%;">
<img alt="Expansion" src="//jeffbradberry.com/images/mcts_expansion.png" style="width: 347px; height: auto; max-width: 100%;"/>
<p class="caption">Expansion</p>
<div class="legend">
The position marked 1/1 at the bottom of the tree has no further
statistics records under it, so we choose a random move and add a
new record for it (bold), initialized to 0/0.</div>
</div>
<p class="group">The second phase, <em>expansion</em>, occurs when you can no longer apply
UCB1. An unvisited child position is randomly chosen, and a new
record node is added to the tree of statistics.</p>
<div class="figure align-left" style="width: 347px; height: auto; max-width: 100%;">
<img alt="Simulation" src="//jeffbradberry.com/images/mcts_simulation.png" style="width: 347px; height: auto; max-width: 100%;"/>
<p class="caption">Simulation</p>
<div class="legend">
Once the new record is added, the Monte Carlo simulation begins,
here depicted with a dashed arrow. Moves in the simulation may be
completely random, or may use calculations to weight the randomness
in favor of moves that may be better.</div>
</div>
<p class="group">After expansion occurs, the remainder of the playout is in phase 3,
<em>simulation</em>. This is done as a typical Monte Carlo simulation,
either purely random or with some simple weighting heuristics if a
<em>light playout</em> is desired, or by using some computationally expensive
heuristics and evaluations for a <em>heavy playout</em>. For games with a
lower branching factor, a light playout can give good results.</p>
<div class="group figure align-right" style="width: 347px; height: auto; max-width: 100%;">
<img alt="Back-propagation" src="//jeffbradberry.com/images/mcts_backprop.png" style="width: 347px; height: auto; max-width: 100%;"/>
<p class="caption">Back-Propagation</p>
<div class="legend">
After the simulation reaches an end, all of the records in the path
taken are updated. Each has its play count incremented by one, and
each that matches the winner has its win count incremented by one,
here shown by the bolded numbers.</div>
</div>
<p class="group">Finally, the fourth phase is the <em>update</em> or <em>back-propagation</em> phase.
This occurs when the playout reaches the end of the game. All of the
positions visited during this playout have their play count
incremented, and if the player for that position won the playout, the
win count is also incremented.</p>
<p>This algorithm may be configured to stop after any desired length of
time, or on some other condition. As more and more playouts are run,
the tree of statistics grows in memory and the move that will finally
be chosen will converge towards the actual optimal play, though that
may take a very long time, depending on the game.</p>
<p>For more details about the mathematics of UCB1 and UCT, see
<a class="reference external" href="http://homes.di.unimi.it/~cesabian/Pubblicazioni/ml-02.pdf">Finite-time Analysis of the Multiarmed Bandit Problem</a> and
<a class="reference external" href="https://www.lri.fr/~sebag/Examens_2008/UCT_ecml06.pdf">Bandit based Monte-Carlo Planning</a>.</p>
<p>Now let's see some code. To separate concerns, we're going to need a
<tt class="docutils literal">Board</tt> class, whose purpose is to encapsulate the rules of a game
and which will care nothing about the AI, and a <tt class="docutils literal">MonteCarlo</tt> class,
which will only care about the AI algorithm and will query into the
<tt class="docutils literal">Board</tt> object in order to obtain information about the game. Let's
assume a <tt class="docutils literal">Board</tt> class supporting this interface:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Board</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># Returns a representation of the starting state of the game.</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">current_player</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span>
<span class="c1"># Takes the game state and returns the current player's</span>
<span class="c1"># number.</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">next_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">play</span><span class="p">):</span>
<span class="c1"># Takes the game state, and the move to be applied.</span>
<span class="c1"># Returns the new game state.</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">legal_plays</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state_history</span><span class="p">):</span>
<span class="c1"># Takes a sequence of game states representing the full</span>
<span class="c1"># game history, and returns the full list of moves that</span>
<span class="c1"># are legal plays for the current player.</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">winner</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state_history</span><span class="p">):</span>
<span class="c1"># Takes a sequence of game states representing the full</span>
<span class="c1"># game history. If the game is now won, return the player</span>
<span class="c1"># number. If the game is still ongoing, return zero. If</span>
<span class="c1"># the game is tied, return a different distinct value, e.g. -1.</span>
<span class="k">pass</span>
</pre></div>
<p>For the purposes of this article I'm not going to flesh this part out
any further, but for example code you can find one of my
implementations on <a class="reference external" href="https://github.com/jbradberry/ultimate_tictactoe/blob/master/t3/board.py">github</a>.
However, it is important to note that we will require that the
<tt class="docutils literal">state</tt> data structure is hashable and equivalent states hash to the
same value. I personally use flat tuples as my state data structures.</p>
<p>The AI class we will be constructing will support this interface:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MonteCarlo</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">board</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="c1"># Takes an instance of a Board and optionally some keyword</span>
<span class="c1"># arguments. Initializes the list of game states and the</span>
<span class="c1"># statistics tables.</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span>
<span class="c1"># Takes a game state, and appends it to the history.</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">get_play</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># Causes the AI to calculate the best move from the</span>
<span class="c1"># current game state and return it.</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">run_simulation</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># Plays out a "random" game from the current position,</span>
<span class="c1"># then updates the statistics tables with the result.</span>
<span class="k">pass</span>
</pre></div>
<p>Let's begin with the initialization and bookkeeping. The <tt class="docutils literal">board</tt>
object is what the AI will be using to obtain information about where
the game is going and what the AI is allowed to do, so we need to
store it. Additionally, we need to keep track of the state data as we
get it.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MonteCarlo</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">board</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">board</span> <span class="o">=</span> <span class="n">board</span>
<span class="bp">self</span><span class="o">.</span><span class="n">states</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
</pre></div>
<p>The UCT algorithm relies on playing out multiple games from the
current state, so let's add that next.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">datetime</span>
<span class="k">class</span> <span class="nc">MonteCarlo</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">board</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="c1"># ...</span>
<span class="n">seconds</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'time'</span><span class="p">,</span> <span class="mi">30</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">calculation_time</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="n">seconds</span><span class="p">)</span>
<span class="c1"># ...</span>
<span class="k">def</span> <span class="nf">get_play</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">begin</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span>
<span class="k">while</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">-</span> <span class="n">begin</span> <span class="o"><</span> <span class="bp">self</span><span class="o">.</span><span class="n">calculation_time</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_simulation</span><span class="p">()</span>
</pre></div>
<p>Here we've defined a configuration option for the amount of time to
spend on a calculation, and <tt class="docutils literal">get_play</tt> will repeatedly call
<tt class="docutils literal">run_simulation</tt> until that amount of time has passed. This code
won't do anything particularly useful yet, because we still haven't
defined <tt class="docutils literal">run_simulation</tt>, so let's do that now.</p>
<div class="highlight"><pre><span></span><span class="c1"># ...</span>
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">choice</span>
<span class="k">class</span> <span class="nc">MonteCarlo</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">board</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="c1"># ...</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_moves</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'max_moves'</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span>
<span class="c1"># ...</span>
<span class="k">def</span> <span class="nf">run_simulation</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">states_copy</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">[:]</span>
<span class="n">state</span> <span class="o">=</span> <span class="n">states_copy</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">xrange</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">max_moves</span><span class="p">):</span>
<span class="n">legal</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">legal_plays</span><span class="p">(</span><span class="n">states_copy</span><span class="p">)</span>
<span class="n">play</span> <span class="o">=</span> <span class="n">choice</span><span class="p">(</span><span class="n">legal</span><span class="p">)</span>
<span class="n">state</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">next_state</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">play</span><span class="p">)</span>
<span class="n">states_copy</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="n">winner</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">winner</span><span class="p">(</span><span class="n">states_copy</span><span class="p">)</span>
<span class="k">if</span> <span class="n">winner</span><span class="p">:</span>
<span class="k">break</span>
</pre></div>
<p>This adds the beginnings of the <tt class="docutils literal">run_simulation</tt> method, which
either selects a move using UCB1 or chooses a random move from the set
of legal moves each turn until the end of the game. We have also
introduced a configuration option for limiting the number of moves
forward that the AI will play.</p>
<p>You may notice at this point that we are making a copy of
<tt class="docutils literal">self.states</tt> and adding new states to it, instead of adding
directly to <tt class="docutils literal">self.states</tt>. This is because <tt class="docutils literal">self.states</tt> is the
authoritative record of what has happened so far in the game, and we
don't want to mess it up with these speculative moves from the
simulations.</p>
<p>Now we need to start keeping statistics on the game states that the AI
hits during each run of <tt class="docutils literal">run_simulation</tt>. The AI should pick the
first unknown game state it reaches to add to the tables.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MonteCarlo</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">board</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="c1"># ...</span>
<span class="bp">self</span><span class="o">.</span><span class="n">wins</span> <span class="o">=</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">plays</span> <span class="o">=</span> <span class="p">{}</span>
<span class="c1"># ...</span>
<span class="k">def</span> <span class="nf">run_simulation</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">visited_states</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
<span class="n">states_copy</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">[:]</span>
<span class="n">state</span> <span class="o">=</span> <span class="n">states_copy</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="n">player</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">current_player</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="n">expand</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">xrange</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">max_moves</span><span class="p">):</span>
<span class="n">legal</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">legal_plays</span><span class="p">(</span><span class="n">states_copy</span><span class="p">)</span>
<span class="n">play</span> <span class="o">=</span> <span class="n">choice</span><span class="p">(</span><span class="n">legal</span><span class="p">)</span>
<span class="n">state</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">next_state</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">play</span><span class="p">)</span>
<span class="n">states_copy</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="c1"># `player` here and below refers to the player</span>
<span class="c1"># who moved into that particular state.</span>
<span class="k">if</span> <span class="n">expand</span> <span class="ow">and</span> <span class="p">(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">plays</span><span class="p">:</span>
<span class="n">expand</span> <span class="o">=</span> <span class="kc">False</span>
<span class="bp">self</span><span class="o">.</span><span class="n">plays</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)]</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">wins</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)]</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">visited_states</span><span class="o">.</span><span class="n">add</span><span class="p">((</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">))</span>
<span class="n">player</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">current_player</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="n">winner</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">winner</span><span class="p">(</span><span class="n">states_copy</span><span class="p">)</span>
<span class="k">if</span> <span class="n">winner</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">for</span> <span class="n">player</span><span class="p">,</span> <span class="n">state</span> <span class="ow">in</span> <span class="n">visited_states</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">plays</span><span class="p">:</span>
<span class="k">continue</span>
<span class="bp">self</span><span class="o">.</span><span class="n">plays</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)]</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">if</span> <span class="n">player</span> <span class="o">==</span> <span class="n">winner</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">wins</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)]</span> <span class="o">+=</span> <span class="mi">1</span>
</pre></div>
<p>Here we've added two dictionaries to the AI, <tt class="docutils literal">wins</tt> and <tt class="docutils literal">plays</tt>,
which will contain the counts for every game state that is being
tracked. The <tt class="docutils literal">run_simulation</tt> method now checks to see if the
current state is the first new one it has encountered this call, and,
if not, adds the state to both <tt class="docutils literal">plays</tt> and <tt class="docutils literal">wins</tt>, setting both
values to zero. This method also adds every game state that it goes
through to a set, and at the end updates <tt class="docutils literal">plays</tt> and <tt class="docutils literal">wins</tt> with
those states in the set that are in the <tt class="docutils literal">plays</tt> and <tt class="docutils literal">wins</tt> dicts.
We are now ready to base the AI's final decision on these statistics.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">division</span>
<span class="c1"># ...</span>
<span class="k">class</span> <span class="nc">MonteCarlo</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="c1"># ...</span>
<span class="k">def</span> <span class="nf">get_play</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_depth</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">state</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="n">player</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">current_player</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="n">legal</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">legal_plays</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">[:])</span>
<span class="c1"># Bail out early if there is no real choice to be made.</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">legal</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">legal</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">return</span> <span class="n">legal</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">games</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">begin</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span>
<span class="k">while</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">-</span> <span class="n">begin</span> <span class="o"><</span> <span class="bp">self</span><span class="o">.</span><span class="n">calculation_time</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_simulation</span><span class="p">()</span>
<span class="n">games</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">moves_states</span> <span class="o">=</span> <span class="p">[(</span><span class="n">p</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">next_state</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">p</span><span class="p">))</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">legal</span><span class="p">]</span>
<span class="c1"># Display the number of calls of `run_simulation` and the</span>
<span class="c1"># time elapsed.</span>
<span class="nb">print</span> <span class="n">games</span><span class="p">,</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">-</span> <span class="n">begin</span>
<span class="c1"># Pick the move with the highest percentage of wins.</span>
<span class="n">percent_wins</span><span class="p">,</span> <span class="n">move</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span>
<span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">wins</span><span class="o">.</span><span class="n">get</span><span class="p">((</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">),</span> <span class="mi">0</span><span class="p">)</span> <span class="o">/</span>
<span class="bp">self</span><span class="o">.</span><span class="n">plays</span><span class="o">.</span><span class="n">get</span><span class="p">((</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">),</span> <span class="mi">1</span><span class="p">),</span>
<span class="n">p</span><span class="p">)</span>
<span class="k">for</span> <span class="n">p</span><span class="p">,</span> <span class="n">S</span> <span class="ow">in</span> <span class="n">moves_states</span>
<span class="p">)</span>
<span class="c1"># Display the stats for each possible play.</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span>
<span class="p">((</span><span class="mi">100</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">wins</span><span class="o">.</span><span class="n">get</span><span class="p">((</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">),</span> <span class="mi">0</span><span class="p">)</span> <span class="o">/</span>
<span class="bp">self</span><span class="o">.</span><span class="n">plays</span><span class="o">.</span><span class="n">get</span><span class="p">((</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">),</span> <span class="mi">1</span><span class="p">),</span>
<span class="bp">self</span><span class="o">.</span><span class="n">wins</span><span class="o">.</span><span class="n">get</span><span class="p">((</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">),</span> <span class="mi">0</span><span class="p">),</span>
<span class="bp">self</span><span class="o">.</span><span class="n">plays</span><span class="o">.</span><span class="n">get</span><span class="p">((</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">),</span> <span class="mi">0</span><span class="p">),</span> <span class="n">p</span><span class="p">)</span>
<span class="k">for</span> <span class="n">p</span><span class="p">,</span> <span class="n">S</span> <span class="ow">in</span> <span class="n">moves_states</span><span class="p">),</span>
<span class="n">reverse</span><span class="o">=</span><span class="kc">True</span>
<span class="p">):</span>
<span class="nb">print</span> <span class="s2">"</span><span class="si">{3}</span><span class="s2">: </span><span class="si">{0:.2f}</span><span class="s2">% (</span><span class="si">{1}</span><span class="s2"> / </span><span class="si">{2}</span><span class="s2">)"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">*</span><span class="n">x</span><span class="p">)</span>
<span class="nb">print</span> <span class="s2">"Maximum depth searched:"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_depth</span>
<span class="k">return</span> <span class="n">move</span>
</pre></div>
<p>We have added three things in this step. First, we allow <tt class="docutils literal">get_play</tt>
to return early if there are no choices or only one choice to make.
Next, we've added output of some debugging information, including the
statistics for the possible moves this turn and an attribute that
will keep track of the maximum depth searched in the selection phase
of the playouts. Finally, we've added code that picks out the move
with the highest win percentage out of the possible moves, and returns
it.</p>
<p>But we are not quite finished yet. Currently, our AI is using pure
randomness for its playouts. We need to implement UCB1 for positions
where the legal plays are all in the stats tables, so the next trial
play is based on that information.</p>
<div class="highlight"><pre><span></span><span class="c1"># ...</span>
<span class="kn">from</span> <span class="nn">math</span> <span class="kn">import</span> <span class="n">log</span><span class="p">,</span> <span class="n">sqrt</span>
<span class="k">class</span> <span class="nc">MonteCarlo</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">board</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="c1"># ...</span>
<span class="bp">self</span><span class="o">.</span><span class="n">C</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'C'</span><span class="p">,</span> <span class="mf">1.4</span><span class="p">)</span>
<span class="c1"># ...</span>
<span class="k">def</span> <span class="nf">run_simulation</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># A bit of an optimization here, so we have a local</span>
<span class="c1"># variable lookup instead of an attribute access each loop.</span>
<span class="n">plays</span><span class="p">,</span> <span class="n">wins</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">plays</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">wins</span>
<span class="n">visited_states</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
<span class="n">states_copy</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">[:]</span>
<span class="n">state</span> <span class="o">=</span> <span class="n">states_copy</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="n">player</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">current_player</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="n">expand</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_moves</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span>
<span class="n">legal</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">legal_plays</span><span class="p">(</span><span class="n">states_copy</span><span class="p">)</span>
<span class="n">moves_states</span> <span class="o">=</span> <span class="p">[(</span><span class="n">p</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">next_state</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">p</span><span class="p">))</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">legal</span><span class="p">]</span>
<span class="k">if</span> <span class="nb">all</span><span class="p">(</span><span class="n">plays</span><span class="o">.</span><span class="n">get</span><span class="p">((</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">))</span> <span class="k">for</span> <span class="n">p</span><span class="p">,</span> <span class="n">S</span> <span class="ow">in</span> <span class="n">moves_states</span><span class="p">):</span>
<span class="c1"># If we have stats on all of the legal moves here, use them.</span>
<span class="n">log_total</span> <span class="o">=</span> <span class="n">log</span><span class="p">(</span>
<span class="nb">sum</span><span class="p">(</span><span class="n">plays</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">)]</span> <span class="k">for</span> <span class="n">p</span><span class="p">,</span> <span class="n">S</span> <span class="ow">in</span> <span class="n">moves_states</span><span class="p">))</span>
<span class="n">value</span><span class="p">,</span> <span class="n">move</span><span class="p">,</span> <span class="n">state</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span>
<span class="p">((</span><span class="n">wins</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">)]</span> <span class="o">/</span> <span class="n">plays</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">)])</span> <span class="o">+</span>
<span class="bp">self</span><span class="o">.</span><span class="n">C</span> <span class="o">*</span> <span class="n">sqrt</span><span class="p">(</span><span class="n">log_total</span> <span class="o">/</span> <span class="n">plays</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">S</span><span class="p">)]),</span> <span class="n">p</span><span class="p">,</span> <span class="n">S</span><span class="p">)</span>
<span class="k">for</span> <span class="n">p</span><span class="p">,</span> <span class="n">S</span> <span class="ow">in</span> <span class="n">moves_states</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># Otherwise, just make an arbitrary decision.</span>
<span class="n">move</span><span class="p">,</span> <span class="n">state</span> <span class="o">=</span> <span class="n">choice</span><span class="p">(</span><span class="n">moves_states</span><span class="p">)</span>
<span class="n">states_copy</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="c1"># `player` here and below refers to the player</span>
<span class="c1"># who moved into that particular state.</span>
<span class="k">if</span> <span class="n">expand</span> <span class="ow">and</span> <span class="p">(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">plays</span><span class="p">:</span>
<span class="n">expand</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">plays</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)]</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">wins</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)]</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">if</span> <span class="n">t</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">max_depth</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_depth</span> <span class="o">=</span> <span class="n">t</span>
<span class="n">visited_states</span><span class="o">.</span><span class="n">add</span><span class="p">((</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">))</span>
<span class="n">player</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">current_player</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="n">winner</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">board</span><span class="o">.</span><span class="n">winner</span><span class="p">(</span><span class="n">states_copy</span><span class="p">)</span>
<span class="k">if</span> <span class="n">winner</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">for</span> <span class="n">player</span><span class="p">,</span> <span class="n">state</span> <span class="ow">in</span> <span class="n">visited_states</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">plays</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">plays</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)]</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">if</span> <span class="n">player</span> <span class="o">==</span> <span class="n">winner</span><span class="p">:</span>
<span class="n">wins</span><span class="p">[(</span><span class="n">player</span><span class="p">,</span> <span class="n">state</span><span class="p">)]</span> <span class="o">+=</span> <span class="mi">1</span>
</pre></div>
<p>The main addition here is the check to see if all of the results of
the legal moves are in the <tt class="docutils literal">plays</tt> dictionary. If they aren't
available, it defaults to the original random choice. But if the
statistics are all available, the move with the highest value
according to the confidence interval formula is chosen. This formula
adds together two parts. The first part is just the win ratio, but
the second part is a term that grows slowly as a particular move
remains neglected. Eventually, if a node with a poor win rate is
neglected long enough, it will begin to be chosen again. This term
can be tweaked using the configuration parameter <tt class="docutils literal">C</tt> added to
<tt class="docutils literal">__init__</tt> above. Larger values of <tt class="docutils literal">C</tt> will encourage more
exploration of the possibilities, and smaller values will cause the AI
to prefer concentrating on known good moves. Also note that the
<tt class="docutils literal">self.max_depth</tt> attribute from the previous code block is now
updated when a new node is added and its depth exceeds the previous
<tt class="docutils literal">self.max_depth</tt>.</p>
<p>So there we have it. If there are no mistakes, you should now have an
AI that will make reasonable decisions for a variety of board games.
I've left a suitable implementation of <tt class="docutils literal">Board</tt> as an exercise for
the reader, but one thing I've left out here is a way of actually
allowing a user to play against the AI. A toy framework for this can
be found at <a class="reference external" href="https://github.com/jbradberry/boardgame-socketserver">jbradberry/boardgame-socketserver</a> and
<a class="reference external" href="https://github.com/jbradberry/boardgame-socketplayer">jbradberry/boardgame-socketplayer</a>.</p>
<p>This version that we've just built uses light playouts. Next time,
we'll explore improving our AI by using heavy playouts, by training
some evaluation functions using machine learning techniques and
hooking in the results.</p>
<p><strong>UPDATE:</strong> The diagrams have been corrected to more accurately
reflect the possible node values.</p>
<p><strong>UPDATE:</strong> Achievement Unlocked! This article has been republished
in <a class="reference external" href="http://hackermonthly.com/issue-66.html">Hacker Monthly, Issue 66, November 2015</a>.</p>
<script type='text/javascript'>if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>