Jekyll2019-08-07T12:10:03-04:00https://sheagcraig.github.io/feed.xmlTaco DestroyerAn amazing website.Shea CraigPython List Growth Throwdown2017-04-11T11:00:00-04:002017-04-11T11:00:00-04:00https://sheagcraig.github.io/list-growth-throwdown<p><img src="/images/2017-04-11-list-growth-throwdown/face_kick.jpg" alt="Face kick" /></p>
<h3 id="inplace-list-addition-vs-list-extend">Inplace List Addition vs. List Extend</h3>
<p>I wanted to know whether it was “better” to do</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">some_list</span> <span class="o">+=</span> <span class="n">another_list</span></code></pre></figure>
<p>or</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">some_list</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">another_list</span><span class="p">)</span></code></pre></figure>
<p>So I cooked this up:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">timeit</span>
<span class="n">results</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
<span class="n">add_list</span> <span class="o">=</span> <span class="s">"["</span> <span class="o">+</span> <span class="s">", "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">i</span><span class="p">))</span> <span class="o">+</span> <span class="s">"]"</span>
<span class="n">inplace_add</span> <span class="o">=</span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s">"l += "</span> <span class="o">+</span> <span class="n">add_list</span><span class="p">,</span> <span class="n">setup</span><span class="o">=</span><span class="s">"l = ['initial_item']"</span><span class="p">,</span> <span class="n">number</span><span class="o">=</span><span class="mi">1000000</span><span class="p">)</span>
<span class="n">extend</span> <span class="o">=</span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s">"l.extend({})"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">add_list</span><span class="p">),</span> <span class="n">setup</span><span class="o">=</span><span class="s">"l = ['initial_item']"</span><span class="p">,</span> <span class="n">number</span><span class="o">=</span><span class="mi">1000000</span><span class="p">)</span>
<span class="n">results</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">inplace_add</span><span class="p">,</span> <span class="n">extend</span><span class="p">)</span></code></pre></figure>
<h3 id="winner-ish">Winner-ish</h3>
<p>The in-place addition method (theoretically called when you use the ‘+=’
operator with lists as the left and right sides) is marginally faster. But not
enough to make me really care whether to use one vs. the other.</p>
<p><img src="/images/2017-04-11-list-growth-throwdown/list_growth_mega_battle.png" alt="Science" /></p>
<h3 id="methodology">Methodology</h3>
<p>Just for safesies I tried this with a variety of right-hand list sizes, with
1-100 elements. I then timed each one 1,000,000 times. There are some strange
bumps in the graph where the extend method beats in-place addition, but I
attribute this to my computer having to work harder for the chorus of this
sweet Mastodon song I’m listening to.</p>
<h3 id="further">Further…</h3>
<p>Apparently once the lists you’re trying to add together get longer (about 14
items in the rh-side), itertools.chain is much faster, <em>for the initial
extension</em>…</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">timeit</span>
<span class="n">results</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
<span class="n">add_list</span> <span class="o">=</span> <span class="s">"["</span> <span class="o">+</span> <span class="s">", "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">i</span><span class="p">))</span> <span class="o">+</span> <span class="s">"]"</span>
<span class="n">inplace_add</span> <span class="o">=</span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s">"l += "</span> <span class="o">+</span> <span class="n">add_list</span><span class="p">,</span> <span class="n">setup</span><span class="o">=</span><span class="s">"l = ['initial_item']"</span><span class="p">,</span> <span class="n">number</span><span class="o">=</span><span class="mi">1000000</span><span class="p">)</span>
<span class="n">extend</span> <span class="o">=</span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s">"l.extend({})"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">add_list</span><span class="p">),</span> <span class="n">setup</span><span class="o">=</span><span class="s">"l = ['initial_item']"</span><span class="p">,</span> <span class="n">number</span><span class="o">=</span><span class="mi">1000000</span><span class="p">)</span>
<span class="n">chain</span> <span class="o">=</span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s">"x = itertools.chain(l, {})"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">add_list</span><span class="p">),</span> <span class="n">setup</span><span class="o">=</span><span class="s">"l = ['initial_item']"</span><span class="p">,</span> <span class="n">number</span><span class="o">=</span><span class="mi">1000000</span><span class="p">)</span>
<span class="n">results</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">inplace_add</span><span class="p">,</span> <span class="n">extend</span><span class="p">,</span> <span class="n">chain</span><span class="p">)</span></code></pre></figure>
<p><img src="/images/2017-04-11-list-growth-throwdown/part2.png" alt="Even more science" />
However, the overhead of chain for iteration means that it is
slightly slower for iteration.</p>
<h3 id="round-2-fight">Round 2… Fight!</h3>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">add_list</span> <span class="o">=</span> <span class="s">"["</span> <span class="o">+</span> <span class="s">", "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">100</span><span class="p">))</span> <span class="o">+</span> <span class="s">"]"</span>
<span class="n">chained</span> <span class="o">=</span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s">"for _ in itertools.chain(l, {}):pass"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">add_list</span><span class="p">),</span> <span class="n">setup</span><span class="o">=</span><span class="s">"l = ['initial_item']"</span><span class="p">,</span> <span class="n">number</span><span class="o">=</span><span class="mi">1000000</span><span class="p">)</span>
<span class="n">grown</span> <span class="o">=</span> <span class="n">timeit</span><span class="o">.</span><span class="n">timeit</span><span class="p">(</span><span class="s">"for _ in l + {}):pass"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">add_list</span><span class="p">),</span> <span class="n">setup</span><span class="o">=</span><span class="s">"l = ['initial_item']"</span><span class="p">,</span> <span class="n">number</span><span class="o">=</span><span class="mi">1000000</span><span class="p">)</span>
<span class="k">print</span> <span class="s">"Chained: {} Grown: {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">chained</span><span class="p">,</span> <span class="n">grown</span><span class="p">)</span>
<span class="k">print</span> <span class="s">"Chained: {:.4}s Grown: {:.4}s Ratio: {:.2</span><span class="si">%</span><span class="s">}</span><span class="si">%</span><span class="s">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">chained</span><span class="p">,</span> <span class="n">grown</span><span class="p">,</span> <span class="n">grown</span> <span class="o">/</span> <span class="n">chained</span><span class="p">)</span>
<span class="c"># Chained: 2.553s Grown: 2.272s Ratio: 88.98%%</span></code></pre></figure>
<p>So it’s marginally faster to chain two iterables if one is over about 14 items,
but then not as fast to actually iterate over them. My assumption is that (I’m
not going to test it) that as you perform list += or extend more times, the
overhead of chain dissolves into the extra cost of the list growth.</p>
<p>My recommendation? Don’t be such a nerd, and just use what you like the look of
and are willing to maintain! Using += is simple, succinct, and performs better,
and everyone who knows python will recognize what’s happening. Use extend if
you want to be marginally more self-documenting. Use chain if you’re a
greybeard who is into functional programming, or you’ve got a bunch of
generators and don’t want to dump them into RAM by converting them to lists
first.</p>
<p>But if speed <em>actually</em> matters for you, then profile it yourself!
<img src="/images/2017-04-11-list-growth-throwdown/shreddin.gif" alt="Shreddin'" /></p>Shea CraigClean up Your Filthy Repo: Part 2 (Out of date stuff)2016-09-15T11:00:00-04:002016-09-15T11:00:00-04:00https://sheagcraig.github.io/clean-up-your-filthy-repo-part-2<p><img src="/images/2016-09-15-clean-up-your-filthy-repo/36th-chamber.jpg" alt="WuTang" /></p>
<h3 id="ill-let-you-try-my-wu-tang-style">I’ll let you try my Wu-Tang Style</h3>
<p>Picking up where we left off, it’s time to do something a little more fancy.</p>
<p>All of my software patches are automated with AutoPkg. This creates, over time,
a lot of unused pkgsinfo and pkgs files in my repository. Drive space seems to
be of no real concern, however, I like to only keep things active in my
repository that I am actually using. I think this goes back to the general
philosophy mentioned in the last post of leaving something a little better than
when you checked it out. Certainly, it makes cloning the entire repository a
good deal faster if you’re managing multiple servers and provisioning new ones.</p>
<p>Spruce has a number of reports that we’ll be looking at over the next several
posts, but I think that the most immediately useful one is the “Out of Date
Items Report”. We will also talk about the corresponding <code class="highlighter-rouge">deprecate</code> verb and
it’s <code class="highlighter-rouge">--auto</code> flag.</p>
<h3 id="what-is-not-in-use">What is not in use?</h3>
<p><img src="/images/2016-09-15-clean-up-your-filthy-repo/dimmak.jpg" alt="Clan of the White Lotus" /></p>
<p>Spruce uses the same code to provide a report on out-of-date items for both the
<code class="highlighter-rouge">spruce deprecate --auto</code> and <code class="highlighter-rouge">spruce report</code> features. Before we get into
usage, let’s talk about how it works.</p>
<p>Spruce can determine which items are required by your current production
machines and help you clean up and remove out-of-date ones. This post discusses
the <code class="highlighter-rouge">auto</code> mode. If you want to pick and choose, you can generate an
out-of-date report in plist format using <code class="highlighter-rouge">spruce report -p</code>, and then edit that
plist to only remove the items you want to remove using <code class="highlighter-rouge">spruce deprecate -p</code>.
The ultimate in control.</p>
<p>When Spruce looks through your repo, it builds a dependency tree by first
looking through all of the manifests. This brings up an important point: if you
have manifests that are out-of-date, not used, or reference things you really
don’t use, you should get them out of here. That being said, the worst case
scenario is that Spruce’s out-of-date “protects” items that it need not protect
from removal.</p>
<p>Each manifest is parsed, and a set is built of items that are used for any type
of manifest item (<code class="highlighter-rouge">managed_installs</code>, <code class="highlighter-rouge">managed_uninstalls</code>,
<code class="highlighter-rouge">optional_installs</code>, etc, and even these types within conditionals).</p>
<p>Next, Spruce parses all of the pkginfo files in the repository. I made a
decision early on that this feature should only look at items with the catalog
“production”. My feeling is that this is a pretty practice to denote items that
are available for all clients. Anything not in production is by definition some
kind of “testing” item. Items in testing should be excluded from automated
cleanup until they’re either rejected as unsuitable and removed manually, or
promoted to production.</p>
<p>With all of the pkginfo information and all of the manifest information set,
Spruce builds a list of “used” items. This includes every single pkginfo that
matches a name used in a manifest. This is used later to calculate what is
out-of-date.</p>
<p>Spruce then builds the dependency tree. It starts with the first item in the
list of items used in manifests. It then looks through all items that match
that item name and adds the most recent one and a configurable number of
next-most-recent ones to a list of “current” items. (<code class="highlighter-rouge">spruce report</code> uses only
the most recent item for each dependency, <code class="highlighter-rouge">spruce deprecate --auto</code> is
configurable; I usually keep 3 deep. Also, only 3 are kept if there are 3 to
begin with!).</p>
<p>As Spruce adds in these items, it checks to see if they have any <code class="highlighter-rouge">update_for</code>
or <code class="highlighter-rouge">requires</code> dependencies. If it does, it will check resolve those
dependencies in the same manner, keeping the most recent item that resolves
that dependency. Of note, this includes handling for pinning a dependency with
a version number. For example:</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><key></span>requires<span class="nt"></key></span>
<span class="nt"><array></span>
<span class="nt"><string></span>Sal_Config-1.0.5<span class="nt"></string></span>
<span class="nt"></array></span></code></pre></figure>
<p>…will specifically select the 1.0.5 version of Sal_config to consider as
current.</p>
<p>Once all of the manifest items and dependencies have been built (including the
configurable number of “backup” versions), Spruce does set subtraction between
the “used” list and the “current” list. What is left are the out-of-date items.
This thus ignores the testing items, and the items which are not actually in
use by any manifests (there is a report for that which we’ll look at in another
post).</p>
<h3 id="enough-talk">Enough Talk!</h3>
<p><img src="/images/2016-09-15-clean-up-your-filthy-repo/Shaolin.jpg" alt="Training" /></p>
<p>To begin cleaning up out-of-date items, make sure your repo is mounted, and
then run <code class="highlighter-rouge">spruce report | more</code> and page through the results, of which there
are many, looking for the “Out of Date Items Report”. (One of the features in
the works are ways to specify specific reports to run so you don’t have to run
the entire suite every time if you’re only interested in one).</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">./spruce report
Out of Date Items Report:
This report collects all items which are <span class="k">in </span>the production catalog, but
are not the current release version. Items that have dependencies to
current releases through either the <span class="sb">`</span>requires<span class="sb">`</span> or <span class="sb">`</span>update_for<span class="sb">`</span> keys are
excluded. Items <span class="k">in </span>non-production catalogs are also excluded from
consideration by this report.
Items:
<span class="nt">--------------------</span>
name: AdobeAIR
path: /Volumes/munki_repo/pkgsinfo/Digital Media/AdobeAIR-21.0.0.215.plist
version: 21.0.0.215
size: 58.20M
<span class="nt">--------------------</span>
name: AdobeFlashPlayer
path: /Volumes/munki_repo/pkgsinfo/Digital Media/AdobeFlashPlayer-22.0.0.192.plist
version: 22.0.0.192
size: 17.74M
<span class="nt">--------------------</span>
name: AdobeFlashPlayer
path: /Volumes/munki_repo/pkgsinfo/Digital Media/AdobeFlashPlayer-21.0.0.242.plist
version: 21.0.0.242
size: 17.70M
<span class="nt">--------------------</span>
name: AdobeReaderDC
path: /Volumes/munki_repo/pkgsinfo/Productivity/AcroRdrDC_1501620039_MUI-15.016.20039.plist
version: 15.016.20039
size: 147.96M
<span class="nt">--------------------</span>
name: AdobeReaderDC
path: /Volumes/munki_repo/pkgsinfo/apps/Adobe/AdobeReaderDC-15.010.20060.plist
version: 15.010.20060
size: 142.92M
<span class="nt">--------------------</span>
name: AnyConnect
path: /Volumes/munki_repo/pkgsinfo/AnyConnect-4.2.01035.pkginfo
version: 4.2.01035
size: 12.92M
<span class="nt">--------------------</span>
name: AnyConnect
path: /Volumes/munki_repo/pkgsinfo/apps/AnyConnect-4.2.00096.pkginfo
version: 4.2.00096
size: 12.74M
<span class="nt">--------------------</span>
name: Atom
path: /Volumes/munki_repo/pkgsinfo/Productivity/Atom-1.9.7.plist
version: 1.9.7
size: 86.89M</code></pre></figure>
<p>If this all looks good, you’re ready to actually start removing things.</p>
<p>One of the nice things about Spruce is that instead of deleting, it can move
things. It can even <code class="highlighter-rouge">cp</code> and then <code class="highlighter-rouge">git rm</code> them so that you don’t have to work
so hard to maintain your version controlled repo. Spruce will create the
necessary folder structure below the folder you specify so that it matches the
repo from which the deprecated items originate.</p>
<p>The basic deprecation, saving the three most recent of any needed item, would
look like this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">spruce deprecate <span class="nt">--auto</span> 3</code></pre></figure>
<p>To move to an archive for emergency purposes:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">spruce deprecate <span class="nt">--auto</span> 3 <span class="nt">--archive</span> /Volumes/SomeBackupDrive/repo_archive</code></pre></figure>
<p>Finally, to do a git-aware deprecation:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">spruce deprecate <span class="nt">--auto</span> 3 <span class="nt">--archive</span> /Volumes/SomeBackupDrive/repo_archive <span class="nt">--git</span></code></pre></figure>
<p>The deprecate interface looks a little different. This is an interactive tool,
so there is a prompt to continue, and feedback on the success or failure of
each item. Here is a more thorough example of a deprecation:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">spruce deprecate <span class="nt">--auto</span> 3 <span class="nt">--archive</span> /Volumes/SomeBackupDrive/repo_archive <span class="nt">--git</span>
Items to be removed
<span class="nt">---------------------------------------------------------------------------</span>
<pkginfo + pkg> AdobeAIR 21.0.0.215 <span class="o">(</span>10.5.0 - None<span class="o">)</span>: 58.20M
<span class="nt">---------------------------------------------------------------------------</span>
<pkginfo + pkg> AdobeFlashPlayer 21.0.0.242 <span class="o">(</span>10.5.0 - None<span class="o">)</span>: 17.70M
<pkginfo + pkg> AdobeFlashPlayer 22.0.0.192 <span class="o">(</span>10.5.0 - None<span class="o">)</span>: 17.74M
<span class="nt">---------------------------------------------------------------------------</span>
<pkginfo + pkg> AdobeReaderDC 15.010.20060 <span class="o">(</span>10.9.0 - None<span class="o">)</span>: 142.92M
<pkginfo + pkg> AdobeReaderDC 15.016.20039 <span class="o">(</span>10.5.0 - None<span class="o">)</span>: 147.96M
<span class="nt">---------------------------------------------------------------------------</span>
<pkginfo + pkg> AnyConnect 4.2.00096 <span class="o">(</span>10.5.0 - None<span class="o">)</span>: 12.74M
<pkginfo + pkg> AnyConnect 4.2.01035 <span class="o">(</span>10.5.0 - None<span class="o">)</span>: 12.92M Requires: 1
<span class="nt">---------------------------------------------------------------------------</span>
<pkginfo + pkg> Atom 1.9.6 <span class="o">(</span>10.8 - None<span class="o">)</span>: 86.94M
<pkginfo + pkg> Atom 1.9.7 <span class="o">(</span>10.8 - None<span class="o">)</span>: 86.89M
<span class="nt">---------------------------------------------------------------------------</span>
<pkginfo + pkg> Box Sync 4.0.7693 <span class="o">(</span>10.7.0 - None<span class="o">)</span>: 14.31M
<pkginfo + pkg> Box Sync 4.0.7697 <span class="o">(</span>10.7.0 - None<span class="o">)</span>: 14.31M
<span class="nt">---------------------------------------------------------------------------</span>
<pkginfo + pkg> CitrixReceiver 12.1.0 <span class="o">(</span>10.6 - None<span class="o">)</span>: 44.49M
<span class="c"># ... Clipped for blogging clarity</span>
Items to be removed from manifests:
Are you sure you want to <span class="k">continue</span>? <span class="o">(</span>Y|N<span class="o">)</span>:
<span class="c"># ... Lots of stuff getting moved to the archive.</span></code></pre></figure>
<p>In the last example, there was a heading that read “Items to be removed from
manifests”. When we look at some of the other deprecation features, we’ll
explore how Spruce can not only remove packages and pkginfo files, but it can
also remove entries from manifests.</p>
<p>Next time we’ll talk about some of the other reports, and how to remove items
from the repo in other ways that protect you from having to do repetitive work
and making mistakes.</p>
<p>Clone or download the current <a href="https://github.com/sheagcraig/spruce-for-munki">Spruce on GitHub</a>.</p>Shea CraigClean up Your Filthy Repo: Part 1 (Names)2016-08-30T13:05:25-04:002016-08-30T13:05:25-04:00https://sheagcraig.github.io/clean-up-your-filthy-repo-part-1<p><img src="https://sheagcraig.github.io/images/2016-08-31-clean-up-your-filthy-repo/boyscout.jpg" alt="On my honor, I promise to do my best!" /></p>
<h3 id="look-at-you-youre-a-mess">Look at you. You’re a Mess</h3>
<p>I don’t know about you, but I’ve got a lot going on. I get the impression that
most Mac admins are flying solo, and there’s always more to do. Also, I like my
things to be tidy. As a child, I would sit on the floor of my room and
fantasize about the ultimate layout that would solve all of my lego-display and
8-bit gaming needs. Likewise, while it often has no visible impact on my
end-users, I’m constantly scheming about reorganizing and tidying up my digital
assets.</p>
<p>At a previous job I managed a Casper environment. Because I’ll take any excuse
to write code to solve a problem, I wrote a tool called “Spruce” to help me
with both major organizational busywork, and day-to-day structural cleanliness
auditing.</p>
<p>Now I’m using Munki to manage my fleet, and I wanted the same ability to keep
things tidy without a lot of terminal faff. So I started writing a Spruce for
Munki admins.</p>
<p>Of course, that’s a bit revisionist; what actually happened is that I wrote a
script here and there to do different things, mostly to help me gain greater
understanding and insight into the structure of the organization and its
management as I settled into my new job. At a certain point, it made sense to
share the work I had done with others and create a single unified interface for
it. Thus, Spruce.</p>
<p>Anyway, I like the boy scout approach to code espoused by Robert C. Martin:
Every time you take a look at a piece of code, return it cleaner than you found
it (Paraphrased). There are always opportunities to fix formatting, improve
style, and update or improve documentation, let alone refactoring or other
work.</p>
<p>Similarly, for long stretches of time, Munki just kind of ticks along on its
own without too much intervention. AutoPkg fills the repo with patches, I
rotate things through testing into production with another automated tool
(coming soon), and I munkiimport the occasional Xcode Beta for our developers.
But when I do get in there and start working directly with the repo, I try to
improve it and look for inconsistencies, miscategorizations, etc.</p>
<p>Spruce makes this way easier.</p>
<p>So this blog post is the first of many where we’re going to look at ways you
can use Spruce to oil, torque, and maintain the running parts of your
management tool.</p>
<h3 id="getting-spruce-getting-it-running">Getting Spruce. Getting it Running.</h3>
<p>Clone or download the current <a href="https://github.com/sheagcraig/spruce-for-munki">Spruce on GitHub</a>.</p>
<p>Enter the spruce directory and run</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./spruce name</code></pre></figure>
<p>Since this is your first time, Spruce will offer to walk you through
configuring its preferences to point to a Munki repo. It will use your
munkiimport preferences if they are in the usual place. Mount the share to your
repo, and step through the configuration prompts. When mounted, my repo is just
“/Volumes/munki_repo”.</p>
<p>You may want to put spruce in /usr/local for use later. You can alias or
link the spruce executable so it’s available from anywhere.</p>
<h3 id="getting-dirty">Getting Dirty</h3>
<p>Once you’ve configured Spruce, it will continue with the <code class="highlighter-rouge">name</code> command.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># ... snip!</span>
ADPassMonExtras
ADPassMonProfile
AdobeAIR
AdobeFlashPlayer
AdobeReader
AdobeReaderDC
AmazonEC2Tools
AnyConnect
Atom
AutoDMG
AutoPkg-Release
AutoPkgr
Box Sync
BrotherPrinterDrivers
<span class="c"># ...</span></code></pre></figure>
<p>Spruce tore through your pkginfo files and collected the <code class="highlighter-rouge">name</code> values into a
set (A set ignores duplicates). From this, we can quickly see all of the items
we offer, and look for mistakes, similar but slightly different items, etc.</p>
<p>For example, say you came across the following:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># ... snip!</span>
GitHub Desktop
Go
GoToMeeting
Google Chrome
GoogleChrome
GoogleDrive
GoogleEarth
GoogleEarthPro
GoogleTalkPlugin
<span class="c"># ...</span></code></pre></figure>
<p>Right away, my human eye, developed over millenia to spot patterns, notices the
doubled Google Chrome entry. I would want to go look and see why I have both a
“Google Chrome” and a “GoogleChrome” in the repo. Possible reasons for this
could be that my Chrome AutoPkg recipe had been overridden at one point (the
official recipe uses “GoogleChrome”), a coworker may have inadvertently added
one manually, or there could be old packages that predate AutoPkg + MunkiImport
usage.</p>
<p>Regardless of the reason, this is a potential problem. A manifest that
specifies “Google Chrome” is not going to get the same software offered that it
would if it had “GoogleChrome” specified.</p>
<p>Spruce has tools to automate standardizing the <code class="highlighter-rouge">name</code> which we’ll look at in
the future.</p>
<p>Another helpful thing that you can do is search your repo’s names. Try the
following:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./spruce name google
Google Chrome
GoogleChrome
GoogleDrive
GoogleEarth
GoogleEarthPro
GoogleTalkPlugin</code></pre></figure>
<p>Spruce does a case-insensitive search for your search term across all of the
names and reports back. I use this all the time when I can’t remember exactly
what <code class="highlighter-rouge">name</code> I use for a particular item that I want to add to a manifest. It’s
a little quicker to use than trying to remember how to do a multiline awk that
results in just the names. For example, say I did want to add a
<code class="highlighter-rouge">managed_update</code> for Google Chrome, and I couldn’t remember whether there was a
space or not in the name. Running the above command quickly answers that
question for me.</p>
<h3 id="and-now-for-my-final-trick">And now, for my final trick</h3>
<p>Add a <code class="highlighter-rouge">-v</code> and the output changes a bit. Now, Spruce lists, for each <code class="highlighter-rouge">name</code>,
the versions of that software currently in the repository:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./spruce name <span class="nt">-v</span> GoogleChrome
GoogleChrome
44.0.2403.155
44.0.2403.157
45.0.2454.85
45.0.2454.93
45.0.2454.99
45.0.2454.101
46.0.2490.71
46.0.2490.80
46.0.2490.86
47.0.2526.73
47.0.2526.80
47.0.2526.106
47.0.2526.111
48.0.2564.97
48.0.2564.103
48.0.2564.109
48.0.2564.116
49.0.2623.75
49.0.2623.87
49.0.2623.110
49.0.2623.112
50.0.2661.75
50.0.2661.94
50.0.2661.102
51.0.2704.63
51.0.2704.79
51.0.2704.84
51.0.2704.106
52.0.2743.82</code></pre></figure>
<p>The <code class="highlighter-rouge">-v</code> flag works both for global name output and searches. It provides you
with a quick way to see how many versions you’re currently archiving (and can
potentially remove). I often use it with no search term to get an overall view
of what products AutoPkg has been pumping out updates for. For example, the
above snippet shows me both that the Chrome developers have been super
release-happy <em>and</em> that I haven’t bothered to clean up my Chrome packages for
awhile.</p>
<h3 id="so-long-for-now">So long, for now</h3>
<p>Check back soon for more Spruce fun.</p>Shea CraigConfiguring and Reconfiguring the Default Mail Reader. Self Service through Munki, and a Tale of Woe2016-01-26T12:05:25-05:002016-01-26T12:05:25-05:00https://sheagcraig.github.io/configuring-and-reconfiguring-the-default-mail-reader-self-service-through-munki-and-a-tale-of-woe<p><img src="https://sheagcraig.github.io/images/apple_mail_reader/pbr.jpg" alt="Pabst Blue Ribbon!" /></p>
<h3 id="introduction">Introduction</h3>
<p>Like many institutions, our supported email client is Microsoft Outlook. Over
the years, a not insignificant number of support-cases have been initiated over
the confusing configuration of a default mail reader on OS X. Specifically,
when users click a <code class="highlighter-rouge">mailto://</code> link on a webpage, rather than opening Outlook
as they might expect, OS X opens the default mail handler, Apple Mail. The
problem becomes compounded when Apple Mail refuses to let you do anything until
you have successfully configured a mail account through its new account
wizard. Enough users have a hard time figuring out how to scroll text windows
without grabbing the side-sliders that expecting them to go rooting around in
the preferences is a tall order, especially when we can set it for them.</p>
<p>This post will cover how to set the default handlers for your client machines,
and importantly, also how to set them back again, which may look simple
after you have solved the first problem, but be wary! Looks can be deceiving!</p>
<h3 id="tldr">TL;DR</h3>
<p>You can grab all of the bits and pieces from these two gists:
<a href="https://gist.github.com/sheagcraig/16e9d6a01406de06c524">Set Microsoft Outlook as Default Handler for mailto, vcf, and ics</a>.
<a href="https://gist.github.com/sheagcraig/4fa2a11b7f1738e11c79">Munki/Outset/PyObjC Self Service Set Apple Mail as Default Handler for mailto</a>.</p>
<h3 id="heavy-support-load-the-problem">Heavy Support Load: The Problem</h3>
<p>Unlike some OS’s, where a system preference pane allows you to configure all of
the default handlers for different file types and URL schemes, Apple scatters
them throughout their bundled apps. The default mail reader is set in Mail’s
preferences; Calendar in Calendar, contacts in Contacts… etc.</p>
<p>However, this preference is not <em>set</em> in the preference domain for Mail,
Calendar, or Contacts; it shows up in LaunchServices, although setting default
handlers by writing to LaunchServices’s preference file with <code class="highlighter-rouge">defaults</code> or
editing a plist is not the supported means, nor does it seem to work correctly.</p>
<p>Apple provides a LaunchServices framework, however, for managing default
handlers for URL schemes and file extensions. This is a bit more involved than
just running a commandline utility, but it’s not terribly complicated either.</p>
<p>Therefore, savvy admins have been making using of the C program duti to
configure their client machines to use non-Apple handlers. For example, see
<a href="https://derflounder.wordpress.com/2015/12/01/setting-microsoft-outlook-as-the-default-application-for-email-contacts-and-calendars-via-automator/">Rich Trouton’s solution</a>.</p>
<p>When I began looking at configuring my fleet to use Outlook, I downloaded the
source for duti and tried to compile it myself (there are no binaries on
duti.org or GitHub). This failed, of course, since it hasn’t been updated for
10.11. This was easy to fix, but it’s when I began to smell the oncoming “This
isn’t going to be as quick as I initially thought” smell that was wafting off
of the still smoking binary I had just linked.</p>
<p>For those wanting to build duti, it’s perfectly functional to just grab the
current code and build it on a pre-10.11 machine. I think Rich told me his was
built on 10.5!</p>
<p>I wrote a three line script to use duti to set the mailto handler (a URL
scheme) to Outlook, and to also configure Outlook for ics (Calendars) and vcf
(contacts). I set up a pkginfo to deploy duti to my machines, as well as
another pkginfo to execute the above script that <code class="highlighter-rouge">requires</code> duti.</p>
<p>Without going into extensive details about the testing and release process,
this solution worked great, and I moved on to the next thing. Client machines
built and had the correct app configured for the communications tasks intended.</p>
<h3 id="grumbling-out-of-earshot-the-next-problem">Grumbling Out of Earshot: The Next Problem</h3>
<p>(UPDATE: The bug described below has been fixed by Apple)
Then I started to hear from the die-hard Apple users about how they were both
disgusted at my complicity in Microsofting their Apple, and more importantly,
their inability to undo what I had done and use Apple Mail as their default
mail reader. Apple Mail of course still worked correctly; the specific problem
was that trying to use Apple Mail’s preferences to set the default mail reader
to Apple Mail failed. After setting the default mail handler to Mail, clicking
on a “mailto” link still opened Outlook, and upon examining the setting in the
Apple Mail preferences, the setting had reverted back to Outlook.</p>
<p><img src="https://sheagcraig.github.io/images/apple_mail_reader/frank.jpg" alt="Frank" /></p>
<p>I immediately double-checked all of the configuration profiles, scripts, and
other stuff that comprises our build process. The reverting seemed to occur
about 10 seconds after setting the preference, which seems to point towards
CFPrefsd restoring something.</p>
<p>A combination of <code class="highlighter-rouge">system_profiler SPManagedClientDataType</code> and <code class="highlighter-rouge">mcxquery</code> is a
good start to getting a look at managed preferences. This, however, didn’t turn
up anything aside from the above duti script.</p>
<p>Next I took a closer look at duti’s source, to see if it was doing anything
strange or unexpected. You can see where URL schemes are set
<a href="https://github.com/moretension/duti/blob/master/handler.c#L100">here</a></p>
<p>duti makes a single LaunchServices function call to set the handler, and I
didn’t see anything else going on.</p>
<p>Next I dug into the LaunchServices
<a href="https://developer.apple.com/library/mac/documentation/Carbon/Reference/LaunchServicesReference/#//apple_ref/c/func/LSSetDefaultHandlerForURLScheme">documentation</a>.
To eliminate duti from the mix, I spun up a Python interpretor and did the
following:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="kn">import</span> <span class="nn">LaunchServices</span>
<span class="o">>>></span> <span class="n">LaunchServices</span><span class="o">.</span><span class="n">LSSetDefaultHandlerForURLScheme</span><span class="p">(</span><span class="s">"mailto"</span><span class="p">,</span> <span class="s">"com.apple.mail"</span><span class="p">)</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">LaunchServices</span><span class="o">.</span><span class="n">LSCopyDefaultHandlerForURLScheme</span><span class="p">(</span><span class="s">"mailto"</span><span class="p">)</span>
<span class="s">"com.apple.mail"</span></code></pre></figure>
<p>…and I was thinking to myself, “fine, I’ll just do it with python. And then
to be safe I repeated the last command…</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">LaunchServices</span><span class="o">.</span><span class="n">LSCopyDefaultHandlerForURLScheme</span><span class="p">(</span><span class="s">"mailto"</span><span class="p">)</span>
<span class="s">"com.microsoft.outlook"</span></code></pre></figure>
<p>It had reset on me while I was prematurely congratulating myself for a job
well-done.</p>
<p>I set up an infinite while loop to poll the mailto handler every 5 seconds and
tried a number of things. What I discovered after a number of tests was that
not only could I not set the default mail handler back to Mail in any way (duti
or the GUI), but also that I could recreate this problem without ever using
duti, or the python/PyObjC above. I was able to recreate it with <em>just</em> the
Apple Mail preferences. Sure enough, with a little bit of Googling, I found a
<a href="https://discussions.apple.com/thread/7261031?start=0&tstart=0">thread</a> where
people were complaining about this very problem.</p>
<p>Just to confirm, I threw Apple Mail into Hopper and could verify that it wasn’t
using any super-secret private methods or anything; it was using the exact same
function duti and my python were using.</p>
<h3 id="setdefaultemailhandler">setDefaultEmailHandler</h3>
<figure class="highlight"><pre><code class="language-objective-c" data-lang="objective-c">void -[DefaultApplicationPopUpButton _setDefaultEmailHandler:]](void * self, void * _cmd, void * arg2) {
rdx = arg2;
r14 = self;
r12 = _objc_msgSend;
rbx = [[self selectedItem] retain];
r15 = [[rbx representedObject] retain];
[rbx release];
if (r15 != 0x0) {
# Here it is!
rcx = LSSetDefaultHandlerForURLScheme(@"mailto", r15);
if ((rcx != 0xffffd5bb) && (rcx != 0x0)) {
rdx = rcx;
NSLog(@"Error setting %@ as the default email handler: %d", r15, rdx);
}
}
rax = (r12)(r14, @selector(indexOfSelectedItem), rdx, rcx);
(r12)(r14, @selector(setIndexOfSelectedHandler:), rax, rcx);
rdi = r15;
[rdi release];
return;
}</code></pre></figure>
<p>After a lot of fiddling around, I learned a few more things:</p>
<ul>
<li>Only the “mailto” handler seems to be an issue. The vcf and ics stuff worked
back and forth without issue.</li>
<li>Once you set a mail handler <em>once</em> through any means, that handler is now set
<em>forever</em>!</li>
<li>However, I discovered that you can set it, and then immediately log out, and
the new setting is retained. If you dilly-dally, the preference gets set
back, and you’re stuck on whatever the first changed default handler was.</li>
</ul>
<p>I have filed, and have had closed as a duplicate, a bug with Apple about this
issue, so they know about it, and it will probably go away soon. So while the
specific need to allow users to reset their mail handler to Apple Mail will go
away, I think the solution I came up with serves as a good example of how to
make use of Munki’s OnDemand feature and PyObjC to add descriptive and helpful
self service items to Managed Software Center. Casper Admins can of course
modify the following to make use of their Self Service feature without too much
work.</p>
<h3 id="how-i-did-it">How I Did It</h3>
<p>At this point I had already been setting and getting my mail handlers with
Python, so I decided to drop using duti. This saves me from an added dependency
on my client machines.</p>
<p>Then, I have two sets of pieces:
Setting Microsoft Outlook as the Default Handler for mailto, vcf, and ics:</p>
<ul>
<li>Munki pkginfo to install…
<ul>
<li>Package built with the Luggage.
<ul>
<li>Outset login-once python script to set handlers.</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>Setting Apple Mail back:</p>
<ul>
<li>Munki OnDemand pkginfo to install…
<ul>
<li>Package built with the Luggage.
<ul>
<li>Outset OnDemand python script.
<ul>
<li>Presents a popup dialog explaining what is about to happen.</li>
</ul>
</li>
</ul>
</li>
<li>Postinstall script to touch Outset’s OnDemand trigger.</li>
</ul>
</li>
</ul>
<p>The initial configuration is installed using a Configuration manifest that is
applied to all client machines. Setting the mail handler back to Apple Mail is
placed in a Self Service manifest, along with a category of Self Service, to
help users find it. It has our corporate logo, and a detailed description of
the process to supplement the dialog box that the script will present. Really,
this is the easy part.</p>
<p>Here are the interesting parts of the initial configuration. See the
<a href="https://gist.github.com/sheagcraig/16e9d6a01406de06c524">full gist</a> for all of
the pieces.</p>
<script src="https://gist.github.com/sheagcraig/16e9d6a01406de06c524.js?file=set_outlook_default_handler.py"> </script>
<script src="https://gist.github.com/sheagcraig/16e9d6a01406de06c524.js?file=set_outlook_default_handler.pkginfo"> </script>
<p>Both tasks use outset to run the script. The default handler settings
only work if they are run as the user to which the setting should apply. The
initial configuration to Outlook is done with a login-once script, which runs
the script only once per user, when they log in. The reset script uses outset’s
new on demand execution type to run immediately as the current console user.</p>
<p>Setting the mail handler back after you’ve set it the first time is a little
more involved, mostly because we don’t want to just kill loginwindow like a
savage. I had timed Alert class already lying around from <a href="https://github.com/sheagcraig/auto_logout">this
project</a>, so I just grafted it in,
even though I don’t make use of the timer.</p>
<p>The logout is executed, strangely enough, by shelling out to Applescript,
sending the «event aevtrlgo» event to loginwindow. This allows us to skip the
“Are you sure you want to quit all applications and log out now?” dialog.
However, things like active terminal sessions will still block the logout,
thus, the explanatory text requesting the user to prepare for logout. While
it’s of course very easy to just drop the user to the loginwindow, or do a
forced logout, I prefer to give the user more control of what is happening
given that this is a self service environment. The auto_logout project
mentioned earlier demonstrated that rebooting rather than a forced logout
avoided some weird graphics glitches that resulted from killing loginwindow
(which immediately gets respawned), with the same result (user ends up at the
login window). However, having to logout to set the mail handler seemed silly
enough; rebooting just seemed like a travesty.</p>
<p>As before, here are the interesting parts; see the <a href="https://gist.github.com/sheagcraig/4fa2a11b7f1738e11c79">full gist</a>
for the complete set of pieces.</p>
<script src="https://gist.github.com/sheagcraig/4fa2a11b7f1738e11c79.js?file=set_apple_mail_default_handler.py"> </script>
<script src="https://gist.github.com/sheagcraig/4fa2a11b7f1738e11c79.js?file=set_apple_mail_default_handler.pkginfo"> </script>
<script src="https://gist.github.com/sheagcraig/4fa2a11b7f1738e11c79.js?file=postinstall"> </script>
<p>Hopefully this problem will go away (UPDATE: It did) <em>and</em> very few users will
even need it in the first place. However, it was a good opportunity for me to
put into place these pieces to serve as an example or template for how to solve
future issues, including prompting users, running scripts in the console user’s
context, and making use of Munki and outset’s on demand features.</p>Shea CraigPutting the Hopper to Work: Broken Preferences Edition2015-08-11T15:24:05-04:002015-08-11T15:24:05-04:00https://sheagcraig.github.io/putting-the-hopper-to-work-broken-preferences-edition<p><img src="https://sheagcraig.github.io/images/2015-08-11-putting-the-hopper-to-work-broken-preferences-edition/ecf1b5c2b2d2930afe3e7285ceb0825c.jpg" alt="ecf1b5c2b2d2930afe3e7285ceb0825c" />
Just a quick one:
Today I was trying to replace a preference file that we install via a package
with a profile. It just didn’t work. Restore the plist file and it worked fine.
This mystified me, and initially I thought I was just doing something wrong.
And then I had an inkling of what could be happening.</p>
<p>I tossed the binary for the software-in-question into the Hopper dissambler and
started searching for references to CFPreferences and to the path for the plist
file. Sure enough, I was able to follow through the code and determine that the
software created a proper plist, but that it was manually creating and reading
that plist file; it didn’t use the preferences system to <em>get</em> its preferences.</p>
<p><img src="https://sheagcraig.github.io/images/2015-08-11-putting-the-hopper-to-work-broken-preferences-edition/Gotcha.png" alt="Gotcha" /></p>Shea CraigJust a quick one: Today I was trying to replace a preference file that we install via a package with a profile. It just didn’t work. Restore the plist file and it worked fine. This mystified me, and initially I thought I was just doing something wrong. And then I had an inkling of what could be happening.Now I get it… Updating Mac firmware2015-07-16T15:38:24-04:002015-07-16T15:38:24-04:00https://sheagcraig.github.io/now-i-get-it-updating-mac-firmware<p><img src="https://sheagcraig.github.io/images/2015-07-16-now-i-get-it-updating-mac-firmware/Spinal-Tap-Harry-Shearer.jpg" alt="Spinal-Tap-Harry-Shearer" /></p>
<p>Perusing <a href="https://www.afp548.com/2015/03/05/thunderstrike-need-to-know/">Allister Banks’ look into Thunderstrike
vulnerability</a>
it didn’t quite click for me at first what I was looking at.</p>
<p>The condensed version that I finally was able to tease out was that you can get
yourself into a situation where you:</p>
<ol>
<li>Have some machines with old, unpatched firmware</li>
<li>You build an image with AutoDmg or some other method to upgrade those
machines (in our case, we went from 10.9.5 to 10.10.4 during the summer
vacation)</li>
<li>The machine retains the old, unpatched firmware.</li>
</ol>
<p>The reason for this is that the FirmwareUpdate part of the combo updater isn’t
included in the images built by AutoDMG. If you instead update through the App
Store/SUS, part of the update process includes patching the firmware.</p>
<p>Upon delivering the next point update, the firmware would probably get updated.
Probably.</p>
<p>In terms of Thunderstrike vulnerability, you only need to have a newer firmware
version than as detected in this <a href="https://gist.github.com/sheagcraig/962b1ec99882b80d03dc#file-thunderstrikevulnerabilityea-py">extension
attribute</a>.</p>
<p>So the final word is, if you’re imaging machines with out-of-date firmware, and
you care about the firmware being up-to-date, AND you don’t have pending point
updates to apply, you can go grab the FirmwareUpdate.pkg package from the most
recent OS X combo updater in your SUS and install it on the affected machines.</p>
<p>You can find this update by looking in your SUS, which is of course Reposado…</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> find /path/to/reposado <span class="nt">-name</span> <span class="s2">"OSXUpdCombo*"</span></code></pre></figure>
<p>…will give you the paths to whatever OS X combo updaters you still have
kicking around. Within (probably the newest one) you’ll find the
FirmwareUpdate.pkg you seek.</p>
<p>For example, the 10.10.4 one is here:
<code class="highlighter-rouge">/reposado/html/content/downloads/02/26/031-25777/01sza4ly2cuww3yxfpsbeov51p5n3v7l87/FirmwareUpdate.pkg</code></p>
<p>The package safely does nothing if a machine is already up-to-date, or if the
package is actually too old, so there’s no harm in a speculative or
better-safe-than-sorry run (or two).</p>
<p>Hopefully this is an infrequent circumstance. And you could find yourself in a
circumstance where a firmware that ships with a machine is newer than the most
recent OS X update contains. That firmware will probably be a part of the
<em>next</em> update, but you might not want to wait.</p>Shea CraigHow I Roll2015-06-25T09:22:53-04:002015-06-25T09:22:53-04:00https://sheagcraig.github.io/how-i-roll<p>I use a mouse or trackpad all day. Don’t get me wrong, I don’t hate those
devices.</p>
<p>But I don’t like repetitive click-click-click workflows, of any kind.
Especially in a webpage that has to refresh. Too error prone, too mind-numbing!</p>
<p>Today I had to remove some computers from our Casper JSS that we sold. Yes, I
could just click each one and hit delete. Or yes, I could make the appropriate
search and then perform an “Action/Delete Comptuers”. But I didn’t write
python-jss to click around in a web form!</p>
<p>First I grabbed all of the computers from the lab we sold:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python">
<span class="n">ms503</span> <span class="o">=</span> <span class="p">[</span><span class="n">j</span><span class="o">.</span><span class="n">Computer</span><span class="p">(</span><span class="n">computer</span><span class="o">.</span><span class="nb">id</span><span class="p">)</span> <span class="k">for</span> <span class="n">computer</span> <span class="ow">in</span> <span class="n">j</span><span class="o">.</span><span class="n">Computer</span><span class="p">()</span> <span class="k">if</span> <span class="n">computer</span><span class="o">.</span><span class="n">name</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">"MS503"</span><span class="p">)]</span>
</code></pre></figure>
<p>Rather than pull the full record for each computer on the JSS, which is costly
(time-intensive), I did this preliminary list first. So, it grabbed all
computers whose name starts with “MS503”, regardless of case.</p>
<p>We already have new computers in place with duplicate names; thus, the
comprehension uses name for the name matching, but pulls the full record using
the id property. Otherwise, the JSS returns the first result with that name,
even though there may be more than one.</p>
<p><em>Then</em> I filtered the list down just to the two older-model iMacs which were in
the first list:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python">
<span class="n">old_ms503</span> <span class="o">=</span> <span class="p">[</span><span class="n">computer</span> <span class="k">for</span> <span class="n">computer</span> <span class="ow">in</span> <span class="n">ms503</span> <span class="k">if</span> <span class="n">computer</span><span class="o">.</span><span class="n">findtext</span><span class="p">(</span><span class="s">"hardware/model"</span><span class="p">)</span><span class="o">.</span><span class="n">rsplit</span><span class="p">(</span><span class="s">"("</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s">")"</span><span class="p">)</span> <span class="ow">in</span> <span class="p">(</span><span class="s">"Early 2009"</span><span class="p">,</span> <span class="s">"Mid 2007"</span><span class="p">)]</span></code></pre></figure>
<p>This is admittedly an ugly chain of rsplit/rstrip to get the model name. If it weren’t just for expediency I would probably write a regex to do the job.</p>
<p>From there, it’s a simple:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python">
<span class="k">for</span> <span class="n">computer</span> <span class="ow">in</span> <span class="n">old_ms503</span><span class="p">:</span>
<span class="n">computer</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span></code></pre></figure>
<p>So fresh and so clean!</p>Shea CraigI use a mouse or trackpad all day. Don’t get me wrong, I don’t hate those devices.Expanding Text Replacement in JSSImporter2015-04-01T14:54:02-04:002015-04-01T14:54:02-04:00https://sheagcraig.github.io/expanding-text-replacement-in-jssimporter<p>After some PR’s and ideas tossed around with other admins using JSSImporter, it
became pretty clear that there are a lot of creative uses of JSSImporter that
the original code doesn’t make very easy, or possible, without some hacking.</p>
<p>One big change that comes with JSSImporter 0.3.8 is that it allows you to use
any string-type AutoPkg environment variable in your template text
substitutions.</p>
<p>To use a variable for text substitution, just wrap it in %’s in the template
file. For example, if I wanted to include the value of AUTOPKG_VERSION as part
of a description in my policy template, I could do this (only the relevant
parts of the XML are included for brevity!)</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml">
<span class="nt"><description></span>Built with AutoPkg version %AUTOPKG_VERSION% for your pleasure!<span class="nt"></description></span></code></pre></figure>
<p>One thing that many people don’t realize is that AutoPkg doesn’t freak out if
you add arguments or input variables to your recipe that don’t “exist”. You can
use this to fill arguments to the JSSImporter as well as templates; you don’t
necessarily need an argument for every conceivable setting. For example, say I
want to have a second smart group scoped in my policy, but I want the name to
be overrideable with input variables.</p>
<p>First I would add a new input variable named something appropriately:</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml">
<span class="nt"><description></span>Built with AutoPkg version %AUTOPKG_VERSION% for your pleasure!<span class="nt"></description></span>
<span class="nt"><key></span>Input<span class="nt"></key></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>SecondGroupName<span class="nt"></key></span>
<span class="nt"><string></span>SuperTesters<span class="nt"></string></span>
...</code></pre></figure>
<p>Then, later, in the JSSImporter arguments, in the groups array, I could have my
second group use it:</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml">
<span class="nt"><key></span>groups<span class="nt"></key></span>
<span class="nt"><array></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>name<span class="nt"></key></span>
<span class="nt"><string></span>%GROUP_NAME%<span class="nt"></string></span>
<span class="nt"><key></span>smart<span class="nt"></key></span>
<span class="nt"><true></true></span>
<span class="nt"><key></span>template_path<span class="nt"></key></span>
<span class="nt"><string></span>%GROUP_TEMPLATE%<span class="nt"></string></span>
<span class="nt"></dict></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>name<span class="nt"></key></span>
<span class="nt"><string></span>%SecondGroupName%<span class="nt"></string></span>
<span class="nt"><key></span>smart<span class="nt"></key></span>
<span class="nt"><true></true></span>
<span class="nt"><key></span>template_path<span class="nt"></key></span>
<span class="nt"><string></span>%GROUP_TEMPLATE2%<span class="nt"></string></span>
<span class="nt"></dict></span>
<span class="nt"></array></span></code></pre></figure>
<p><em>and</em> I can include that somewhere in a template as well. So my policy template
could include this:</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml">
<span class="nt"><policy></span>
<span class="nt"><general></span>
<span class="nt"><name></span>Install Latest %PROD_NAME%<span class="nt"></name></span>
<span class="nt"><enabled></span>true<span class="nt"></enabled></span>
<span class="nt"><frequency></span>Ongoing<span class="nt"></frequency></span>
<span class="nt"><category></span>
<span class="nt"><name></span>%POLICY_CATEGORY%<span class="nt"></name></span>
<span class="nt"></category></span>
<span class="nt"></general></span>
<span class="nt"><scope></span>
<span class="nt"></scope></span>
<span class="nt"><package_configuration></span>
<span class="nt"></package_configuration></span>
<span class="nt"><scripts></span>
<span class="nt"></scripts></span>
<span class="nt"><self_service></span>
<span class="nt"><use_for_self_service></span>true<span class="nt"></use_for_self_service></span>
<span class="nt"><install_button_text></span>Install %VERSION%<span class="nt"></install_button_text></span>
<span class="nt"><self_service_description></span>%SELF_SERVICE_DESCRIPTION%<span class="nt"></self_service_description></span>
<span class="nt"></self_service></span>
<span class="nt"><maintenance></span>
<span class="nt"><recon></span>true<span class="nt"></recon></span>
<span class="nt"></maintenance></span>
<span class="nt"><user_interaction></span>
<span class="nt"><message_start></span>Greetings %SecondGroupName%<span class="nt"></message_start></span>
<span class="nt"><allow_users_to_defer></span>false<span class="nt"></allow_users_to_defer></span>
<span class="nt"><allow_deferral_until_utc></allow_deferral_until_utc></span>
<span class="nt"><message_finish></message_finish></span>
<span class="nt"></user_interaction></span>
<span class="nt"></policy></span></code></pre></figure>
<p>Granted, that’s kind of a contrived use, but it points to a way to manage a lot
of settings with a pretty flexible ability to later override things.</p>
<p>Which reminds me: Try to set all of the arguments to JSSImporter in your
recipes with input variables. This makes it a lot easier for other people to
then borrow your recipe and make it work in their environment. Indeed, once we
get it off the ground, this will be a style requirement for the “official” JSS
recipes repo… Stay tuned.</p>Shea CraigAfter some PR’s and ideas tossed around with other admins using JSSImporter, it became pretty clear that there are a lot of creative uses of JSSImporter that the original code doesn’t make very easy, or possible, without some hacking.Shout out!2015-03-30T08:24:35-04:002015-03-30T08:24:35-04:00https://sheagcraig.github.io/shout-out<p><img src="https://sheagcraig.github.io/images/2015-03-30-shout-out/6627494917_c093c2f68a.jpg" alt="6627494917_c093c2f68a" /></p>
<p>Check out:
<a href="https://onemoreadmin.wordpress.com/2015/03/26/using-luggage-outset-and-yo-for-awesome-user-notifications/">using-luggage-outset-and-yo-for-awesome-user-notifications</a>
for a sweet usage of yo, my recent Swift user notification app.</p>Shea CraigHow We Are Removing Adware2015-03-25T12:23:21-04:002015-03-25T12:23:21-04:00https://sheagcraig.github.io/how-we-are-removing-adware<p>Over the last few months the number of mac laptops at our organization with
adware infections has slowly gone from nonexistent, to a slow trickle, to,
prior to the amelioration I’m about to describe, about 10% of our managed macs
having some blacklisted file present.</p>
<p><strong>Update:</strong>All you reckless folks using 9.7 already, guess what? The “Execute
Command” described below doesn’t work. Stand by for a way to do this using a
script that is bulletproof.</p>
<p><strong>Update:</strong> It turns out that if you try to run yo from a Casper policy using
the Files and Processes/Execute Command task, yo will run, but never do
anything. I don’t quite grasp why this is, because it works if you login or ssh
in and do sudo jamf policy, but not if the trigger runs “naturally. Fortunately
@golby and I figured out an alternate. Use <code class="highlighter-rouge">open /Applications/Utilities/yo.app
--args -t "Adware Detected"</code> as your command (substitute your desired arguments
of course!). Updated in the post below as well.</p>
<p><strong>Update:</strong> People have been really enthusiastic about both
<a href="https://github.com/sheagcraig/yo">yo</a> and the AdwareCheck extension attribute.
So I started expanding AdwareCheck into something even better. Check out
<a href="https://github.com/sheagcraig/SavingThrow">SavingThrow</a> for more power!</p>
<p>I blogged about the first one of these adware packages that we’ve had to deal
with:
<a href="/cleaning-up-stupid-mac-malware-projectx/">here</a>.
Researching that, while fun, was also time consuming. Fortunately, Apple just
released <a href="https://support.apple.com/en-us/ht203987">an article</a> detailing files
to look for and procedures to use to remove common adware programs.</p>
<p>My todo list included several items like “write an extension attribute to
detect projectX” and “write automated removal script for projectX”. With
Apple’s KBase article, the research was all done; I just had to implement it.</p>
<p>And then, the prolific <a href="http://krypted.com/wp-content/uploads/2015/03/Allister.gif">Allister
Banks</a> taunted me
with his solution to detecting adware as an extension attribute. This was the
kick in the butt I needed.</p>
<p>So I wrote a quick one up myself that also adds in the ability to remove the
files. You can check it out
<a href="https://gist.github.com/sheagcraig/69a473f00ce434fffd5b">HERE</a></p>
<p>But that’s just the first piece of the puzzle. Now, what do we do with it? How
can you implement this procedure yourself?</p>
<p>First off, this procedure is based around us using the Casper Suite to manage
our fleet, but it is conceivable to use this method as a template for applying
to any other management system.</p>
<p>First, I added the AdwareCheckExtensionAttribute.py script to our JSS via the
Management Settings/Computer Management/Extension Attributes menu:</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Extension_Attributes.png" alt="Extension_Attributes" /></p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Edit_Extension_Attribute_AdwarePresent.png" alt="Edit_Extension_Attribute_AdwarePresent" /></p>
<p>Check the above screenshot for the correct extension attribute settings!</p>
<p>Next, I created a smart group to collect computers which had been identified as
<em>infected by dirty adware</em>:</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Screen-Shot-2015-03-25-at-11.31.28-AM.png" alt="Screen Shot 2015-03-25 at 11.31.28 AM" /></p>
<p>Notice, the criteria is that the value of AdwarePresent is <em>like</em> True. This is
because the extension attribute also reports back which specific files were
found, so it will never report back <em>exactly</em> True.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/ST-Loaner06.png" alt="ST-Loaner06" /></p>
<p>This is a nice added feature for IT; we like to be able to see what exactly the
user has acquired in their web surfing.</p>
<p>At this point, after computers start to recon, you should start to see some
results! Who has been handing out their admin privileges willy-nilly?</p>
<p>Now, on to how to excise the infection!</p>
<p>As a group, we decided to force the users to have to manually remove the adware
themselves. We felt that, unlike many of the Windows crapware that we see, our
Mac users had to actively authenticate and install these adware programs, and
this was our opportunity to do some targeted training. While we could have
automatically detected the adware and removed it without them ever knowing, we
felt like it was better to force them to become aware of the situation.</p>
<p>I have been working on another little project that displays OS X user
notifications. terminal-notifier already does this, but it does so rather
politely. I wanted a notification that wouldn’t go away until the user
interacted with it. So I wrote <a href="https://github.com/sheagcraig/yo">yo</a> in Swift.
The project includes an already built app that you can grab and install with no
screwing around. If you really really really want to use your organization’s
logo or some other icon AND ONLY that icon, yo has a README that details the
process of changing this icon and building the project.</p>
<p>The next step, then, was to craft a policy to install the yo app to
/Applications/Utilities on our fleet. I made a custom-build for our
organization that uses our logo, and deployed it across campus. Once that was
safely in place, it was time to remove some adware.</p>
<p>First, add the AdwareCheckExtensionAttribute.py file to your scripts through
Casper Admin or the Management/Computer Management/Scripts page so that it’s
available for your policy.</p>
<p>The next step was to create a Self Service policy named “Remove Adware”. Please
take a close look at the screenshots for the exact settings, and I’ll detail
the important bits below.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Edit_Policy_Remove_Adware.png" alt="Edit_Policy_Remove_Adware" /></p>
<p>Create a new policy, naming it something appropriate (Remove Adware?). Make
sure that it doesn’t trigger off of any of the general page triggers, since it
will be a self-service policy.</p>
<p>The frequency should be “Ongoing” because you want the policy to be available
as long as the user’s computer tests True for Adware.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Edit_Policy_Remove_Adware-2.png" alt="Edit_Policy_Remove_Adware 2" /></p>
<p>In the “Scripts” section, select the AdwareCheckExtensionAttribute.py script
and set the “Parameter 4” value to “ –remove”. This is how the script knows
its in removal mode vs. extension attribute mode.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Edit_Policy_Remove_Adware-4.png" alt="Edit_Policy_Remove_Adware 4" /></p>
<p>Next, add a Maintenance/Update Inventory task to the policy so that the
computer has a chance to drop out of the smart group.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Edit_Policy_Remove_Adware-3.png" alt="Edit_Policy_Remove_Adware 3" /></p>
<p>Finally, set the Maintenance/User Logged In Action to “Restart Immediately”.
Since some of the adware has multiple launchd jobs running, and it’s
complicated to remove them in the “correct” order, it’s much easier to just
force a restart on the user. (This will be addressed to the user in the Self
Service section to come…)</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Edit_Policy_Remove_Adware-5.png" alt="Edit_Policy_Remove_Adware 5" /></p>
<p>Scope the policy to the smart group you created above.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Edit_Policy_Remove_Adware-6.png" alt="Edit_Policy_Remove_Adware 6" /></p>
<p>Finally, in the Self Service tab, select “Make the policy available in Self
Service” toggle. I also set the button name and icon to be more helpful.</p>
<p>I put in a short description of what would happen, and, importantly, that it
would require a restart. I also selected the “Ensure that users view
description” to make sure that they are forced to “read” this description. I
put “read” in quotation marks because they won’t necessarily read it, but it’s
the best we can do!</p>
<p>Checking the “Feature the policy on the main page” button puts the policy front
and center for infected users. The best part is that this policy won’t show up
for computers not in the smart group, so you don’t have to worry about it
interfering with “normal” operation.</p>
<p>Once the self service policy has been created, anyone in the Adware smart group
can now remove their adware. Once all of the preceding work is done, the last
step is to notify the users that they have adware, hopefully directing them
towards Self Service.</p>
<p>Create one final policy, titled “Notify Users of Adware” or something similar.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Edit_Policy_Notify_user_of_Adware1.png" alt="Edit_Policy_Notify_user_of_Adware" /></p>
<p>Here, I selected the Recurring Check-In trigger. The notification won’t fire
off if the user is not logged in (which I could handle better in yo…), and
it also won’t work if we use the Login trigger, since it occurs before the UI
is fully set up. Trust me-recurring check-in is fine!</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Edit_Policy_Notify_user_of_Adware.png" alt="Edit_Policy_Notify_user_of_Adware" /></p>
<p>Next, set up a Maintenance/Execute Command task with the following call to yo:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> open /Applications/Utilities/yo.app <span class="nt">--args</span> <span class="nt">-t</span> <span class="s1">'Adware detected'</span> <span class="nt">-b</span> <span class="s1">'Clean'</span> <span class="nt">-n</span> <span class="s1">'Please remove with Self Service: Remove Adware.'</span> <span class="nt">-a</span> <span class="s1">'/Applications/Self Service.app'</span><span class="p">;</span>logger <span class="s1">'Sending adware notification.'</span></code></pre></figure>
<p>The <code class="highlighter-rouge">-t</code> argument is the notification’s title, the <code class="highlighter-rouge">-b</code> sets the text on the
action button, and the <code class="highlighter-rouge">-n</code> argument sets the body text on the notification.
The <code class="highlighter-rouge">-a</code> is a path to the Self Service app. This is passed to the
<code class="highlighter-rouge">/usr/bin/open</code> commandline program as an argument; in our case, it says that,
when someone clicks on the notification’s “action” button (titled “Clean” in
this example), Self Service should be opened. The extra logger command at the
end has the dual purpose of logging to the system log and ensuring that our
policy exits 0, rather than failing due to a mysterious error.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Screen-Shot-2015-04-01-at-2.11.46-PM.png" alt="Screen Shot 2015-04-01 at 2.11.46 PM" /></p>
<p>And lastly, scope the policy to our adware smart group.</p>
<p>Once you hit save, computers who have been added to the smart group after their
last recon determined that they had an adware infection will have a policy
scoped to them to pop the notification on screen.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Screen-Shot-2015-04-01-at-2.06.12-PM.png" alt="Screen Shot 2015-04-01 at 2.06.12 PM" /></p>
<p>They can then click on the clean button, authenticate Self Service, and run the
Clean Adware policy to clean and reboot their computer.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/Self_Service_and_How_We_Are_Removing_Adware___Shea_Craig.png" alt="Screenshot" /></p>
<p>This is all well and good… But to take it to the next level, maybe I’ll write
a JSS recipe so that you can AutoPkg/JSSImporter this entire procedure to your
JSS with no other work than running the recipe.</p>
<p><img src="https://sheagcraig.github.io/images/2015-03-25-how-we-are-removing-adware/tumblr_n0dspuc1Yx1trues8o1_500.gif" alt="Screenies." /></p>Shea CraigOver the last few months the number of mac laptops at our organization with adware infections has slowly gone from nonexistent, to a slow trickle, to, prior to the amelioration I’m about to describe, about 10% of our managed macs having some blacklisted file present.