{{ metadata.feeds.posts.href }}Giacomo Debidda's blog postsThings I have written about2024-02-07T21:30:00ZGiacomo Debiddagiacomo@giacomodebidda.comhttps://www.giacomodebidda.comhttps://www.giacomodebidda.com/posts/playwright-on-nixos/Playwright on NixOS2024-02-07T21:30:00Z2024-02-07T21:30:00ZHow to run Playwright e2e tests on NixOS.
<p>I have been using NixOS for a few months now, and while it’s awesome to have an immutable configuration for my laptop, it’s not always easy to setup a development environment with all the necessary dependencies.</p>
<p>In this post I will show you how to get <a href="https://playwright.dev/">Playwright</a> to work in a Node.js project.</p>
<h2 id="h-tldr" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-tldr"><span aria-hidden="true">#</span></a> TL;DR</h2>
<ol>
<li>Replace <code>playwright</code> with <code>playwright-core</code> in your npm <code>dependencies</code>.</li>
<li>Set <code>executablePath</code> in you chromium launch options and point it to where it can find a chromium binary (e.g. you can run <code>which chromium</code> to figure this out).</li>
</ol>
<h2 id="h-full-explanation" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-full-explanation"><span aria-hidden="true">#</span></a> Full explanation</h2>
<p>Let’s take this simple script <code>index.cjs</code> as an example.</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> <span class="token punctuation">{</span> chromium <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'playwright'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> <span class="token function-variable function">main</span> <span class="token operator">=</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> browser <span class="token operator">=</span> <span class="token keyword">await</span> chromium<span class="token punctuation">.</span><span class="token function">launch</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">headless</span><span class="token operator">:</span> <span class="token boolean">false</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> ctx <span class="token operator">=</span> <span class="token keyword">await</span> browser<span class="token punctuation">.</span><span class="token function">newContext</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> page <span class="token operator">=</span> <span class="token keyword">await</span> ctx<span class="token punctuation">.</span><span class="token function">newPage</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">await</span> page<span class="token punctuation">.</span><span class="token function">goto</span><span class="token punctuation">(</span><span class="token string">'https://www.reddit.com/r/NixOS/'</span><span class="token punctuation">)</span>
<span class="token keyword">await</span> browser<span class="token punctuation">.</span><span class="token function">close</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>If we try to execute the script with <code>node index.cjs</code>, we encounter an exception. It’s Playwright telling us that a bunch of shared objects are missing. To be more precise, when we are on any Linux distro it’s the <a href="https://github.com/microsoft/playwright/blob/19a4f15eb67fd82a0b78b12dd94e3564504f83f9/packages/playwright-core/src/server/registry/dependencies.ts#L188C23-L188C48"><code>validateLinuxDependencies</code></a> that throws this exception. Playwright gathers all the missing dependencies, creates an error message, and wraps it in this nicely-formatted double border box.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1707340475/shared-libraries-playwright_ujlh1x.png">https://res.cloudinary.com/jackdbd/image/upload/v1707340475/shared-libraries-playwright_ujlh1x.png</a></p>
<p>Here is the first pice of information we need to solve the puzzle: by default, Playwright uses the <strong>chromium revision it bundles with</strong> (this is also what Puppeteer does).</p>
<h3 id="h-why-bundle-a-chromium-revision%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-why-bundle-a-chromium-revision%3F"><span aria-hidden="true">#</span></a> Why bundle a chromium revision?</h3>
<p>The Playwright team (like the Puppeteer team) bundles a specific chromium revision so then they can focus on supporting only the specific chromium revision they bundle with. The chromium revision they currently target can be found in the <a href="https://github.com/microsoft/playwright/blob/19a4f15eb67fd82a0b78b12dd94e3564504f83f9/docs/src/release-notes-js.md?plain=1#L158">Playwright release notes</a>.</p>
<h3 id="h-where-can-we-find-the-browsers-used-by-playwright%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-where-can-we-find-the-browsers-used-by-playwright%3F"><span aria-hidden="true">#</span></a> Where can we find the browsers used by Playwright?</h3>
<p><a href="https://playwright.dev/docs/browsers#managing-browser-binaries">As they write in the documentation</a>, on Linux all browsers bundled with Playwright are stored in the <code>~/.cache/ms-playwright</code> directory.</p>
<h3 id="h-double-checking-the-required-shared-libraries" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-double-checking-the-required-shared-libraries"><span aria-hidden="true">#</span></a> Double checking the required shared libraries</h3>
<p>Now that we know where to find the browsers, we can double check which shared objects are required by the particular chromium revision used by Playwright:</p>
<pre class="language-sh"><code class="language-sh">ldd ~/.cache/ms-playwright/chromium-1091/chrome-linux/chrome</code></pre>
<p>On my laptop (NixOS 24.05) I got this:</p>
<pre class="language-sh"><code class="language-sh">linux-vdso.so.1 <span class="token punctuation">(</span>0x00007ffca03c5000<span class="token punctuation">)</span>
libdl.so.2 <span class="token operator">=</span><span class="token operator">></span> /nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27/lib/libdl.so.2 <span class="token punctuation">(</span>0x00007f2c90e98000<span class="token punctuation">)</span>
libpthread.so.0 <span class="token operator">=</span><span class="token operator">></span> /nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27/lib/libpthread.so.0 <span class="token punctuation">(</span>0x00007f2c90e93000<span class="token punctuation">)</span>
libgobject-2.0.so.0 <span class="token operator">=</span><span class="token operator">></span> not found
libglib-2.0.so.0 <span class="token operator">=</span><span class="token operator">></span> not found
libnss3.so <span class="token operator">=</span><span class="token operator">></span> not found
libnssutil3.so <span class="token operator">=</span><span class="token operator">></span> not found
libsmime3.so <span class="token operator">=</span><span class="token operator">></span> not found
libnspr4.so <span class="token operator">=</span><span class="token operator">></span> not found
libdbus-1.so.3 <span class="token operator">=</span><span class="token operator">></span> not found
libatk-1.0.so.0 <span class="token operator">=</span><span class="token operator">></span> not found
libatk-bridge-2.0.so.0 <span class="token operator">=</span><span class="token operator">></span> not found
libcups.so.2 <span class="token operator">=</span><span class="token operator">></span> not found
libgio-2.0.so.0 <span class="token operator">=</span><span class="token operator">></span> not found
libdrm.so.2 <span class="token operator">=</span><span class="token operator">></span> not found
libexpat.so.1 <span class="token operator">=</span><span class="token operator">></span> not found
libxcb.so.1 <span class="token operator">=</span><span class="token operator">></span> not found
libxkbcommon.so.0 <span class="token operator">=</span><span class="token operator">></span> not found
libatspi.so.0 <span class="token operator">=</span><span class="token operator">></span> not found
libX11.so.6 <span class="token operator">=</span><span class="token operator">></span> not found
libXcomposite.so.1 <span class="token operator">=</span><span class="token operator">></span> not found
libXdamage.so.1 <span class="token operator">=</span><span class="token operator">></span> not found
libXext.so.6 <span class="token operator">=</span><span class="token operator">></span> not found
libXfixes.so.3 <span class="token operator">=</span><span class="token operator">></span> not found
libXrandr.so.2 <span class="token operator">=</span><span class="token operator">></span> not found
libgbm.so.1 <span class="token operator">=</span><span class="token operator">></span> not found
libpango-1.0.so.0 <span class="token operator">=</span><span class="token operator">></span> not found
libcairo.so.2 <span class="token operator">=</span><span class="token operator">></span> not found
libasound.so.2 <span class="token operator">=</span><span class="token operator">></span> not found
libm.so.6 <span class="token operator">=</span><span class="token operator">></span> /nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27/lib/libm.so.6 <span class="token punctuation">(</span>0x00007f2c90da9000<span class="token punctuation">)</span>
libgcc_s.so.1 <span class="token operator">=</span><span class="token operator">></span> /nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc/lib/libgcc_s.so.1 <span class="token punctuation">(</span>0x00007f2c90d88000<span class="token punctuation">)</span>
libc.so.6 <span class="token operator">=</span><span class="token operator">></span> /nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27/lib/libc.so.6 <span class="token punctuation">(</span>0x00007f2c81218000<span class="token punctuation">)</span>
/lib64/ld-linux-x86-64.so.2 <span class="token operator">=</span><span class="token operator">></span> /nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27/lib64/ld-linux-x86-64.so.2 <span class="token punctuation">(</span>0x00007f2c90e9f000<span class="token punctuation">)</span></code></pre>
<p>In fact, if we set <code>PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS = true</code> in our nix shell and then try running <code>node index.cjs</code>, we bypass that nicely-formatted black box and get a more traditional stack trace. Here is the most relevant part of the entire stack trace:</p>
<pre class="language-sh"><code class="language-sh">- <span class="token operator"><</span>launched<span class="token operator">></span> <span class="token assign-left variable">pid</span><span class="token operator">=</span><span class="token number">179437</span>
- <span class="token punctuation">[</span>pid<span class="token operator">=</span><span class="token number">179437</span><span class="token punctuation">]</span><span class="token punctuation">[</span>err<span class="token punctuation">]</span> Could not start dynamically linked executable: /home/jack/.cache/ms-playwright/chromium-1091/chrome-linux/chrome
- <span class="token punctuation">[</span>pid<span class="token operator">=</span><span class="token number">179437</span><span class="token punctuation">]</span><span class="token punctuation">[</span>err<span class="token punctuation">]</span> NixOS cannot run dynamically linked executables intended <span class="token keyword">for</span> generic
- <span class="token punctuation">[</span>pid<span class="token operator">=</span><span class="token number">179437</span><span class="token punctuation">]</span><span class="token punctuation">[</span>err<span class="token punctuation">]</span> linux environments out of the box. For <span class="token function">more</span> information, see:
- <span class="token punctuation">[</span>pid<span class="token operator">=</span><span class="token number">179437</span><span class="token punctuation">]</span><span class="token punctuation">[</span>err<span class="token punctuation">]</span> https://nix.dev/permalink/stub-ld
- <span class="token punctuation">[</span>pid<span class="token operator">=</span><span class="token number">179437</span><span class="token punctuation">]</span> <span class="token operator"><</span>process did exit: <span class="token assign-left variable">exitCode</span><span class="token operator">=</span><span class="token number">127</span>, <span class="token assign-left variable">signal</span><span class="token operator">=</span>null<span class="token operator">></span>
- <span class="token punctuation">[</span>pid<span class="token operator">=</span><span class="token number">179437</span><span class="token punctuation">]</span> starting temporary directories cleanup</code></pre>
<h2 id="h-three-solutions-with-their-pros-and-cons" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-three-solutions-with-their-pros-and-cons"><span aria-hidden="true">#</span></a> Three solutions, with their pros and cons</h2>
<p>Now that we have double-checked with <code>ldd</code> what <code>validateLinuxDependencies</code> already told us, we have a few options to make Playwright (and Nix) happy. I can think of at least three:</p>
<ol>
<li>Install all shared objects in one way or another. For example, if we know that a particular nixpkgs package includes a shared object, we can declare that package in our nix shell <code>packages</code>. For example, we can declare <code>pkgs.cairo</code> and <code>pkgs.pango</code> to obtain <code>libcairo.so.2</code> and <code>libpango-1.0.so.0</code>, respectively.</li>
<li>Add <code>pkgs.playwright-driver.browsers</code> to the <code>nativeBuildInputs</code> and set <code>export PLAYWRIGHT_BROWSERS_PATH=${pkgs.playwright-driver.browsers}</code> in our <code>shellHook</code> (it doesn’t seem mandatory to set <code>export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS = true</code> in <code>shellHook</code>).</li>
<li>Tell Playwright to use a version of chromium we explicitly specify, instead of the bundled one.</li>
</ol>
<h3 id="h-option-1" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-option-1"><span aria-hidden="true">#</span></a> Option 1</h3>
<p>This option seems a bit too complicated to me. In some cases it might be easy to figure out how to get the shared object from a package (e.g. <code>pkgs.cairo</code> => <code>libcairo.so.2</code>), but others are more challenging. Also, having to list all of these packages in the nix shell is annoying.</p>
<h3 id="h-option-2" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-option-2"><span aria-hidden="true">#</span></a> Option 2</h3>
<p>This option works, but it forces us to download and compile chromium (and maybe other browsers bundled with Playwright?). Downloading chromium is not a big deal, but compiling it takes forever (I tried once on my laptop and it took an entire night).</p>
<h3 id="h-option-3" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-option-3"><span aria-hidden="true">#</span></a> Option 3</h3>
<p>This option doesn’t require us to compile chromium, as long as we already have a chromium revision on our machine. Also, in some cases we might need to rely on features not provided by the chromium revision bundled with Playwright, so we would need to use a custom chromium anyway. Let’s go for this option then!</p>
<h2 id="h-double-checking-the-chromium-revision-installed-via-nixpkgs" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-double-checking-the-chromium-revision-installed-via-nixpkgs"><span aria-hidden="true">#</span></a> Double checking the chromium revision installed via nixpkgs</h2>
<p>I use <a href="https://github.com/nix-community/home-manager">Home Manager</a>, and if I run which chromium I see that the chromium binary is at:</p>
<pre class="language-sh"><code class="language-sh">/home/jack/.nix-profile/bin/chromium</code></pre>
<p>That location is actually a symlink, and if I run <code>ls -l /home/jack/.nix-profile/bin/</code> I see that on my laptop chromium is somewhere in the nix store:</p>
<pre class="language-sh"><code class="language-sh">/nix/store/8595809xjaq1a04djljzp3r3h9ham4z4-chromium-120.0.6099.129/bin/chromium</code></pre>
<p>What’s the difference between the chromium revision used by Playwright and the chromium revision found in my nix store? The former is dynamically linked, the latter is statically linked. Let’s double check using <code>ldd</code>:</p>
<pre class="language-sh"><code class="language-sh">ldd <span class="token variable"><span class="token variable">$(</span><span class="token function">which</span> chromium<span class="token variable">)</span></span></code></pre>
<p>We get: <code>not a dynamic executable</code>.</p>
<h2 id="h-revisiting-the-js-script" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/playwright-on-nixos/#h-revisiting-the-js-script"><span aria-hidden="true">#</span></a> Revisiting the JS script</h2>
<p>Now that we have decided to <strong>not</strong> use the chromium revision bundled with Playwright, we can update the <code>index.cjs</code> script:</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> <span class="token punctuation">{</span> chromium <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'playwright-core'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> <span class="token function-variable function">main</span> <span class="token operator">=</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> browser <span class="token operator">=</span> <span class="token keyword">await</span> chromium<span class="token punctuation">.</span><span class="token function">launch</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token comment">// tip: use `which chromium` to figure out where the chromium binary is</span>
<span class="token literal-property property">executablePath</span><span class="token operator">:</span> <span class="token string">'/home/jack/.nix-profile/bin/chromium'</span><span class="token punctuation">,</span>
<span class="token literal-property property">headless</span><span class="token operator">:</span> <span class="token boolean">false</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> ctx <span class="token operator">=</span> <span class="token keyword">await</span> browser<span class="token punctuation">.</span><span class="token function">newContext</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> page <span class="token operator">=</span> <span class="token keyword">await</span> ctx<span class="token punctuation">.</span><span class="token function">newPage</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">await</span> page<span class="token punctuation">.</span><span class="token function">goto</span><span class="token punctuation">(</span><span class="token string">'https://www.reddit.com/r/NixOS/'</span><span class="token punctuation">)</span>
<span class="token keyword">await</span> browser<span class="token punctuation">.</span><span class="token function">close</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
By declaring <code>playwright-core</code> instead of <code>playwright</code> in our <code>package.json</code>, we avoid downloading the chromium revision bundled with Playwright. Another way to avoid downloading chromium is to set <code>PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1</code> (see <a href="https://github.com/microsoft/playwright/issues/2905#issuecomment-660083858">this comment</a>) before running <code>npm install</code>.</p>
</div>
</div>
<p>Finally, we can define a nix shell in this <code>flake.nix</code>:</p>
<pre class="language-nix"><code class="language-nix"><span class="token punctuation">{</span>
description <span class="token operator">=</span> <span class="token string">"Playwright demo"</span><span class="token punctuation">;</span>
inputs <span class="token operator">=</span> <span class="token punctuation">{</span>
nixpkgs<span class="token punctuation">.</span>url <span class="token operator">=</span> <span class="token string">"github:nixos/nixpkgs/nixos-23.11"</span><span class="token punctuation">;</span>
<span class="token comment"># nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span>
outputs <span class="token operator">=</span> <span class="token punctuation">{</span>
nixpkgs<span class="token punctuation">,</span>
self<span class="token punctuation">,</span>
<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span>
<span class="token punctuation">}</span> <span class="token operator">@</span> inputs<span class="token punctuation">:</span> <span class="token keyword">let</span>
overlays <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">(</span>final<span class="token punctuation">:</span> prev<span class="token punctuation">:</span> <span class="token punctuation">{</span>
nodejs <span class="token operator">=</span> prev<span class="token punctuation">.</span>nodejs_20<span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token punctuation">]</span><span class="token punctuation">;</span>
supportedSystems <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">"x86_64-linux"</span> <span class="token string">"aarch64-linux"</span> <span class="token string">"x86_64-darwin"</span> <span class="token string">"aarch64-darwin"</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
forEachSupportedSystem <span class="token operator">=</span> f<span class="token punctuation">:</span>
nixpkgs<span class="token punctuation">.</span>lib<span class="token punctuation">.</span>genAttrs supportedSystems <span class="token punctuation">(</span>system<span class="token punctuation">:</span>
f <span class="token punctuation">{</span>
pkgs <span class="token operator">=</span> <span class="token function">import</span> nixpkgs <span class="token punctuation">{</span><span class="token keyword">inherit</span> overlays system<span class="token punctuation">;</span><span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">in</span> <span class="token punctuation">{</span>
devShells <span class="token operator">=</span> forEachSupportedSystem <span class="token punctuation">(</span><span class="token punctuation">{</span>pkgs<span class="token punctuation">}</span><span class="token punctuation">:</span> <span class="token punctuation">{</span>
default <span class="token operator">=</span> pkgs<span class="token punctuation">.</span>mkShell <span class="token punctuation">{</span>
packages <span class="token operator">=</span> <span class="token keyword">with</span> pkgs<span class="token punctuation">;</span> <span class="token punctuation">[</span>nodejs<span class="token punctuation">]</span><span class="token punctuation">;</span>
shellHook <span class="token operator">=</span> <span class="token string">''
echo "welcome to this nix shell"
echo "- Node.js $(node --version)"
echo "- npm $(npm --version)"
echo "- $(chromium --version)"
''</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>We can also add an <code>.envrc</code>, whose content will be this single line:</p>
<pre class="language-txt"><code class="language-txt">use flake</code></pre>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
There is no need to set <code>PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS = true</code> in this case, since there are no dynamic dependencies to check (because the chromium binary we specified in <code>executablePath</code> is a static executable).</p>
</div>
</div>
<p>I have to admit I haven’t quite figure out how to work with Node.js projects in a <em>purely nix way</em>, so for now I create nix shells which are <strong>impure</strong> and require to install npm dependencies by manually typing <code>npm install</code> the first time.</p>
<p><em>Originally <a href="https://discourse.nixos.org/t/running-playwright-tests/25655/12">posted in the NixOS forum</a>.</em></p>
https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/Auditing an eCommerce website2023-08-16T13:05:00Z2023-08-16T13:05:00ZThis is a website audit of the webpage www.vino.com. The audit contains info about security, SEO, web performance.
<p>Slow websites leave money on the table. A slow eCommerce might face a higher customer acquisition cost and a lower average order value than a fast eCommerce.</p>
<p>In 2016, <a href="https://wpostats.com/2016/12/27/aliexpress-load-time.html">AliExpress reduced load time by 36% and saw a 10.5% increase in orders and a 27% increase in conversion for new customers</a>.</p>
<p>Speed is not everything though. A website also needs to rank well in search engines and to treat its visitors’ money and data with the utmost care.</p>
<p>In this post I will show you everything I discovered when auditing <a href="https://www.vino.com/">www.vino.com</a>, an eCommerce where you can shop for wines, beers and spirits.</p>
<h2 id="h-is-it-safe%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-is-it-safe%3F"><span aria-hidden="true">#</span></a> Is it safe?</h2>
<p>The first thing we want to know when auditing a website is whether it is safe or not to browse it. And when we say safe, we mean that it is both secure and respects its users’ privacy.</p>
<p>We can assess the security of a website in various ways, from a quick glance at the <a href="https://developer.chrome.com/docs/devtools/security/">Security tab in Chrome DevTools</a>, to a careful inspection of the output of website scanners like <a href="https://github.com/trailofbits/twa">twa</a>, to fuzzing for vulnerabilities using <a href="https://portswigger.net/burp">Burp Suite</a>.</p>
<h3 id="h-ssltls-certificate" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-ssltls-certificate"><span aria-hidden="true">#</span></a> SSL/TLS certificate</h3>
<p>We can start by visiting <a href="https://www.ssllabs.com/ssltest/">SSL Server Test</a> and assessing the grade of the SSL/TLS certificate issued for <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a>. In this case the grade is A+, the best we can hope for.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690536265/vino-com-tls-certificate_gkddfy.png">https://res.cloudinary.com/jackdbd/image/upload/v1690536265/vino-com-tls-certificate_gkddfy.png</a></p>
<p>Browsers accept three types of SSL/TLS certificates. Certificate authorities (CAs) like DigiCert validate each type of certificate to a different level of user trust. From the lowest level of trust to the highest one, there are:</p>
<ol>
<li>Domain Validation (DV)</li>
<li>Organization Validation (OV)</li>
<li>Extended Validation (EV)</li>
</ol>
<p>As we can see, this certificate is of type Extended Validation.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690536265/vino-com-tls-certificate-extended-validation_jnqjel.png">https://res.cloudinary.com/jackdbd/image/upload/v1690536265/vino-com-tls-certificate-extended-validation_jnqjel.png</a></p>
<p>We are off to a good start!</p>
<h3 id="h-strict-transport-security-hsts-and-hsts-preloading" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-strict-transport-security-hsts-and-hsts-preloading"><span aria-hidden="true">#</span></a> Strict-Transport-Security (HSTS) and HSTS preloading</h3>
<p>When users visit a web page served over HTTPS, their connection with the web server is encrypted with TLS. During the <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_handshake">TLS handshake</a>, browser and server negotiate which <a href="https://en.wikipedia.org/wiki/Cipher_suite">cipher suite</a> they should use to establish a TLS connection. The more updated the version of TLS used by the server (e.g. TLS v1.3), the more safeguarded from sniffers and man-in-the-middle attacks the website users will be.</p>
<p>However, the fact that a web page is served over HTTPS doesn’t necessarily mean that all resources included in that page will be served over HTTPS. For example, a page served over HTTPS may include images, scripts, stylesheets, or other resources served over plain HTTP. This is called <a href="https://web.dev/what-is-mixed-content/">mixed content</a> and it’s not good, because it leaves the site vulnerable to <a href="https://en.wikipedia.org/wiki/Downgrade_attack">downgrade attacks</a>.</p>
<p>Luckily, there is a way to tell browsers to <strong>automatically upgrade</strong> any HTTP request to HTTPS: the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security">Strict-Transport-Security (HSTS)</a> HTTP response header.</p>
<p>If we have a look at the <strong>Network</strong> panel in Chrome DevTools, we can see that the web page at <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> is served with a <code>Strict-Transport-Security</code> response header that instructs the browser to keep accessing the site over HTTPS for the next 63,072,000 seconds (i.e. two years).</p>
<pre class="language-text"><code class="language-text">Server: Apache
Strict-Transport-Security: max-age=63072000
Via: 1.1 google</code></pre>
<p>Even with HSTS, there is still a case where an initial, unencrypted HTTP connection to the website is possible: it’s when the user has a <strong>fresh install</strong> and connects to the website for the <strong>first time</strong>. Because of this security issue, Chromium, Edge, and Firefox maintain an <a href="https://www.chromium.org/hsts/#preloaded-hsts-sites">HSTS Preload List</a>, and allow websites to be included in this list if they submit a form on <a href="https://hstspreload.org/">hstspreload.org</a>.</p>
<p>As we can see, <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> is preloaded.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690539944/www-vino-com-preloaded_mest8e.png">https://res.cloudinary.com/jackdbd/image/upload/v1690539944/www-vino-com-preloaded_mest8e.png</a></p>
<p>However, there is a slight issue with this website. If we check the HSTS preload status of <code>vino.com</code> we see that is <strong>not</strong> preloaded. This is due to an incorrect redirect, as explained in the second error down below.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690470869/vino-com-hsts-preload_vcy5vl.png">https://res.cloudinary.com/jackdbd/image/upload/v1690470869/vino-com-hsts-preload_vcy5vl.png</a></p>
<p>Another issue with the <code>Strict-Transport-Security</code> header of this website is that it does not include the <code>includeSubdomains</code> directive. Without that, some resources of some subdomain of <code>vino.com</code> could be served over HTTP. And this leaves the website <a href="https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.html">vulnerable to cookie-related attacks</a> that would otherwise be prevented.</p>
<p>So, ideally, the server should follow <a href="https://hstspreload.org/#deployment-recommendations">the HSTS guidelines</a> and return this header:</p>
<pre class="language-text"><code class="language-text">Strict-Transport-Security: max-age=63072000; includeSubDomains; preload</code></pre>
<h3 id="h-content-security-policy-csp" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-content-security-policy-csp"><span aria-hidden="true">#</span></a> Content-Security-Policy (CSP)</h3>
<p>Even with HTTPS, a website is still vulnerable to <a href="https://owasp.org/www-community/attacks/xss/">Cross-Site Scripting (XSS)</a> attacks. For example, a malicious script (e.g. a <a href="https://developer.chrome.com/docs/extensions/mv3/content_scripts/">content script</a> of a Chrome Extension) could be injected in the page and executed by the browser.</p>
<p>Today, browsers have a really strong line of defense against this kind of attacks: the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP">Content-Security-Policy (CSP)</a> HTTP response header. This header allows a website to define a set of <strong>directives</strong> to only allow fetching specific resources, executing specific JS code, connecting to specific origins. Developers design the policy, browsers enforce it.</p>
<p>Unfortunately, a quick scan on <a href="https://securityheaders.com/">Security Headers</a> reveals that <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> does not have a CSP header.</p>
<div class="callout callout--tip">
<div class="callout__content">
<p>💡 —
Writing a good CSP header is not easy. And neither is maintaining it. That's why <a href="https://github.com/jackdbd/content-security-policy">I created a library</a> that can help you with these tasks.</p>
</div>
</div>
<h3 id="h-permissions-policy" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-permissions-policy"><span aria-hidden="true">#</span></a> Permissions-Policy</h3>
<p>Ecommerce websites typically have several third-party scripts for ads, analytics and <a href="https://en.wikipedia.org/wiki/Behavioral_retargeting">behavioral retargeting</a>. Aside from an impact on performance, these scripts can raise legitimate privacy concerns.</p>
<p>A recent initiative named <a href="https://developer.chrome.com/en/docs/privacy-sandbox/">Privacy Sandbox</a> aims to define solutions that protect people’s privacy and allow final users to opt-in specific browser features by giving an explicit permission via a prompt.</p>
<p>One of the solutions proposed by this initiative is the <a href="https://w3c.github.io/webappsec-permissions-policy/">Permissions-Policy</a> HTTP response header. This header defines a mechanism to allow a <strong>website’s developers</strong> to selectively enable and disable access to various browser features and APIs. This way a website can set boundaries to protect its users’ privacy by any third-party script.</p>
<p>For example, a website could use the <a href="https://w3c.github.io/geolocation-api/#permissions-policy">geolocation</a> directive to allow/disallow third-party scripts from interacting with the browser’s <a href="https://developer.mozilla.org/en-US/docs/Web/API/Geolocation_API">Geolocation API</a>.</p>
<p>Permissions-Policy is quite a recent header, so I am not surprised I did not find it on this website.</p>
<h3 id="h-vulnerabilities-of-frontend-javascript-libraries" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-vulnerabilities-of-frontend-javascript-libraries"><span aria-hidden="true">#</span></a> Vulnerabilities of frontend JavaScript libraries</h3>
<p>Most websites depend on some JavaScript library. Likely more than one. Modern JavaScript frameworks like <a href="https://trends.builtwith.com/javascript/React">React</a> or <a href="https://trends.builtwith.com/javascript/Vue">Vue</a> are all the rage now, but in reality most of the websites today still use <a href="https://trends.builtwith.com/javascript/jQuery">jQuery</a>. This website is one of them.</p>
<p>Using libraries we didn’t write doesn’t mean that their bugs and vulnerabilties are not ours to care about. We should never use a library without auditing it first, and without keeping it up to date. New vulnerabilites are discovered every day, and we should update our libraries to get the latest bug fixes and security patches.</p>
<blockquote>
<p>It may not be your fault, but it’s always your responsibility</p>
<p>— <a href="https://codewithoutrules.com/2017/06/27/its-always-your-fault-in-practice/">Itamar Turner-Trauring</a></p>
</blockquote>
<p>There are many ways to check the vulnerabilities of a website’s frontend JavaScript dependencies. I like to use a tiny tool called <a href="https://github.com/lirantal/is-website-vulnerable">is-website-vulnerable</a>.</p>
<p>If we run this command we can see that this website depends on an old version of jQuery and an old version of Bootstrap.</p>
<pre class="language-sh"><code class="language-sh">npx is-website-vulnerable https://www.vino.com</code></pre>
<h3 id="h-securitytxt" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-securitytxt"><span aria-hidden="true">#</span></a> security.txt</h3>
<p>Even when developers deeply care about security, they can still make mistakes and leave vulnerabilities that an attacker could exploit. It is then a good idea to leave some information for anyone that detected a vulnerability and is willing to disclose it to the website’s owner.</p>
<p>But how to do it? What should the website’s developers do? And how should the person that found the vulnerability contact them? Well, there is a standard for this, and it’s called <a href="https://securitytxt.org/">security.txt</a>.</p>
<p><code>security.txt</code> is a small text file that should be uploaded to the website’s <a href="https://en.wikipedia.org/wiki/Well-known_URI"><code>.well-known</code> directory</a>.</p>
<p>Here are a few examples of <code>security.txt</code>:</p>
<ul>
<li><a href="https://www.google.com/.well-known/security.txt">Google’s</a></li>
<li><a href="https://scotthelme.co.uk/.well-known/security.txt">Scott Helme’s</a></li>
<li><a href="https://giacomodebidda.com/.well-known/security.txt">mine</a></li>
</ul>
<p>It’s a shame most websites don’t have a <code>security.txt</code>. Unfortunately, <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> is one of them.</p>
<h3 id="h-data-privacy" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-data-privacy"><span aria-hidden="true">#</span></a> Data privacy</h3>
<p>Online businesses and advertisers would probably like to obtain as many data as possible about their users. However, there are laws in place that protect users’ privacy and limit the amount of information that can be collected.</p>
<p>Two of the most popular pieces of legislation in this matter are the <a href="https://gdpr-info.eu/">GDPR (General Data Protection Regulation)</a> in Europe and the <a href="https://theccpa.org/">CCPA (California Consumer Privacy Act)</a> in California.</p>
<p>Violating these laws can result in hefty fines. In case of the GDPR, financial penalties can reach up to 4% of the annual turnover of the non-compliant business.</p>
<p>The GDPR requires a website to obtain explicit consent <strong>before</strong> collecting personal data or setting <em>identifiers</em> (cookies, keys in localStorage, etc) that could be used to profile the users.</p>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
<a href="https://www.privacy-regulation.eu/en/recital-30-GDPR.htm">Recital 30 EU GDPR</a>: Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags. This may leave traces which, in particular when combined with unique identifiers and other information received by the servers, may be used to create profiles of the natural persons and identify them.</p>
</div>
</div>
<p>In general, websites use a banner to ask for consent. Here is the consent banner of <a href="https://www.vino.com/">www.vino.com</a>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690472594/vino-com-consent-banner_asskxl.png">https://res.cloudinary.com/jackdbd/image/upload/v1690472594/vino-com-consent-banner_asskxl.png</a></p>
<p>Any connection to a third-party server could result in a violation of the GDPR if the website didn’t obtain the user’s permission first. Even a simple <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel/preconnect">preconnect browser hint</a> like the one below might be considered a non-compliance for the GDPR, if the third-party server processes some information about the user.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>preconnect<span class="token punctuation">"</span></span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>third-party-server<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span></code></pre>
<p>For example, in 2022 a website owner was <a href="https://wptavern.com/german-court-fines-website-owner-for-violating-the-gdpr-by-using-google-hosted-fonts">fined for communicating the visitor’s IP address to Google through the use of Google Fonts</a>.</p>
<p>There are many <a href="https://www.cookieyes.com/blog/gdpr-compliance-checkers-for-your-website/">online tools that can assess a website’s GDPR compliance</a>. A couple that I particularly like are <a href="https://www.cookiemetrix.com/">CookieMetrix</a> and <a href="https://2gdpr.com/">2GDPR</a>. In particular, 2GDPR tells us that on several pages of <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> a few third-party cookies are set before clicking <code>ACCEPT</code> in the consent banner.</p>
<h2 id="h-is-it-easily-discoverable%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-is-it-easily-discoverable%3F"><span aria-hidden="true">#</span></a> Is it easily discoverable?</h2>
<p>Search engine crawlers like <a href="https://developers.google.com/search/docs/crawling-indexing/googlebot">Googlebot</a> constantly browse the internet and index web pages using an algorithm called <a href="https://en.wikipedia.org/wiki/PageRank">PageRank</a>. We don’t know the details of how this algorithm works, but we know that if we structure our web pages in a certain way, and if we include certain files to assist these crawlers in their job, they will assign our pages a better ranking, making our website easier to find by people.</p>
<h3 id="h-robotstxt" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-robotstxt"><span aria-hidden="true">#</span></a> robots.txt</h3>
<p>The most important file a website can have to help search engine crawlers in their job, is the <code>robots.txt</code>. This file should be hosted at the root of the website.</p>
<p>If we have a look at <a href="https://www.vino.com/robots.txt">https://www.vino.com/robots.txt</a>, we can see that it contains the location of a single sitemap and a few <code>Disallow</code> rules that asks web crawlers to refrain from visiting certain pages.</p>
<div class="callout callout--warn">
<div class="callout__content">
<p>⚠️ —
Every crawler might interpret the <code>robots.txt</code> file differently. For example, <a href="https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt">this is how Googlebot processes it</a>. A crawler might decide to disregard some (or all) rules completely.</p>
</div>
</div>
<h3 id="h-sitemaps" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-sitemaps"><span aria-hidden="true">#</span></a> Sitemap/s</h3>
<p>One way to tell web crawlers how a website is structured, is to provide <strong>one or more</strong> sitemaps.</p>
<p>Most websites have a single <code>sitemap.xml</code>, but news websites and huge eCommerces can have more than one. This can be either for necessary reasons (a <code>sitemap.xml</code> can contain a <a href="https://www.sitemaps.org/protocol.html">maximum of 50,000 URLs and can be up to 50MB in size</a>), or for SEO reasons (an eCommerce that has a sitemap for each one of its categories—and maybe another one for its images—could achieve a better ranking on search engines).</p>
<p>For example, <a href="https://www.repubblica.it/">la Repubblica</a> is the <a href="https://en.wikipedia.org/wiki/List_of_newspapers_in_Italy">second newspaper in Italy by circulation</a>, and its website has too many URLs to fit a single sitemap. In fact, if we have a look at <a href="https://www.repubblica.it/robots.txt">its robots.txt</a>, we see a sitemap for Rome, another one for Milan, one for Florence, etc…</p>
<pre class="language-text"><code class="language-text">Sitemap: https://bari.repubblica.it/sitemap.xml
Sitemap: https://bologna.repubblica.it/sitemap.xml
Sitemap: https://firenze.repubblica.it/sitemap.xml
Sitemap: https://genova.repubblica.it/sitemap.xml
Sitemap: https://milano.repubblica.it/sitemap.xml
Sitemap: https://napoli.repubblica.it/sitemap.xml
Sitemap: https://palermo.repubblica.it/sitemap.xml
Sitemap: https://parma.repubblica.it/sitemap.xml
Sitemap: https://roma.repubblica.it/sitemap.xml
Sitemap: https://torino.repubblica.it/sitemap.xml</code></pre>
<p><a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> has a single sitemap that can be found at <a href="http://www.vino.com/sitemap.xml">http://www.vino.com/sitemap.xml</a>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690471063/vino-com-sitemap_of6gpl.png">https://res.cloudinary.com/jackdbd/image/upload/v1690471063/vino-com-sitemap_of6gpl.png</a></p>
<p>Its size is well within the size limit of 50MB and contains ~10,000 URLs. However, the website might benefit from creating a sitemap for each category of products (e.g. a sitemap for wines, another one for beers), and one or more sitemaps <a href="https://developers.google.com/search/docs/crawling-indexing/sitemaps/image-sitemaps">just for images</a>.</p>
<h3 id="h-open-graph-protocol-meta-tags" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-open-graph-protocol-meta-tags"><span aria-hidden="true">#</span></a> Open Graph Protocol meta tags</h3>
<p>Another way to help web crawlers in their job is to add a few <a href="https://www.opengraph.xyz/">Open Graph Protocol</a> <code><meta></code> tags. These tags turn a web page into a graph that can be easily processed by parsers and understood by crawlers.</p>
<p>The four required properties for every page are:</p>
<ol>
<li><code>og:title</code> - The title of the object as it should appear within the graph.</li>
<li><code>og:type</code> - The type of the object, e.g. <code>website</code> for a home page, <code>article</code> for a blog post.</li>
<li><code>og:image</code> - An image URL which should represent the object within the graph.</li>
<li><code>og:url</code> - The <a href="https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls">canonical URL</a> of the object that will be used as its permanent ID in the graph.</li>
</ol>
<p>We can open the <strong>Elements</strong> panel in Chrome DevTools and look for these <code><meta></code> tags in the <code><head></code>, or we can use several online tools like <a href="https://www.opengraph.xyz/">this one by OpenGraph.xyz</a> to check if they are present on the page.</p>
<h3 id="h-structured-data" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-structured-data"><span aria-hidden="true">#</span></a> Structured data</h3>
<p>Somewhat similar to OGP <code><meta></code> tags, a website can add structured data to allow Google converting a web page into a <a href="https://developers.google.com/search/docs/appearance/structured-data/search-gallery">rich result</a>. These data can be expressed in different formats. Google Search supports <a href="https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data#supported-formats">JSON-LD, Microdata, RDFa</a>.</p>
<p>Websites with many product pages might benefit greatly from including structured data in their markup. In an <a href="https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data#why">example reported by Google</a>, <a href="https://www.rottentomatoes.com/">Rotten Tomatoes</a> added structured data to 100,000 unique pages and measured a 25% higher click-through rate for pages enhanced with structured data, compared to pages without structured data.</p>
<p>If we use <a href="https://search.google.com/test/rich-results">Rich Results Test</a> to check for structured data, we see that the home page of <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> doesn’t have any, while <a href="https://www.vino.com/dettaglio/brunello-di-montalcino-docg-banfi-2017.html">this product page</a> does.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690642252/vino-com-product-page-structured-data_i2viot.png">https://res.cloudinary.com/jackdbd/image/upload/v1690642252/vino-com-product-page-structured-data_i2viot.png</a></p>
<h2 id="h-how-fast-is-it-really%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-how-fast-is-it-really%3F"><span aria-hidden="true">#</span></a> How fast is it, really?</h2>
<p>When we want to know how fast a website is, we can either <strong>simulate</strong> a few website visits, or analyze historical data of <strong>real</strong> visits.</p>
<p>A simulation can be very close to reality. For example, we can <a href="https://docs.webpagetest.org/private-instances/mobile-devices/">install the WebPageTest Mobile Agent on a real device</a> (e.g. a Moto G4) and visit the website using a real browser. But even in this case, it’s just <em>one</em> visit. Users can have weird combinations of browser, device, viewport size and network connectivity. Simulating all of them is not feasible.</p>
<p>Do we have more luck with historical data? Yes, we do. If a user visits a website using Chrome, the browser records the most important metrics she experiences during her visit, and sends them to the <a href="https://developers.google.com/web/tools/chrome-user-experience-report">Chrome User Experience Report (CrUX)</a>.</p>
<p>It’s impossible to obtain the entire <em>population</em> of real visits to a website. But thanks to CrUX we have access to a reasonably large <em>sample</em> of them.</p>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
<a href="https://caniuse.com/usage-table">Most desktop and mobile users browse the internet using Chrome</a>, maybe save for mobile users in the US, where <a href="https://gs.statcounter.com/browser-market-share/mobile/united-states-of-america">the market leader seems to be Safari</a>.</p>
</div>
</div>
<p>Since CrUX data tell us a story about real experiences from real users, we call them <strong>field performance data</strong>.</p>
<p>Among the metrics collected in CrUX, Barry Pollard says that we should start any performance analysis from the ones called <a href="https://web.dev/vitals/">Core Web Vitals (CWV)</a>.</p>
<p><a href="https://twitter.com/tunetheweb/status/1661699182657241090">https://twitter.com/tunetheweb/status/1661699182657241090</a></p>
<h3 id="h-where-do-we-get-field-performance-data%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-where-do-we-get-field-performance-data%3F"><span aria-hidden="true">#</span></a> Where do we get field performance data?</h3>
<p>We can obtain field performance data from the <a href="https://developer.chrome.com/docs/crux/bigquery/">CrUX BigQuery dataset</a>, the <a href="https://developer.chrome.com/docs/crux/api/">CrUX API</a> and the <a href="https://developer.chrome.com/docs/crux/history-api/">CrUX History API</a>. These data sources follow different schemas and have a different level of granularity. For each query we have in mind, one data source might be more appropriate than another one.</p>
<h3 id="h-traffic-by-device-and-connectivity-grouped-by-country" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-traffic-by-device-and-connectivity-grouped-by-country"><span aria-hidden="true">#</span></a> Traffic by device and connectivity, grouped by country</h3>
<p>An important question we want to answer is: “where do our users come from?”. Since we don’t have access to the analytics of <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a>, we can use CrUX to get an idea of the popularity of the website in different countries. CrUX contains anonymized data, but it keeps the information about the country where the visit originated.</p>
<p>Unsurprisingly, <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> is most popular in Italy. According to the data from CrUX, it seems much more users visited the website using phones rather than desktops or tablets. Almost 74% of the traffic that originated from Italy in the trimester from April to June 2023 came from phones. 96% of these connections were <strong>4G or better</strong>.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Traffic by connectivity and device, grouped by country (first 5 results)</caption>
<thead class="">
<tr class=""><th scope="col" class="">country</th><th scope="col" class="">coarse popularity</th><th scope="col" class="">desktop traffic</th><th scope="col" class="">phone traffic</th><th scope="col" class="">tablet traffic</th><th scope="col" class="">4G percentage</th><th scope="col" class="">3G percentage</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">Italy</td><td class="">5000</td><td class="">23.60</td><td class="">73.82</td><td class="">1.58</td><td class="">96.13</td><td class="">2.87</td></tr><tr class=""><td class="">Belgium</td><td class="">36667</td><td class="">26.90</td><td class="">68.24</td><td class="">3.93</td><td class="">97.52</td><td class="">1.49</td></tr><tr class=""><td class="">Luxembourg</td><td class="">36667</td><td class="">0</td><td class="">33</td><td class="">0</td><td class="">33</td><td class="">0</td></tr><tr class=""><td class="">Sweden</td><td class="">50000</td><td class="">20.82</td><td class="">78.14</td><td class="">0</td><td class="">98.97</td><td class="">0</td></tr><tr class=""><td class="">Germany</td><td class="">50000</td><td class="">17.29</td><td class="">77.72</td><td class="">4.06</td><td class="">97.25</td><td class="">1.82</td></tr>
</tbody>
</table>
</div>
</div>
<div class="callout callout--warn">
<div class="callout__content">
<p>⚠️ —
A phone connected to a WiFi network is still a phone, but has a much better connectivity than the same phone connected to the a cell network. Don't forget this when interpreting that <strong>4G percentage</strong> in the table.</p>
</div>
</div>
<h3 id="h-performance-metrics-over-time" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-performance-metrics-over-time"><span aria-hidden="true">#</span></a> Performance metrics over time</h3>
<p>Knowing which countries and devices the traffic comes from is key. It suggests us to focus on metrics recorded for this combination of factors. It would be pointless to look at the performance of <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> on tablets in the UK, if we know that the website is mostly visited from phones, from Italy.</p>
<h4 id="h-ttfb" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-ttfb"><span aria-hidden="true">#</span></a> TTFB</h4>
<p>I agree with Barry that Core Web Vitals are really important, but there is another metric that is absolutely crucial for web performance: Time To First Byte (TTFB).</p>
<p>The thing is, we can’t really <em>optimize for</em> TTFB, because TTFB is the sum of the many things:</p>
<ul>
<li>HTTP redirects</li>
<li>Service worker’s startup time</li>
<li>DNS lookup</li>
<li>Connection and TLS negotiation</li>
<li>Request, up until the point at which the first byte of the response has arrived</li>
</ul>
<p>Also, is <a href="https://web.dev/ttfb/#what-is-a-good-ttfb-score">not always appropriate to compare</a> the TTFB of a website built with one technology (e.g. client-side rendering), to the TTFB of another website built with a different technology (e.g. server-side rendering).</p>
<p>But there is one thing we can say by looking at the value of TTFB: if it’s bad, we know we have some work to do.</p>
<blockquote>
<p>While a good TTFB doesn’t necessarily mean you will have a fast website, a bad TTFB almost certainly guarantees a slow one.</p>
<p>— <a href="https://csswizardry.com/2019/08/time-to-first-byte-what-it-is-and-why-it-matters/">Harry Roberts</a></p>
</blockquote>
<p>I created <a href="https://github.com/jackdbd/performance-audit">a tool</a> that allows me to obtain data from CrUX (both the CrUX BigQuery dataset and the CrUX History API) without ever leaving Google Sheets. Let’s start with TTFB.</p>
<p>Six months of historical field performance data show that TTFB has been good for over 75% of the phone users.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690471378/vino-com-ttfb_jup4x0.png">https://res.cloudinary.com/jackdbd/image/upload/v1690471378/vino-com-ttfb_jup4x0.png</a></p>
<p>In the timeline chart below we can see that even if TTFB started degrading a bit after mid-June 2023, its 75th percentile value is still below the threshold of what Google defines as <strong>good TTFB: 800ms</strong>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690471769/vino-com-ttfb-75th-percentile_eyrkxy.png">https://res.cloudinary.com/jackdbd/image/upload/v1690471769/vino-com-ttfb-75th-percentile_eyrkxy.png</a></p>
<p>It would be interesting to know why the website reached its best TTFB in early June and then started to worsen. But it’s hard to tell without having more information about the infrastructure the website is built on, and in general where the backend spends its time (e.g. cache hit/miss ratio on the CDN, CPU utilization and memory footprint on the server, etc).</p>
<p>Had CrUX reported a poor TTFB for the website, I would have suggested using the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing">Server-Timing</a> HTTP response header to understand where the backend spends its time.</p>
<h4 id="h-fcp" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-fcp"><span aria-hidden="true">#</span></a> FCP</h4>
<p>The first thing a user sees after having requested a web page, is a blank screen. Until the browser renders <em>something</em> on the page, the user doesn’t know if the page is really loading, if there are connection issues, or simply if the website is broken. There are basically two interpretations about this <em>something</em>, hence two very similar metrics: <strong>Start Render</strong> and <strong>First Contentful Paint (FCP)</strong>.</p>
<p>There are some differences in what Start Render and FCP consider <strong>DOM content</strong>, but the important thing is that they basically mark the end of the blank screen.</p>
<p>The best tool that shows when FCP occurs is the <strong>Details</strong> section of WebPageTest. For example, down below we can see that the FCP element is a small gray <code><div></code> that belong to the website banner and appears at ~2.3 seconds (the rest of the banner appears much later).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690897816/vino-com-FCP_n07rqm.png">https://res.cloudinary.com/jackdbd/image/upload/v1690897816/vino-com-FCP_n07rqm.png</a></p>
<p>The value of FCP here (2.266 seconds) is actually worse than the 75th percentile of FCP of the last six months of data extracted from CrUX. I think this is due to an excessive network throttling performed by the WebPageTest configuration profile I chose. I instructed WebPageTest to run this audit using a server in Milan, simulating a Google Pixel 2 XL with a 4G connection.</p>
<p>Here is the timeline of FCP experienced by real users over a period of six months.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690900215/vino-com-FCP-p75_einqhi.png">https://res.cloudinary.com/jackdbd/image/upload/v1690900215/vino-com-FCP-p75_einqhi.png</a></p>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
Several articles about web performance often run a performance audit on a Moto G4. I argue that nowadays most users have a more powerful device. Google Pixel 2 XL was once a really good smartphone, but in 2023 I would consider it low-mid tier, so a good option for performance audits. Probably it's even a bit too conservative option, as we have just saw.</p>
</div>
</div>
<h4 id="h-lcp" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-lcp"><span aria-hidden="true">#</span></a> LCP</h4>
<p>Rendering <strong>some</strong> DOM content as soon as possible is important. But arguably more important is to render the <strong>largest</strong> DOM content in the viewport.</p>
<p>When a user sees the biggest element on the screen, she <strong>perceives</strong> that the page loads fast and it is <strong>useful</strong>. The metric that tracks this moment is called Largest Contentful Paint (LCP).</p>
<p>Down below you can see six months of data of FCP vs LCP extracted from CrUX.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690902348/vino-com-fcp-vs-lcp_jwmyms.png">https://res.cloudinary.com/jackdbd/image/upload/v1690902348/vino-com-fcp-vs-lcp_jwmyms.png</a></p>
<p>The best tool that shows when LCP occurs is the <strong>Performance Insights</strong> panel of Chrome DevTools. As we can see, in this case the LCP element is one of the product images.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690895982/vino-com-LCP-element_varsdn.png">https://res.cloudinary.com/jackdbd/image/upload/v1690895982/vino-com-LCP-element_varsdn.png</a></p>
<p>We can also understand what the LCP element is, using another tool: <a href="https://requestmap.pages.dev/">Request Mapper</a>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690969812/vino-com-lcp-request-mapper_mwbod1.png">https://res.cloudinary.com/jackdbd/image/upload/v1690969812/vino-com-lcp-request-mapper_mwbod1.png</a></p>
<h4 id="h-cls" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-cls"><span aria-hidden="true">#</span></a> CLS</h4>
<p>When a browser renders a page, it has to figure out where to place each element. Ideally, once it has established where to position the elements, we shouldn’t cause it to rearrange them. Failing to do so might be ok and might even be required for a particular page, but these adjustments add up and can cause a bad user experience.</p>
<p>Cumulative Layout Shift is a metric that take into account such rearrangements in the page (layout shifts). Its definition is <a href="https://web.dev/cls/#what-is-cls">quite complex</a>, but the idea is simple: to offer a good user experience, a web page needs to have a good <strong>layout stability</strong>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1690877826/vino-com-load-page-google-pixel2-xl_jobuhd.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1690877826/vino-com-load-page-google-pixel2-xl_jobuhd.mp4</a></p>
<div class="callout callout--tip">
<div class="callout__content">
<p>💡 —
You can click/tap to pause this video and use the horizontal scrollbar or the left/right arrow keys. Try to pause at 3.4s to observe a <a href="https://fonts.google.com/knowledge/glossary/fout">Flash Of Unstyled Text (FOUT)</a>.</p>
</div>
</div>
<p>The <strong>Web Vitals</strong> section of a WebPageTest audit highlights in red all layout shifts occurring in two adjacent frames.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1690908681/vino-com-CLS_df5yft.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1690908681/vino-com-CLS_df5yft.mp4</a></p>
<p>I ran the test on WebPageTest using this script, to simulate a user that accepts the cookies in the consent banner.</p>
<pre class="language-text"><code class="language-text">setEventName loadPage
navigate %URL%
setEventName acceptCookies
execAndWait document.querySelector('#info-cookies button').click()</code></pre>
<p>We can see that the banner disappears, but there is no other effect on the page layout.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1690877826/vino-com-after-accepted-cookies-google-piexl2-xl_sxbejj.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1690877826/vino-com-after-accepted-cookies-google-piexl2-xl_sxbejj.mp4</a></p>
<p>By the look of these videos, we can say the layout is very stable, so we expect CLS to be quite good. In fact, the last six months of field performance show that CLS has indeed been very good on phones (and also on desktops and tablets, not shown here).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690471938/vino-com-cls_bzvfho.png">https://res.cloudinary.com/jackdbd/image/upload/v1690471938/vino-com-cls_bzvfho.png</a></p>
<p>Again, we notice a slight degradation starting from June 2023, but the 75th percentile value of CLS is still well below the threshold of what Google defines as <strong>good CLS: 0.1</strong>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690471946/vino-com-cls-75th-percentile_vyyro5.png">https://res.cloudinary.com/jackdbd/image/upload/v1690471946/vino-com-cls-75th-percentile_vyyro5.png</a></p>
<h2 id="h-how-is-it-made%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-how-is-it-made%3F"><span aria-hidden="true">#</span></a> How is it made?</h2>
<p>Now that we have seen how the home page of <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> performs in the real world, we can have a look at how it is made. That is, we want to know:</p>
<ul>
<li>which assets are on this page</li>
<li>where they are hosted</li>
<li>when they are requested</li>
<li>who requests them</li>
</ul>
<p>To answer these questions we are going to use a variety of tools: Chrome DevTools, Request Mapper, <a href="https://redbot.org/">REDbot</a>, WebPageTest. But first we need to find a way to dismiss the consent banner programmatically, because we need to analyze the HTTP requests before and after a user expresses her consent.</p>
<h3 id="h-before-and-after-consent" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-before-and-after-consent"><span aria-hidden="true">#</span></a> Before and after consent</h3>
<p>Even <strong>before</strong> expressing our consent, the website sets a few cookies. We can inspect them in Chrome DevTools > <strong>Application</strong> > <strong>Cookies</strong>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690472989/vino-com-cookies-before-consent_vpo8q9.png">https://res.cloudinary.com/jackdbd/image/upload/v1690472989/vino-com-cookies-before-consent_vpo8q9.png</a></p>
<p>This is not great. In particular, that <code>cto_bundle</code> is a third-party cookie set by <a href="https://www.criteo.com/">criteo</a>, a service that performs <a href="https://en.wikipedia.org/wiki/Behavioral_retargeting">behavioral retargeting</a>. That <code>_gcl_au</code> is a cookie set by <a href="https://business.safety.google/adscookies/">Google AdWords</a>. And those cookies that start with <code>_ga</code> are set by <a href="https://support.google.com/analytics/answer/11397207?hl=en">Google Analytics</a>.</p>
<p>I don’t think that setting any of those cookies before asking for consent is going to fly with the GDPR.</p>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
<a href="https://www.privacy-regulation.eu/en/recital-24-GDPR.htm">Recital 24 EU GDPR</a>: […] In order to determine whether a processing activity can be considered to monitor the behaviour of data subjects, it should be ascertained whether natural persons are tracked on the internet including potential subsequent use of personal data processing techniques which consist of profiling a natural person, particularly in order to take decisions concerning her or him or for analysing or predicting her or his personal preferences, behaviours and attitudes.</p>
</div>
</div>
<p>But let’s proceed and see what happens after we click <code>ACCEPT</code> in the consent banner and allow the website to process the cookies as stated in its <a href="https://www.vino.com/en/content/cookie">cookie policy</a>. If we type this code in the browser’s console, the consent banner disappears.</p>
<pre class="language-js"><code class="language-js">document<span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token string">'#info-cookies button'</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">click</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>We can see that a new <code>acceptCookies</code> cookie appears, and its value is set to <code>true</code>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690472989/vino-com-cookies-after-consent_lyrmfj.png">https://res.cloudinary.com/jackdbd/image/upload/v1690472989/vino-com-cookies-after-consent_lyrmfj.png</a></p>
<h3 id="h-visualize-http-requests" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-visualize-http-requests"><span aria-hidden="true">#</span></a> Visualize HTTP requests</h3>
<p>To give actionable advice on how to improve the web performance of a page, we need to examine the HTTP requests of that page.</p>
<p>We could study each HTTP request in detail using <a href="https://developer.chrome.com/docs/devtools/network/reference/#waterfall">the Waterfall in Chrome DevTools</a> or <a href="https://nooshu.com/blog/2019/10/02/how-to-read-a-wpt-waterfall-chart/">the one in WebPageTest</a>, but I think it’s better to have first an overview of how the browser makes these requests. The best tool for this job is Request Mapper.</p>
<h4 id="h-before-accepting-the-cookies" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-before-accepting-the-cookies"><span aria-hidden="true">#</span></a> Before accepting the cookies</h4>
<p>If we plot the request map of <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/www.vino.com">www.vino.com</a> we see a bunch of small <b><span style="color:#a565dc">images</span></b>, some <b><span style="color:#e15040">fonts</span></b>, a couple of <b><span style="color:#8fe064">CSS</span></b> files, and a few, big <b><span style="color:#e7942b">JavaScript</span></b> files.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690907551/vino-com-request-mapper-overview_pglykp.png">https://res.cloudinary.com/jackdbd/image/upload/v1690907551/vino-com-request-mapper-overview_pglykp.png</a></p>
<p>We also notice <strong>four clusters</strong> of requests. The center of each cluster is important because it’s both the origin and the bottleneck for the requests of that cluster.</p>
<p>The browser starts parsing the file that sits at the center of the cluster, then gradually discovers other resources to fetch. We see arrows coming out from the main <b><span style="color:#71abfb">HTML</span></b> document and requesting <b><span style="color:#e7942b">JavaScript</span></b> and <b><span style="color:#8fe064">CSS</span></b> assets first, and <b><span style="color:#a565dc">images</span></b> and <b><span style="color:#e15040">fonts</span></b> later.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1690971131/vino-com-request-map_rbmobd.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1690971131/vino-com-request-map_rbmobd.mp4</a></p>
<p>The green circle in the <strong>center-left cluster</strong> is the single <b><span style="color:#8fe064">CSS</span></b> bundle of the website. It’s rather big (~44 kB) because the site has <a href="https://getbootstrap.com/">Bootstrap</a> as a dependency. When the browser parses that <b><span style="color:#8fe064">CSS</span></b>, it discovers it needs to fetch other <b><span style="color:#a565dc">images</span></b>. These images are hosted at <code>resources.vino.com</code>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690907551/vino-com-request-mapper-cluster-css_nbaept.png">https://res.cloudinary.com/jackdbd/image/upload/v1690907551/vino-com-request-mapper-cluster-css_nbaept.png</a></p>
<p>The orange circle in the <strong>lower-right cluster</strong> is the <b><span style="color:#e7942b">JavaScript</span></b> snippet for Font Awesome. It includes several <b><span style="color:#a565dc">SVG icons</span></b>. As soon as the browser sees them, it starts fetching them. These SVGs are hosted at <code>ka-p.fontawesome.com</code>, the Font Awesome CDN which runs on the Cloudflare network.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690907551/vino-com-request-mapper-cluster-font-awesome_asrwvg.png">https://res.cloudinary.com/jackdbd/image/upload/v1690907551/vino-com-request-mapper-cluster-font-awesome_asrwvg.png</a></p>
<p>The big, orange circles in the <strong>upper-right cluster</strong> are <b><span style="color:#e7942b">JS</span></b> snippets for <a href="https://www.google.com/recaptcha/about/">reCAPTCHA</a> (left) and <a href="https://marketingplatform.google.com/about/tag-manager/">Google Tag Manager</a> (center and right). Google Tag Manager then requests its <a href="https://support.google.com/tagmanager/answer/3281060?hl=en">tags</a>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690990214/vino-com-google-tag-manager_lzlalf.png">https://res.cloudinary.com/jackdbd/image/upload/v1690990214/vino-com-google-tag-manager_lzlalf.png</a></p>
<p>In total on this page there are a bit more than 100 HTTP requests. Most for images (JPEGs and SVGs) and JS files.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Breakdown (First View, loadPage) - Requests</caption>
<thead class="">
<tr class=""><th scope="col" class="">MIME type</th><th scope="col" class="">Requests</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">image</td><td class="">64</td></tr><tr class=""><td class="">js</td><td class="">23</td></tr><tr class=""><td class="">other</td><td class="">9</td></tr><tr class=""><td class="">font</td><td class="">4</td></tr><tr class=""><td class="">css</td><td class="">3</td></tr><tr class=""><td class="">html</td><td class="">3</td></tr>
</tbody>
</table>
</div>
</div>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Breakdown (First View, loadPage) - Bytes</caption>
<thead class="">
<tr class=""><th scope="col" class="">MIME type</th><th scope="col" class="">Bytes</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">js</td><td class="">974892</td></tr><tr class=""><td class="">image</td><td class="">560041</td></tr><tr class=""><td class="">css</td><td class="">78014</td></tr><tr class=""><td class="">html</td><td class="">64722</td></tr><tr class=""><td class="">font</td><td class="">48128</td></tr><tr class=""><td class="">other</td><td class="">510</td></tr>
</tbody>
</table>
</div>
</div>
<p>If we look at the <a href="https://nooshu.com/blog/2019/12/30/how-to-read-a-wpt-connection-view-chart/">WebPageTest Connection view</a> down below, we can see that the browser needs to perform quite a bit of work to process these requests, not so much for the required bandwidth, as for the JavaScript execution. We can see that CPU utilization stays high all the time and the main thread is overworked, with a lot of spikes due to JS execution, and many <a href="https://web.dev/optimize-long-tasks/">long tasks</a>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690985174/vino-com-connection-view-loadPage_pjtyok.png">https://res.cloudinary.com/jackdbd/image/upload/v1690985174/vino-com-connection-view-loadPage_pjtyok.png</a></p>
<h4 id="h-after-accepting-the-cookies" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-after-accepting-the-cookies"><span aria-hidden="true">#</span></a> After accepting the cookies</h4>
<p>After the user accepts the cookies, a few other requests are made. They are for a couple of JS snippets and a tracking pixel set by <a href="https://developers.facebook.com/docs/meta-pixel/">Meta Pixel</a>.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Breakdown (First View, acceptCookies) - Requests</caption>
<thead class="">
<tr class=""><th scope="col" class="">MIME type</th><th scope="col" class="">Requests</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">js</td><td class="">2</td></tr><tr class=""><td class="">other</td><td class="">2</td></tr><tr class=""><td class="">image</td><td class="">1</td></tr>
</tbody>
</table>
</div>
</div>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Breakdown (First View, acceptCookies) - Bytes</caption>
<thead class="">
<tr class=""><th scope="col" class="">MIME type</th><th scope="col" class="">Bytes</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">js</td><td class="">4782</td></tr><tr class=""><td class="">image</td><td class="">43</td></tr>
</tbody>
</table>
</div>
</div>
<p>There are very few bytes to download this time, so the <code>Bandwidth In</code> line is almost flat. The browser still needs to execute some JavaScript though, so <code>CPU utilization</code> does not stay flat. Luckily there is not that much JavaScript this time, so there are no long tasks.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690985174/vino-com-connection-view-acceptCookies_g9abja.png">https://res.cloudinary.com/jackdbd/image/upload/v1690985174/vino-com-connection-view-acceptCookies_g9abja.png</a></p>
<h3 id="h-main-html-document" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-main-html-document"><span aria-hidden="true">#</span></a> Main HTML document</h3>
<p>We can make a <code>HEAD</code> request for the main HTML document using <a href="https://everything.curl.dev/usingcurl/verbose">curl in verbose mode</a>.</p>
<pre class="language-sh"><code class="language-sh"><span class="token function">curl</span> https://www.vino.com/ <span class="token parameter variable">--head</span> <span class="token parameter variable">--verbose</span></code></pre>
<p>We see that curl establishes an HTTP/2 connection:</p>
<pre class="language-sh"><code class="language-sh">* Trying <span class="token number">35.241</span>.26.2:443<span class="token punctuation">..</span>.
* Connected to www.vino.com <span class="token punctuation">(</span><span class="token number">35.241</span>.26.2<span class="token punctuation">)</span> port <span class="token number">443</span> <span class="token punctuation">(</span><span class="token comment">#0)</span>
* ALPN, offering h2
* ALPN, offering http/1.1
* ALPN, server accepted to use h2
* Using HTTP2, server supports multiplexing
<span class="token comment"># other output from curl not shown here...</span></code></pre>
<p>Among the HTTP response headers, we notice this one:</p>
<pre class="language-sh"><code class="language-sh"><span class="token comment"># other output from curl not shown here...</span>
via: <span class="token number">1.1</span> google
<span class="token comment"># other output from curl not shown here...</span></code></pre>
<p>This header is set by a <a href="https://cloud.google.com/load-balancing/docs/target-proxies">target proxy</a>, a piece of configuration of an <a href="https://cloud.google.com/load-balancing/docs/application-load-balancer">Application Load Balancer</a> deployed on Google Cloud Platform (GCP).</p>
<p>GCP Application Load Balancers support HTTP/3. So, why don’t we see a HTTP/3 connection here? Two reasons.</p>
<p>First, <a href="https://curl.se/docs/http3.html">HTTP/3 support in curl is still experimental</a>, and we would have to compile curl from source to enable it.</p>
<p>Second, clients don’t immediately connect to a server using HTTP/3. Instead, <a href="https://cloud.google.com/load-balancing/docs/https#http3-negotiation">the server advertises its support for HTTP/3 and the versions of QUIC it supports</a> using the <a href="https://www.mnot.net/blog/2016/03/09/alt-svc">Alt-Svc header</a>. In fact, the last line of the output looks like this:</p>
<pre class="language-sh"><code class="language-sh"><span class="token comment"># other output from curl not shown here...</span>
alt-svc: <span class="token assign-left variable">h3</span><span class="token operator">=</span><span class="token string">":443"</span><span class="token punctuation">;</span> <span class="token assign-left variable">ma</span><span class="token operator">=</span><span class="token number">2592000</span>,h3-29<span class="token operator">=</span><span class="token string">":443"</span><span class="token punctuation">;</span> <span class="token assign-left variable">ma</span><span class="token operator">=</span><span class="token number">2592000</span></code></pre>
<p>Chrome and all major browsers have been <a href="https://caniuse.com/http3">supporting HTTP/3</a> for quite some time now, but their first connection with an HTTP/3-compatible server will still occurr over HTTP/2. We can see this by examining the first request in the WebPageTest waterfall.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691163842/vino-com-main-http2_ja03gk.png">https://res.cloudinary.com/jackdbd/image/upload/v1691163842/vino-com-main-http2_ja03gk.png</a></p>
<p>Only after the browser has seen the <code>alt-svc</code> header and fetched a few resources over HTTP/2, it will <strong>try</strong> to establish an HTTP/3 connection with the server.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691163842/vino-com-main-http3_l5qjq2.png">https://res.cloudinary.com/jackdbd/image/upload/v1691163842/vino-com-main-http3_l5qjq2.png</a></p>
<p>The first time the browser tries to establish a HTTP/3 connection, it performs a technique called <a href="https://www.smashingmagazine.com/2021/09/http3-practical-deployment-options-part3/#alt-svc">connection racing</a>, and in Chrome DevTools we see this (the tooltip says <em>won a race…</em>):</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691146778/vino-com-chrome-http3-connection-racing_pytbj4.png">https://res.cloudinary.com/jackdbd/image/upload/v1691146778/vino-com-chrome-http3-connection-racing_pytbj4.png</a></p>
<p>After the browser has managed to connect to the server using HTTP/3, this race should no longer be necessary. In fact, in Chrome DevTools we see this (the tooltip says <em>without racing…</em>):</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691157779/vino-com-http3-without-connection-racing_n4snlb.png">https://res.cloudinary.com/jackdbd/image/upload/v1691157779/vino-com-http3-without-connection-racing_n4snlb.png</a></p>
<h3 id="h-static-assets-cloud-storage-and-cloud-cdn" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-static-assets-cloud-storage-and-cloud-cdn"><span aria-hidden="true">#</span></a> Static assets (Cloud Storage and Cloud CDN)</h3>
<p>This website hosts its static assets on <a href="https://cloud.google.com/storage">Cloud Storage</a>. We can understand this by looking at the HTTP response headers of any request to <code>resources.vino.com</code>. For example, the <a href="https://redbot.org/?uri=https://resources.vino.com/vino75-front/3.0.3310/images/frontend/favicon.ico">favicon</a>.</p>
<p>We can also see that Cloud Storage <a href="https://cloud.google.com/storage/docs/request-endpoints">supports HTTP/3</a> and advertises this fact using the <code>alt-svc</code> header.</p>
<pre class="language-sh"><code class="language-sh">HTTP/1.1 <span class="token number">200</span> OK
X-GUploader-UploadID: ADPycduJHAFYnABaw_53rnmGyt4VqCyh07sZzJ4nwRNgtKmlZhYqRn
mnlkks0qBg7pxKOVGjCZ3RgNgVETVeN6DkFWWJQg
Date: Mon, 07 Aug <span class="token number">2023</span> 08:28:03 GMT
Last-Modified: Tue, 01 Aug <span class="token number">2023</span> <span class="token number">11</span>:58:03 GMT
ETag: <span class="token string">"60454b7416845dc9281f707413c9a2f9"</span>
x-goog-generation: <span class="token number">1690891083048549</span>
x-goog-metageneration: <span class="token number">1</span>
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: <span class="token number">15086</span>
Content-Type: image/vnd.microsoft.icon
Content-Language: en
x-goog-hash: <span class="token assign-left variable">crc32c</span><span class="token operator">=</span>0l6J2A<span class="token operator">==</span>
x-goog-hash: <span class="token assign-left variable">md5</span><span class="token operator">=</span>YEVLdBaEXckoH3B0E8mi+Q<span class="token operator">==</span>
x-goog-storage-class: COLDLINE
Accept-Ranges: bytes
Content-Length: <span class="token number">15086</span>
Vary: Origin
Server: UploadServer
Cache-Control: public,max-age<span class="token operator">=</span><span class="token number">3600</span>
Alt-Svc: <span class="token assign-left variable">h3</span><span class="token operator">=</span><span class="token string">":443"</span><span class="token punctuation">;</span> <span class="token assign-left variable">ma</span><span class="token operator">=</span><span class="token number">2592000</span>,h3-29<span class="token operator">=</span><span class="token string">":443"</span><span class="token punctuation">;</span> <span class="token assign-left variable">ma</span><span class="token operator">=</span><span class="token number">2592000</span></code></pre>
<p>The first few connections to Cloud Storage will be over HTTP/2…</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691163830/vino-com-bucket-http2_evdvhj.png">https://res.cloudinary.com/jackdbd/image/upload/v1691163830/vino-com-bucket-http2_evdvhj.png</a></p>
<p>…and only after some successful fetches, the browser will try to switch to HTTP/3.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691163831/vino-com-bucket-http3_khegvk.png">https://res.cloudinary.com/jackdbd/image/upload/v1691163831/vino-com-bucket-http3_khegvk.png</a></p>
<p>Cloud Storage files are organized into <a href="https://cloud.google.com/storage/docs/buckets">buckets</a>. When you create a bucket, you specify its geographic location and choose a default storage class. The class they decided to use for the website’s assets is <a href="https://cloud.google.com/storage/docs/storage-classes#coldline">COLDINE</a>.</p>
<p>I find it rather odd that they opted for <code>COLDLINE</code> as the storage class. Its main use case is for infrequently accessed data. For website content I would have expected to find <code>STANDARD</code>.</p>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
You can store different classes of an object in the same bucket, but I had a look at several assets of this website and they all seem to use <code>COLDLINE</code> as their storage class.</p>
</div>
</div>
<p>A bucket’s <a href="https://cloud.google.com/storage/docs/locations">location</a> can be a single region, a dual-region or a multi-region. In a multi-region, all assets of the bucket are replicated in many datacenters across that region (Asia, Europe, US). However, if we want to guarantee a fast delivery to all areas of the planet, we have to use a Content Delivery Network (CDN).</p>
<p>I don’t think there is a sure way to prove that this website uses Cloud CDN, but I have a strong suspicion that it does. I came to this conclusion by performing several tests on WebPageTest, from different locations, and by looking at the <strong>Time to First Byte</strong> and <strong>Content Download</strong> times of the same asset: the favicon. All of these requests are on a HTTP/3 connection.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Requesting the favicon from different locations</caption>
<thead class="">
<tr class=""><th scope="col" class="">Location</th><th scope="col" class="">TTFB</th><th scope="col" class="">Content Download</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">Dulles (Virginia)</td><td class="">175 ms</td><td class="">14 ms</td></tr><tr class=""><td class="">Milan (Italy)</td><td class="">175 ms</td><td class=""> 13 ms</td></tr><tr class=""><td class="">Melbourne (Australia)</td><td class=""> 188 ms</td><td class=""> 13 ms</td></tr>
</tbody>
</table>
</div>
</div>
<p>Given that <a href="https://www.wolframalpha.com/input?i=distance+between+MXP+and+MEL">it takes 76 ms for the light to travel from Milan to Melbourne (in fiber)</a>, I think that it’s safe to assume that not even a multi-region bucket (e.g. in Europe) would be enough to guarantee this kind of performance for a user in Melbourne. This means that either they copy all assets from one bucket to another one using some mechanism they implemented, or they rely on a CDN. Most likely, they use Cloud CDN to automatically replicate the assets from a Cloud Storage bucket in Europe to an <a href="https://cloud.google.com/cdn/docs/locations">edge Point of Presence (PoP)</a> close to Melbourne.</p>
<p>Cloud CDN allows to add <a href="https://cloud.google.com/cdn/docs/caching#custom-headers">custom headers</a> before sending the response to the client. However, it seems this feature is not used here.</p>
<h3 id="h-caching" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-caching"><span aria-hidden="true">#</span></a> Caching</h3>
<p>Every time a user navigates to a page of a website that uses both Cloud Storage and Cloud CDN, these things might happen:</p>
<ol>
<li>The browser might find the entire snapshot of the web page in memory. This can happen if the website is <a href="https://web.dev/bfcache/#optimize-your-pages-for-bfcache">eligible to use the Back/forward cache</a> and the user already visited the page a few seconds before.</li>
<li>The browser might find some resources in its memory cache.</li>
<li>The browser might have some requests “hijacked” by a service worker that previously stored them using the <a href="https://developer.mozilla.org/en-US/docs/Web/API/CacheStorage">CacheStorage</a> interface (either at the service worker installation time, or at runtime).</li>
<li>The browser might find some resources in its <a href="https://www.chromium.org/developers/design-documents/network-stack/disk-cache/">HTTP Cache (aka disk cache)</a>.</li>
<li>The browser might find some resources on Cloud CDN and fetch them from the closest Point of Presence.</li>
<li>The browser might have to fetch some resources from the Cloud Storage bucket.</li>
</ol>
<p>We saw that <a href="https://www.vino.com/">www.vino.com</a> hosts its own static assets on <code>resources.vino.com</code> and that those assets are on Cloud Storage. Let’s also assume they are replicated on Cloud CDN. There are a few other things we can say.</p>
<p>The browser cannot use its back/forward cache on this website because the main HTML document of <a href="https://www.vino.com/">www.vino.com</a> is served with the <code>Cache-Control: no-store</code>, and because there are two <code>unload</code> events in the page.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691419327/vino-com-bfcache_y7af8g.png">https://res.cloudinary.com/jackdbd/image/upload/v1691419327/vino-com-bfcache_y7af8g.png</a></p>
<p>There is a <a href="https://developer.mozilla.org/en-US/docs/Web/Manifest">Web App Manifest</a>, but the website doesn’t install any service worker. That <code>Cache-Control: no-store</code> header returned with the main HTML document always forces the browser to hit the network, so I don’t think this <code>manifest.json</code> serves any purpose: the website is not a Progressive Web App.</p>
<p>Cloud Storage allows to configure a <code>Cache-Control</code> metadata in the same way we might configure the directives of the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control">Cache-Control HTTP header</a>. If we check a few images hosted at <code>resources.vino.com</code>, we see they are served with this <code>Cache-Control</code> header.</p>
<pre class="language-sh"><code class="language-sh">Cache-Control: public,max-age<span class="token operator">=</span><span class="token number">3600</span>
<span class="token comment"># other headers not shown here...</span></code></pre>
<p>This combination of cache directives (<code>public,max-age=3600</code>) is the <a href="https://cloud.google.com/storage/docs/metadata#caching_data">default value</a> for unencrypted content on a Cloud Storage bucket.</p>
<p>Unfortunately we can’t say for sure that these assets are cacheable for 3600 seconds (1 hour) because Cloud CDN has its own caching behavior. In fact, Cloud CDN has <a href="https://cloud.google.com/cdn/docs/caching#cache-modes">three cache modes</a> and it can either honor or disregard the <code>Cache-Control</code> directives.</p>
<p>The main HTML document is served with a <code>Cache-Control</code> header that includes a <code>no-store</code> directive. This is problematic because that <code>no-store</code> directive prevents the browser from using the back/forward cache, makes it hit the network any time it requests the page, and always causes the load balancer (see that <code>1.1 google</code>) to request the HTML document from the origin server (Apache in this case).</p>
<pre class="language-sh"><code class="language-sh">Server: Apache
Cache-Control: no-cache, no-store, max-age<span class="token operator">=</span><span class="token number">0</span>, must-revalidate
Vary: Accept-Encoding
Via: <span class="token number">1.1</span> google
<span class="token comment"># other headers not shown here...</span></code></pre>
<p>I think a more reasonable <code>Cache-Control</code> header could be something like this:</p>
<pre class="language-sh"><code class="language-sh">Cache-Control: must-revalidate, max-age<span class="token operator">=</span><span class="token number">120</span></code></pre>
<p>This combination of <code>Cache-Control</code> directives tells the browser that if it requests the HTML document within 120 seconds, it can use the cached version. After 120 seconds, the browser must revalidate the cached version against the origin server. If the origin server returns a <code>304 Not Modified</code> response, the browser <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/304">implicitly redirects</a> to the cached version. If the origin server returns a <code>200 OK</code> response, the browser must use the new version.</p>
<h3 id="h-images" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-images"><span aria-hidden="true">#</span></a> Images</h3>
<p>As I wrote in <a href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-images">a previous article</a>, optimizing images is crucial for performance, SEO and accessibility.</p>
<p>There are a lot of images on the home page of <a href="https://www.vino.com/">www.vino.com</a>. The <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-before-accepting-the-cookies">MIME type breakdown</a> table shows that on a first visit, roughly 600 kB of data are images. This is not terrible, but there is certainly room for improvement. There are a few changes we could make.</p>
<h4 id="h-webp-and-avif" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-webp-and-avif"><span aria-hidden="true">#</span></a> WebP and AVIF</h4>
<p>First of all, the images in the home page are JPEG. We can save a lot of bytes if we serve <a href="https://caniuse.com/?search=WebP">WebP</a> or <a href="https://caniuse.com/?search=AVIF">AVIF</a> instead.</p>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
<a href="https://stackoverflow.com/questions/75459594/why-doesnt-edge-support-avif-images">Microsoft Edge doesn't yet support AVIF</a>, but one of its canary releases seems to support it, so I guess AVIF support in Edge is near.</p>
</div>
</div>
<h4 id="h-%3Cimg%3E-with-an-appropriate-alt-attribute" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-%3Cimg%3E-with-an-appropriate-alt-attribute"><span aria-hidden="true">#</span></a> <code><img></code> with an appropriate <code>alt</code> attribute</h4>
<p>Second, most of the images on the home page are background images, meaning they are URL in a <code>background-image</code> CSS property. The simplified HTML markup for one of these images looks like this:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>selezione/serena-1881<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token special-attr"><span class="token attr-name">style</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span><span class="token value css language-css"><span class="token property">background-image</span><span class="token punctuation">:</span> <span class="token url"><span class="token function">url</span><span class="token punctuation">(</span>https://resources.vino.com/data/slideshowItemImmagine/slideshowItemImmagine-6583.jpg<span class="token punctuation">)</span></span><span class="token punctuation">;</span></span><span class="token punctuation">"</span></span></span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span><span class="token punctuation">></span></span>Serena 1881<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span></code></pre>
<p>This is a problem because background images are <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/background-image#accessibility_concerns">not accessible</a>. Without an <code><img></code> element with a proper <code>alt</code> attribute, these images are completely invisible to screen readers.</p>
<p>Not having <code><img></code> elements with an <code>alt</code> attribute is also an issue from a SEO standpoint. The <code>alt</code> attribute should be used to describe the image and <a href="https://developers.google.com/search/docs/appearance/google-images#create-a-great-user-experience">contextualize it</a> in the page.</p>
<h4 id="h-resolution-switching" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-resolution-switching"><span aria-hidden="true">#</span></a> Resolution switching</h4>
<p>Third, we could serve lower resolution images to smaller viewports. This is called <a href="https://cloudfour.com/thinks/responsive-images-the-simple-way/#the-sizes-attribute">resolution switching</a> and involves generating several versions of the same image at different resolutions, and letting the browser decide what to download using the <code>srcset</code> and <code>sizes</code> attributes of the <code><img></code> element.</p>
<h4 id="h-low-quality-image-placeholder-lqip" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-low-quality-image-placeholder-lqip"><span aria-hidden="true">#</span></a> Low Quality Image Placeholder (LQIP)</h4>
<p>Each product card on the home page includes the product’s title and description, two badges that tell us the discount and deadline for the offer, and a black image. The black image is then swapped for the actual image of the product.</p>
<p>Here is how the home page looks when loading with a <strong>Fast 3G</strong> connection (I used Network throttling in Chrome DevTools).</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1691594811/vino-com-videos-with-no-lqip_bmoalz.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1691594811/vino-com-videos-with-no-lqip_bmoalz.mp4</a></p>
<p>A better user experience would be to avoid showing a black image, and use a <a href="https://cloudinary.com/blog/low_quality_image_placeholders_lqip_explained">Low Quality Image Placeholder (LQIP)</a> instead.</p>
<h3 id="h-svg-icons" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-svg-icons"><span aria-hidden="true">#</span></a> SVG icons</h3>
<p>WebPageTest can show us the number of requests made to each domain, and can tell us how many bytes were downloaded from each domain. This information is available in the <a href="https://nooshu.com/blog/2019/12/30/how-to-read-a-wpt-connection-view-chart/">Connection view</a> and in the two <strong>Domains Breakdown</strong> tables.</p>
<p>The first <strong>Domains Breakdown</strong> table shows the number of requests per domain.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Breakdown by domain (First View, loadPage) - Requests</caption>
<thead class="">
<tr class=""><th scope="col" class="">Domain</th><th scope="col" class="">Requests</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">ka-p.fontawesome.com</td><td class="">39</td></tr><tr class=""><td class="">resources.vino.com</td><td class="">32</td></tr><tr class=""><td class="">fonts.gstatic.com</td><td class="">4</td></tr><tr class=""><td class="">www.gstatic.com</td><td class="">4</td></tr><tr class=""><td class="">www.google.com</td><td class="">3</td></tr><tr class=""><td class="">bat.bing.com</td><td class="">3</td></tr><tr class=""><td class="">www.googletagmanager.com</td><td class="">3</td></tr><tr class=""><td class="">www.facebook.com</td><td class="">2</td></tr><tr class=""><td class="">onesignal.com</td><td class="">2</td></tr><tr class=""><td class="">cdn.onesignal.com</td><td class="">2</td></tr><tr class=""><td class="">gum.criteo.com</td><td class="">2</td></tr><tr class=""><td class="">www.vino.com</td><td class="">2</td></tr><tr class=""><td class="">www.google-analytics.com</td><td class="">2</td></tr><tr class=""><td class="">stats.g.doubleclick.net</td><td class="">2</td></tr><tr class=""><td class="">connect.facebook.net</td><td class="">2</td></tr><tr class=""><td class="">others</td><td class="">6</td></tr>
</tbody>
</table>
</div>
</div>
<p>We can see that when we visit <a href="https://www.vino.com/">www.vino.com</a> (before accepting or dismissing the consent banner), the browser makes two requests to <code>www.vino.com</code>, and 32 to <code>resources.vino.com</code>. Since the website’s owner controls these domains, we call these requests <strong>first-party</strong> requests. All other requests the browser makes are towards domains the website’s owner doesn’t control, so we call them <strong>third-party</strong> requests. In general it’s a good idea to <a href="https://csswizardry.com/2019/05/self-host-your-static-assets/">self-host our static assets</a>.</p>
<p>Those 39 requests to <code>ka-p.fontawesome.com</code> are third-party requests that the browser makes to download all SVG icons of the website. SVG icons are in general quite small in size, so those 39 requests account for only 34 KB of data.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1690473571/vino-com-font-awesome-svgs_sewocv.png">https://res.cloudinary.com/jackdbd/image/upload/v1690473571/vino-com-font-awesome-svgs_sewocv.png</a></p>
<p>The second <strong>Domains Breakdown</strong> table shows how many bytes are downloaded per domain.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Breakdown by domain (First View, loadPage) - Bytes</caption>
<thead class="">
<tr class=""><th scope="col" class="">Domain</th><th scope="col" class="">Bytes</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">resources.vino.com</td><td class="">702353</td></tr><tr class=""><td class="">www.gstatic.com</td><td class="">383005</td></tr><tr class=""><td class="">www.googletagmanager.com</td><td class="">238256</td></tr><tr class=""><td class="">connect.facebook.net</td><td class="">136183</td></tr><tr class=""><td class="">cdn.onesignal.com</td><td class="">72234</td></tr><tr class=""><td class="">fonts.gstatic.com</td><td class="">48128</td></tr><tr class=""><td class="">ka-p.fontawesome.com</td><td class="">34045</td></tr><tr class=""><td class="">www.vino.com</td><td class="">30944</td></tr><tr class=""><td class="">www.google.com</td><td class="">27159</td></tr><tr class=""><td class="">www.google-analytics.com</td><td class="">20998</td></tr><tr class=""><td class="">others</td><td class="">72823</td></tr>
</tbody>
</table>
</div>
</div>
<p><code>resources.vino.com</code> hosts most of the high-res JPEG images of the website, so it’s not surprising that most of the data comes from this domain.</p>
<p>All assets of <code>ka-p.fontawesome.com</code> are hosted on the Font Awesome CDN, which itself runs on Cloudflare. We know this because each request to an SVG icon of the website returns these headers:</p>
<pre class="language-sh"><code class="language-sh">CF-Cache-Status: HIT
Server: cloudflare
CF-RAY: 7f4772a43e05faf0-SJC
<span class="token comment"># other headers not shown here...</span></code></pre>
<p>Making 39 requests to a third-party domain just to download the website’s icons seems a bit wasteful. I would create a single SVG sprite of all the icons and self-host it on <code>resources.vino.com</code>.</p>
<div class="callout callout--tip">
<div class="callout__content">
<p>💡 —
You can use <a href="https://github.com/svg-sprite/svg-sprite">svg-sprite</a> to create an SVG sprite.</p>
</div>
</div>
<p>Requests in HTTP/2 are cheap thanks to <a href="https://www.mnot.net/blog/2019/10/13/h2_api_multiplexing">multiplexing</a>, so we might not gain a lot in performance by reducing them from 39 (the individual 39 SVG icons) to one (the single SVG sprite). But eliminating a third party domain means the browser has one less DNS lookup to do, one less TLS handshake to perform, and one less TCP connection to open. These costs add up, so we should see a performance improvement if we avoid them. Also, by hosting an asset on a domain we own, we have full control on the caching policy for that particular asset.</p>
<h3 id="h-fonts" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-fonts"><span aria-hidden="true">#</span></a> Fonts</h3>
<p>Other third-party assets found on this website are the Google Fonts, which are hosted on <code>fonts.gstatic.com</code> and they are served with these HTTP response headers:</p>
<pre class="language-sh"><code class="language-sh">Server: sffe
Cache-Control: public, max-age<span class="token operator">=</span><span class="token number">31536000</span>
Content-Type: font/woff2
<span class="token comment"># other headers not shown here...</span></code></pre>
<p>Both the font format (woff2) and the caching policy are fine (these assets can be cached for up to one year), but <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-did-you-ask-for-permission%3F">as I said before</a>, we can’t connect to a third-party domain without obtaining the user’s consent first. We can avoid regulatory issues and save us another DNS lookup + TLS handshake + TCP connection by self-hosting all fonts on <code>resources.vino.com</code>.</p>
<h3 id="h-meta-pixel" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-meta-pixel"><span aria-hidden="true">#</span></a> Meta Pixel</h3>
<p>This website use Meta Pixel, a tool that I have no doubt is really useful to the marketing department.</p>
<p>If we zoom in on the request map, we can see that the website loads the JS snippet and the tracking pixel directly. For a better performance, a common recommendation is to let <a href="https://www.facebook.com/business/help/1021909254506499">Google Tag Manager load these files for us</a>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691684784/vino-com-meta-pixel_u37jbg.png">https://res.cloudinary.com/jackdbd/image/upload/v1691684784/vino-com-meta-pixel_u37jbg.png</a></p>
<p>Another potential issue is that the JS snippet is downloaded when the consent banner is still on screen. This might be ok, as long as the website <a href="https://developers.facebook.com/docs/meta-pixel/implementation/gdpr/">doesn’t send any data to Facebook before obtaining the user’s consent</a>.</p>
<h3 id="h-fighting-bots" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-fighting-bots"><span aria-hidden="true">#</span></a> Fighting bots</h3>
<p>This website uses reCAPTCHA to fight bot traffic and tell humans and bots apart.</p>
<p>The first big problem with reCAPTCHA is its impact on the website’s performance, especially on low-end mobile devices. The Lighthouse Treemap down below shows that the reCAPTCHA snippet adds 433.753 KiB di JS. Even worse, the <code>Coverage</code> column tells us that almost half of that JS is unused.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691680843/vino-com-lighthouse-treemap_rbefln.png">https://res.cloudinary.com/jackdbd/image/upload/v1691680843/vino-com-lighthouse-treemap_rbefln.png</a></p>
<p>The second big problem with reCAPTCHA is that it’s a Google’s service. Google makes money by selling ads, but every bot identified by reCAPTCHA directly reduces Google’s ad revenue. Google gives up some revenue every time reCAPTCHA does its job, so improving reCAPTCHA’s anti-bot capabilities too much <a href="https://www.hcaptcha.com/report-how-much-is-a-recaptcha-really-worth">doesn’t really make sense for them</a>.</p>
<p>But fighting bots is important, so we can’t really drop reCAPTCHA without replacing it with an alternative.</p>
<p>It’s hard to make recommendations here, but I think that an appropriate solution could be <a href="https://www.hcaptcha.com/">hCaptcha</a>, or some service that stops bot traffic at the edge, like <a href="https://cloud.google.com/armor/docs/bot-management">Google Cloud Armor</a> or <a href="https://www.cloudflare.com/application-services/products/bot-management/">Cloudflare Bot Management</a>.</p>
<h2 id="h-how-can-we-improve-it%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-how-can-we-improve-it%3F"><span aria-hidden="true">#</span></a> How can we improve it?</h2>
<p>Improving a website is an iterative process that goes like this:</p>
<ol>
<li>observe the current situation</li>
<li>raise a question</li>
<li>formulate a hypothesis</li>
<li>run an experiment to test the hypothesis</li>
<li>draw a conclusion</li>
<li>approve or reject change</li>
<li>repeat</li>
</ol>
<p>By looking at charts like the Waterfall View and the Connection View in WebPageTest, we can come up with hypotheses to make a website faster, leaner or more resilient. By running a few <strong>performance experiments</strong>, we can test such hypotheses.</p>
<h3 id="h-self-host-google-fonts" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-self-host-google-fonts"><span aria-hidden="true">#</span></a> Self-host Google fonts</h3>
<p>The home page of <a href="https://www.vino.com/">www.vino.com</a> uses a few web fonts from Google Fonts. More precisely, it requests one variant of Open Sans and three variants of Montserrat from <code>fonts.gstatic.com</code>.</p>
<p>We can simulate an outage of the Google Fonts service by directing the host <code>fonts.gstatic.com</code> to the WebPageTest’s blackhole server, which will hang indefinitely until timing out. Chrome will give up after roughly 30 seconds and it will print a <code>ERR_CONNECTION_TIMED_OUT</code> error in the console for each connection it did not manage to establish.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691948062/vino-com-google-fonts-times-out-console_s4krqp.png">https://res.cloudinary.com/jackdbd/image/upload/v1691948062/vino-com-google-fonts-times-out-console_s4krqp.png</a></p>
<p>If we have a look at the WebPageTest <strong>Connection View</strong>, we notice a huge gap where almost nothing happens: the <code>Bandwidth In</code> chart is flat, the <code>Browser Main Thread</code> chart shows very little activity, the <code>CPU utilization</code> chart stays at roughly 50%. The browser keeps trying to connect to <code>fonts.gstatic.com</code>, so even if the <b><span style="color:#D887E0">DOMContentLoaded</span></b> event fires right after 3 seconds, the <b><span style="color:#0000FF">load</span></b> event takes much longer and fires only at around 34 seconds.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691947828/vino-com-connection-view-google-fonts-experiment_xzeb3i.png">https://res.cloudinary.com/jackdbd/image/upload/v1691947828/vino-com-connection-view-google-fonts-experiment_xzeb3i.png</a></p>
<p>For comparison, the WebPageTest Connection View shows no gaps when the browser manages to connect to <code>fonts.gstatic.com</code> without issues.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691947827/vino-com-connection-view-google-fonts-control_wweyke.png">https://res.cloudinary.com/jackdbd/image/upload/v1691947827/vino-com-connection-view-google-fonts-control_wweyke.png</a></p>
<p>The (simulated) outage of <code>fonts.gstatic.com</code> causes <code>Load Time</code> and <code>CPU Busy Time</code> to take almost 30 seconds longer than the usual.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691948794/vino-com-timings-google-fonts-timeout_euqir8.png">https://res.cloudinary.com/jackdbd/image/upload/v1691948794/vino-com-timings-google-fonts-timeout_euqir8.png</a></p>
<p>It could be worse though. It’s true that when the browser can’t connect to <code>fonts.gstatic.com</code> it can’t get Open Sans or Montserrat, but this website have CSS declarations like this one:</p>
<pre class="language-css"><code class="language-css"><span class="token property">font-family</span><span class="token punctuation">:</span> montserrat<span class="token punctuation">,</span> sans-serif<span class="token punctuation">;</span></code></pre>
<p>This tells the browser that it can fallback to a generic sans-serif <a href="https://fonts.google.com/knowledge/glossary/system_font_web_safe_font">system font</a> if <code>montserrat</code> is not available. Also, this website includes a few <code>@font-face</code> rules like this one:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<span class="token atrule"><span class="token rule">@font-face</span></span> <span class="token punctuation">{</span>
<span class="token property">font-family</span><span class="token punctuation">:</span> <span class="token string">'Montserrat'</span><span class="token punctuation">;</span>
<span class="token property">font-style</span><span class="token punctuation">:</span> normal<span class="token punctuation">;</span>
<span class="token property">font-weight</span><span class="token punctuation">:</span> 700<span class="token punctuation">;</span>
<span class="token property">src</span><span class="token punctuation">:</span> <span class="token function">local</span><span class="token punctuation">(</span><span class="token string">'Montserrat Bold'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token function">local</span><span class="token punctuation">(</span><span class="token string">'Montserrat-Bold'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token url"><span class="token function">url</span><span class="token punctuation">(</span>https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3gnD_vx3rCs.woff2<span class="token punctuation">)</span></span> <span class="token function">format</span><span class="token punctuation">(</span><span class="token string">'woff2'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token property">unicode-range</span><span class="token punctuation">:</span> U+0000-00FF<span class="token punctuation">,</span> U+0131<span class="token punctuation">,</span> U+0152-0153<span class="token punctuation">,</span> U+02BB-02BC<span class="token punctuation">,</span> U+02C6<span class="token punctuation">,</span> U+02DA<span class="token punctuation">,</span> U+02DC<span class="token punctuation">,</span> U+2000-206F<span class="token punctuation">,</span> U+2074<span class="token punctuation">,</span> U+20AC<span class="token punctuation">,</span> U+2122<span class="token punctuation">,</span> U+2191<span class="token punctuation">,</span> U+2193<span class="token punctuation">,</span> U+2212<span class="token punctuation">,</span> U+2215<span class="token punctuation">,</span> U+FEFF<span class="token punctuation">,</span> U+FFFD<span class="token punctuation">;</span>
<span class="token property">font-display</span><span class="token punctuation">:</span> swap<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token comment">/* other font-face rules not shown here... */</span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span></code></pre>
<p>That <code>font-display: swap;</code> is important, because it tells the browser to immediately render the fallback font (i.e. <code>sans-serif</code>) and swap it with Montserrat when it’s ready. If the browser can’t connect to <code>fonts.gstatic.com</code>, it will never use Montserrat, but it will not block rendering.</p>
<p>However, why should we leave our four Google Fonts on <code>fonts.gstatic.com</code>, wasting a DNS lookup, a TCP connection and a TLS handshake to connect to it? It’s better to self-host them on <code>www.vino.com</code> or <code>resources.vino.com</code>.</p>
<h3 id="h-self-host-font-awesome" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-self-host-font-awesome"><span aria-hidden="true">#</span></a> Self-host Font Awesome</h3>
<p>This website uses a <a href="https://fontawesome.com/docs/web/setup/use-kit">Font Awesome Kit</a> for its SVG icons. Using a kit involves fetching a JS script, and letting this script request all the SVG icons included in the kit. We can see this in the Font Awesome cluster <a href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-visualize-http-requests">described above</a>. This kit is hosted at <code>kit.fontawesome.com</code>.</p>
<p>We can simulate an outage of the Font Awesome service in the same way we did for Google Fonts. Once again, Chrome will give up after roughly 30 seconds and it will print a <code>ERR_CONNECTION_TIMED_OUT</code> error in the console.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691953288/vino-com-font-awesome-timeout-console-log_dkhya2.png">https://res.cloudinary.com/jackdbd/image/upload/v1691953288/vino-com-font-awesome-timeout-console-log_dkhya2.png</a></p>
<p>However, the impact on performance is much worse this time. Failing to connect to <code>kit.fontawesome.com</code> delays <strong>Start Render</strong> and <strong>First Contentful Paint</strong> by roughly 30 seconds. We can see this either in the WebPageTest <strong>Waterfall View</strong>…</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691952762/vino-com-font-awesome-timeout-waterfall_ophwzu.png">https://res.cloudinary.com/jackdbd/image/upload/v1691952762/vino-com-font-awesome-timeout-waterfall_ophwzu.png</a></p>
<p>…or in the summary report of the control run versus the experiment run of this performance experiment.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691952762/vino-com-font-awesome-timeout-summary_jkfa4s.png">https://res.cloudinary.com/jackdbd/image/upload/v1691952762/vino-com-font-awesome-timeout-summary_jkfa4s.png</a></p>
<p>The reason for this huge degradation in performance is that the snippet hosted at <code>kit.fontawesome.com</code> is render-blocking.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691952762/vino-com-font-awesome-timeout-spof_kb32xl.png">https://res.cloudinary.com/jackdbd/image/upload/v1691952762/vino-com-font-awesome-timeout-spof_kb32xl.png</a></p>
<p>Chrome blocks rendering until it gives up connecting to <code>kit.fontawesome.com</code>. Only then it starts rendering the page.</p>
<p>In the screenshot below, the black circle would normally include the icon of a truck, but since the browser can’t connect to <code>kit.fontawesome.com</code>, it can’t fetch any SVG icon from the Font Awesome Kit.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1691953472/vino-com-fontawesome-connection-timeout-missing-icons_hnu63z.png">https://res.cloudinary.com/jackdbd/image/upload/v1691953472/vino-com-fontawesome-connection-timeout-missing-icons_hnu63z.png</a></p>
<p>Just like we did for Google Fonts, it makes sense to self-host this Font Awesome Kit as well.</p>
<h3 id="h-avoid-redirects-using-cache-control-immutable" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-avoid-redirects-using-cache-control-immutable"><span aria-hidden="true">#</span></a> Avoid redirects using <code>Cache-Control: immutable</code></h3>
<p>During the <strong>first visit</strong> to the home page, the browser caches all assets hosted at <code>resources.vino.com</code> (CSS, JS, images) because they are served with this <code>Cache-Control</code> header.</p>
<pre class="language-text"><code class="language-text">Cache-Control: public, max-age=3600</code></pre>
<p>However, if we have a look at the WebPageTest <strong>Waterfall View</strong> of a <strong>repeat visit</strong>, we notice many redirects (the rows in yellow).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1692005417/vino-com-http-304-missing-immutable_prst47.png">https://res.cloudinary.com/jackdbd/image/upload/v1692005417/vino-com-http-304-missing-immutable_prst47.png</a></p>
<p>Each HTTP 304 redirect means that the browser revalidated the resource against the server, and since the resource did not change (that’s why it is called 304 Not Modified), the browser performed an implicit redirection to the cached version.</p>
<p>The thing is, these redirects introduce a performance penalty which <a href="http://bitsup.blogspot.com/2016/05/cache-control-immutable.html">sometimes can be rather significant</a>.</p>
<p>Images hosted at <code>resources.vino.com</code> will likely never change, so we can tell the browser to cache them for one year and to never revalidate them agains the server. We can achieve this by serving the images with this <code>Cache-Control</code> header.</p>
<pre class="language-text"><code class="language-text">Cache-Control: public, max-age=31536000, immutable</code></pre>
<p>That <code>immutable</code> directive instructs the browser to immediately use the cached version of an asset, so the request for that asset immediately returns an HTTP 200 instead of an HTTP 304.</p>
<p>Another option would be to use an image CDN like Cloudinary to host, <a href="https://web.dev/image-cdns/">optimize</a> and serve the images of this website. In that case we would completely offload to Cloudinary the responsibility of defining the <a href="https://cloudinary.com/glossary/caching-images">caching policies</a> for the images.</p>
<p>CSS and JS hosted at <code>resources.vino.com</code> will probably change, so it’s better to avoid the <code>immutable</code> directive for them, unless we implement some form of <a href="https://www.keycdn.com/support/what-is-cache-busting">cache busting</a> (e.g. asset fingerprinting by adding a hash to the asset filename).</p>
<h2 id="h-conclusion" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/auditing-an-ecommerce-website/#h-conclusion"><span aria-hidden="true">#</span></a> Conclusion</h2>
<p>There are several things we could try to make the home page of <a href="https://www.vino.com/">www.vino.com</a> faster, leaner or more resilient:</p>
<ul>
<li>Self-host Google fonts, or maybe even avoid web fonts altogether and use a sans-serif system font stack like <a href="https://booking.design/implementing-system-fonts-on-booking-com-a-lesson-learned-bdc984df627f">the one chosen by Booking.com</a>.</li>
<li>Self-host Font Awesome script and SVG icons, to avoid a single point of failure in page rendering.</li>
<li>Use AVIF or WebP as the image format, instead of JPEG.</li>
<li>Review which third-party scripts are executed before and after giving an explicit consent, and make sure there are no GDPR violations.</li>
<li>Use resolution switching with images, so big images will be downloaded only when viewing the website on wide viewports.</li>
<li>Optimize images on the fly using an image CDN like Cloudinary, or at build time using a tool like <a href="https://jampack.divriots.com/">Jampack</a>.</li>
<li>Return a <code>Cache-Control: immutable</code> header when serving the images, so the browser can save a few costly redirects when it already has an image in cache.</li>
<li>Return a <code>Content-Security-Policy</code> header to mitigate XSS attacks, and maybe also a Permissions-Policy to limit which browser APIs a third-party script is allowed to access on a page.</li>
<li>Add a <a href="https://www.w3.org/TR/reporting-1/#header">Reporting-Endpoints</a> header to track Content-Security-Policy violations, browser crashes, deprecation warnings.</li>
<li>Implement some form af cache busting for CSS and JS, so we can use <code>Cache-Control: immutable</code> for those assets too.</li>
<li>Be sure to avoid shipping code with known vulnerabilities, maybe by runnning <code>npm audit --audit-level=moderate</code> in the CI pipeline.</li>
<li>Consider building multiple sitemaps, and see if this translates into SEO improvements.</li>
<li>Consider alternatives to reCAPTCHA. It has a huge performance cost, especially on low-end mobile phones.</li>
<li>Make the product images more accessible (i.e. use an <code><img></code> element with an appropriate <code>alt</code> attribute).</li>
</ul>
https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/Performance audit of an Italian news website2023-02-27T13:30:00Z2023-02-27T13:30:00ZThis is a web performance audit of the web page iltirreno.it/versilia. The audit considers both field data (extracted from the CrUX dataset) and lab data (collected from several WebPageTest test results).
<p>News websites have a lot of text, images and other media, and they often rely on many third party scripts for ads and analytics. This huge amount of assets means that the number of HTTP requests on these websites can be quite high. The browser might take a while to process all of these requests and execute JavaScript, and performance can suffer.</p>
<p>Users spend less time consuming content on a slow website. This can have a negative impact on revenue, in particular for news websites. Google’s DoubleClick reports that publishers whose mobile sites load in 5 seconds earn up to <a href="https://wpostats.com/2016/09/15/double-click-revenue.html">2x more mobile ad revenue</a> than sites loading in 19 seconds.</p>
<p>In this article I’m going to describe what I found out about the digital version of <a href="https://www.iltirreno.it/">Il Tirreno</a>, an Italian newspaper that covers news about Tuscany. I’m going to analyze both data collected from the real users visiting the website, and data collected in a controlled environment under a predefined set of network and device conditions.</p>
<p>Il Tirreno has 14 different editions, and eight out of the ten provinces in Tuscany have at least one edition of this newspaper. Since I live in Versilia, I decided to focus my analysis on the web page <a href="https://www.iltirreno.it/versilia">https://www.iltirreno.it/versilia</a>. This means that all my findings and suggestions are related to that page. The structure of the local news front page stays the same across all 14 local editions of <a href="http://iltirreno.it/">iltirreno.it</a> though, so most conclusions should hold for pages like <a href="https://www.iltirreno.it/lucca">https://www.iltirreno.it/lucca</a> or <a href="https://www.iltirreno.it/pisa">https://www.iltirreno.it/pisa</a>.</p>
<h2 id="h-real-user-metrics" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-real-user-metrics"><span aria-hidden="true">#</span></a> Real user metrics</h2>
<p>The <a href="https://developer.chrome.com/docs/crux/">Chrome User Experience Report (CrUX)</a> provides metrics for how real-world Chrome users experience websites that are <a href="https://developer.chrome.com/docs/crux/methodology/#popularity-eligibility">sufficiently popular</a>. These metrics are <a href="https://developer.chrome.com/docs/crux/methodology/#user-eligibility">collected</a> from Android and desktop users that have Chrome as their browser. In other words, in CrUX we won’t find any data originated from iOS devices or any browser other than Chrome.</p>
<p>These data are <a href="https://developer.chrome.com/docs/crux/methodology/#origin-eligibility">aggregated by origin</a> and collected each month. The following month, these data are then ingested into BigQuery.</p>
<h3 id="h-the-crux-dataset" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-the-crux-dataset"><span aria-hidden="true">#</span></a> The CrUX dataset</h3>
<p>Il Tirreno is a popular newspaper in Tuscany, and its digital edition <a href="https://www.iltirreno.it/">Iltirreno.it</a> has enough traffic to end up in CrUX.</p>
<p>The CrUX BigQuery dataset is publicly available, but in order to explore it you will need a Google Cloud Platform project and some knowledge of SQL. You’ll find links for the queries I ran, so you can try them out yourself if you want.</p>
<div class="callout callout--warn">
<div class="callout__content">
<p>⚠️ —
The CrUX dataset is huge, so be careful when writing queries against it. With BigQuery <a href="https://cloud.google.com/bigquery/pricing">on-demand pricing</a> you can query up to 1TB worth of data per month for free. If you exceed that quota you'll have to pay $5/TB. To avoid unforeseen expenses with BigQuery, always have a look at <a href="https://cloud.google.com/bigquery/docs/estimate-costs">how much data a query would process</a> before running it.</p>
</div>
</div>
<h3 id="h-traffic-by-device" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-traffic-by-device"><span aria-hidden="true">#</span></a> Traffic by device</h3>
<p>First of all, I wanted to know what was the percentage of phone, tablet and desktop traffic on any page of <code>www.iltirreno.it</code>. CrUX contains data from 2017 onwards, but I didn’t want to process too much data (i.e. spend money for this query), so I restricted the analysis to the month of January 2023.</p>
<p>Unsurprisingly, <code>www.iltirreno.it</code> is most popular in Italy. Somewhat surprisingly (at least for me), almost 90% of the Italian traffic in January 2023 came from phones.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Traffic by device (% of users, first 3 results)</caption>
<thead class="">
<tr class=""><th scope="col" class="">Country</th><th scope="col" class="">Popularity</th><th scope="col" class="">Desktop</th><th scope="col" class="">Phone</th><th scope="col" class="">Tablet</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">Italy</td><td class="">1000</td><td class="">9.0</td><td class="">89.0</td><td class="">3.0</td></tr><tr class=""><td class="">San Marino</td><td class="">5000</td><td class="">0.0</td><td class="">100.0</td><td class="">0.0</td></tr><tr class=""><td class="">Albania</td><td class="">10000</td><td class="">0.0</td><td class="">100.0</td><td class="">0.0</td></tr>
</tbody>
</table>
</div>
</div>
<p><a href="https://console.cloud.google.com/bigquery?sq=1051247446620:e61a85e3d7e24024abf7b3c6b70df9d8" target="_blank" rel="external noopener noreferrer">Here is the query.</a> Try it out!</p>
<h3 id="h-traffic-by-effective-connectivity" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-traffic-by-effective-connectivity"><span aria-hidden="true">#</span></a> Traffic by effective connectivity</h3>
<p>The <a href="https://developer.chrome.com/docs/crux/methodology/#ect-dimension">effective connectivity</a> for the majority of those visits was 4G.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Traffic by connectivity (% of users, only Italy)</caption>
<thead class="">
<tr class=""><th scope="col" class="">Country</th><th scope="col" class="">3G</th><th scope="col" class="">4G</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">Italy</td><td class="">5.0</td><td class="">95.0</td></tr>
</tbody>
</table>
</div>
</div>
<p><a href="https://console.cloud.google.com/bigquery?sq=1051247446620:577a4db925fd4f7c8ec0a86f98710294" target="_blank" rel="external noopener noreferrer">Here is the query.</a> Try it out!</p>
<h3 id="h-performance-over-time" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-performance-over-time"><span aria-hidden="true">#</span></a> Performance over time</h3>
<p>I wanted to know whether the performance changed over the course of 3 months. The metrics I decided to extract from CrUX are:</p>
<ul>
<li><a href="https://web.dev/ttfb/">Time To First Byte (TTFB)</a></li>
<li><a href="https://web.dev/fcp/">First Contentful Paint (FCP)</a></li>
<li>The browser’s Window <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/load_event">load event</a></li>
</ul>
<p>Here is why I decided to pick these metrics:</p>
<ul>
<li>TTFB. This is the time between the browser’s request and the first byte of response received from the server. A long TTFB suggests there is some issue on the backend, which could be caused by many factors: a slow webserver, too much requests hitting the server, slow database queries, network congestion. On the other hand, if TTFB is really short (good) but perfomance is still not great, it could be a sign that there is some issue on the frontend.</li>
<li>FCP. This is the time from when the page starts loading to when any part of the page’s content is rendered on the screen. More precisely, when we say content we mean any text, image (including background images), non-white <code><canvas></code> elements or SVG. The content which is first rendered might not be the biggest one which ends on the screen. For that there is another metric called Largest Contentful Paint (LCP). I decided to pick FCP instead of LCP because I noticed that the biggest content on <a href="http://iltirreno.it/versilia">iltirreno.it/versilia</a> is often an ad, not some image of the website itself.</li>
<li><code>load</code> event. The browser fires this event when it has finished loading the page and all of its dependent resources: stylesheets, scripts, images, iframes. This event is important because if the browser takes a long time to fire it, it means it has still some work to do, even if we see a visually completed page.</li>
</ul>
<p>In the tables below, TTFB good/avg/bad is expressed in percentage of users that experienced that metric in that particular way. The same applies to FCP and the load event. Instead, p75 represents the 75th percentile of that metric and it is expressed in milliseconds.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>TTFB over time (% of users; 75th percentile in ms)</caption>
<thead class="">
<tr class=""><th scope="col" class="">yyyymmdd</th><th scope="col" class="">TTFB good</th><th scope="col" class="">TTFB avg</th><th scope="col" class="">TTFB bad</th><th scope="col" class="">TTFB p75</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">20230101</td><td class="">57.24</td><td class="">26.72</td><td class="">16.0</td><td class="">1300</td></tr><tr class=""><td class="">20221201</td><td class="">50.89</td><td class="">30.13</td><td class="">18.92</td><td class="">1500</td></tr><tr class=""><td class="">20221101</td><td class="">45.59</td><td class="">33.34</td><td class="">21.04</td><td class="">1600</td></tr>
</tbody>
</table>
</div>
</div>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>FCP over time (% of users; 75th percentile in ms)</caption>
<thead class="">
<tr class=""><th scope="col" class="">yyyymmdd</th><th scope="col" class="">FCP good</th><th scope="col" class="">FCP avg</th><th scope="col" class="">FCP bad</th><th scope="col" class="">FCP p75</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">20230101</td><td class="">72.47</td><td class="">17.09</td><td class="">10.41</td><td class="">1900</td></tr><tr class=""><td class="">20221201</td><td class="">68.03</td><td class="">19.75</td><td class="">12.16</td><td class="">2100</td></tr><tr class=""><td class="">20221101</td><td class="">66.28</td><td class="">20.88</td><td class="">12.85</td><td class="">2100</td></tr>
</tbody>
</table>
</div>
</div>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>load event over time (% of users; 75th percentile in ms)</caption>
<thead class="">
<tr class=""><th scope="col" class="">yyyymmdd</th><th scope="col" class="">load good</th><th scope="col" class="">load avg</th><th scope="col" class="">load bad</th><th scope="col" class="">load p75</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">20230101</td><td class="">8.71</td><td class="">40.55</td><td class="">50.7</td><td class="">10400</td></tr><tr class=""><td class="">20221201</td><td class="">9.16</td><td class="">43.33</td><td class="">47.47</td><td class="">9500</td></tr><tr class=""><td class="">20221101</td><td class="">10.92</td><td class="">46.62</td><td class="">42.42</td><td class="">8700</td></tr>
</tbody>
</table>
</div>
</div>
<p><a href="https://console.cloud.google.com/bigquery?sq=1051247446620:f9d891fd8dc149fca5059508d999d559" target="_blank" rel="external noopener noreferrer">Here is the query.</a> Try it out!</p>
<h3 id="h-comparison-with-other-news-websites" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-comparison-with-other-news-websites"><span aria-hidden="true">#</span></a> Comparison with other news websites</h3>
<p>We can also compare Il Tirreno to other <a href="https://en.wikipedia.org/wiki/List_of_newspapers_in_Italy#Regional/local">local newspapers</a>:</p>
<ul>
<li><a href="https://www.lanazione.it/">La Nazione</a>, which has the highest print circulation in Tuscany (~1.85 times the one of Il Tirreno).</li>
<li><a href="https://www.lanuovasardegna.it/">La Nuova Sardegna</a>, which is roughly as popular in Sardinia as Il Tirreno is in Tuscany.</li>
</ul>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
Il Tirreno and La Nuova Sardegna are published by the <a href="https://www.grupposae.it/">same editorial group</a>. If we have a look at their websites, we see they have a similar layout and probably share some components of their tech stack. Notice how the TTFB is not too dissimilar between the two websites (see tables below).</p>
</div>
</div>
<p>The majority of the traffic to these news websites comes from phones using a 4G connection. Desktop traffic is only 10-15%. Tablet traffic is negligible.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>Traffic over time (% of users)</caption>
<thead class="">
<tr class=""><th scope="col" class="">yyyymmdd</th><th scope="col" class="">Domain</th><th scope="col" class="">Phone</th><th scope="col" class="">Desktop</th><th scope="col" class="">Tablet</th><th scope="col" class="">3G</th><th scope="col" class="">4G</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">20230101</td><td class="">iltirreno.it</td><td class="">88.56</td><td class="">8.57</td><td class="">2.84</td><td class="">4.9</td><td class="">95.07</td></tr><tr class=""><td class="">20230101</td><td class="">lanazione.it</td><td class="">87.2</td><td class="">9.87</td><td class="">2.9</td><td class="">4.78</td><td class="">95.19</td></tr><tr class=""><td class="">20230101</td><td class="">lanuovasardegna.it</td><td class="">87.31</td><td class="">9.34</td><td class="">3.35</td><td class="">4.81</td><td class="">95.19</td></tr><tr class=""><td class="">20221201</td><td class="">iltirreno.it</td><td class="">85.36</td><td class="">11.38</td><td class="">3.2</td><td class="">4.74</td><td class="">95.2</td></tr><tr class=""><td class="">20221201</td><td class="">lanazione.it</td><td class="">88.57</td><td class="">8.54</td><td class="">2.85</td><td class="">4.91</td><td class="">95.05</td></tr><tr class=""><td class="">20221201</td><td class="">lanuovasardegna.it</td><td class="">86.46</td><td class="">9.9</td><td class="">3.64</td><td class="">5.0</td><td class="">95.0</td></tr><tr class=""><td class="">20221101</td><td class="">iltirreno.it</td><td class="">79.84</td><td class="">16.24</td><td class="">3.93</td><td class="">4.2</td><td class="">95.81</td></tr><tr class=""><td class="">20221101</td><td class="">lanazione.it</td><td class="">85.49</td><td class="">11.6</td><td class="">2.85</td><td class="">4.66</td><td class="">95.28</td></tr><tr class=""><td class="">20221101</td><td class="">lanuovasardegna.it</td><td class="">87.09</td><td class="">9.66</td><td class="">3.23</td><td class="">5.17</td><td class="">94.81</td></tr>
</tbody>
</table>
</div>
</div>
<p><a href="http://iltirreno.it/">iltirreno.it</a> and <a href="http://lanuovasardegna.it/">lanuovasardegna.it</a> have a similar TTFB. These newspapers are published by the <a href="https://www.grupposae.it/">same publisher</a>, so I guessed their websites have a similar codebase and maybe share the same infrastructure. The TTFB of <a href="http://lanazione.it/">lanazione.it</a> is 2-3 better than the other two.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>TTFB over time (% of users; 75th percentile in ms)</caption>
<thead class="">
<tr class=""><th scope="col" class="">yyyymmdd</th><th scope="col" class="">Domain</th><th scope="col" class="">TTFB good</th><th scope="col" class="">TTFB avg</th><th scope="col" class="">TTFB bad</th><th scope="col" class="">TTFB p75</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">20230101</td><td class="">iltirreno.it</td><td class="">57.24</td><td class="">26.72</td><td class="">16.0</td><td class="">1300</td></tr><tr class=""><td class="">20230101</td><td class="">lanazione.it</td><td class="">87.27</td><td class="">9.49</td><td class="">3.21</td><td class="">400</td></tr><tr class=""><td class="">20230101</td><td class="">lanuovasardegna.it</td><td class="">57.18</td><td class="">29.72</td><td class="">13.0</td><td class="">1200</td></tr><tr class=""><td class="">20221201</td><td class="">iltirreno.it</td><td class="">50.89</td><td class="">30.13</td><td class="">18.92</td><td class="">1500</td></tr><tr class=""><td class="">20221201</td><td class="">lanazione.it</td><td class="">88.22</td><td class="">8.79</td><td class="">2.98</td><td class="">400</td></tr><tr class=""><td class="">20221201</td><td class="">lanuovasardegna.it</td><td class="">54.88</td><td class="">30.27</td><td class="">14.73</td><td class="">1300</td></tr><tr class=""><td class="">20221101</td><td class="">iltirreno.it</td><td class="">45.59</td><td class="">33.34</td><td class="">21.04</td><td class="">1600</td></tr><tr class=""><td class="">20221101</td><td class="">lanazione.it</td><td class="">85.35</td><td class="">10.57</td><td class="">4.05</td><td class="">500</td></tr><tr class=""><td class="">20221101</td><td class="">lanuovasardegna.it</td><td class="">54.92</td><td class="">31.03</td><td class="">13.99</td><td class="">1200</td></tr>
</tbody>
</table>
</div>
</div>
<p>The load event shows less variation across the three newspapers. It is quite high, probably because of ads and other third party scripts.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>load event over time (% of users; 75th percentile in ms)</caption>
<thead class="">
<tr class=""><th scope="col" class="">yyyymmdd</th><th scope="col" class="">Domain</th><th scope="col" class="">load good</th><th scope="col" class="">load avg</th><th scope="col" class="">load bad</th><th scope="col" class="">load</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">20230101</td><td class="">iltirreno.it</td><td class="">8.71</td><td class="">40.55</td><td class="">50.7</td><td class="">10400</td></tr><tr class=""><td class="">20230101</td><td class="">lanazione.it</td><td class="">13.08</td><td class="">32.53</td><td class="">54.38</td><td class="">12900</td></tr><tr class=""><td class="">20230101</td><td class="">lanuovasardegna.it</td><td class="">8.93</td><td class="">36.53</td><td class="">54.49</td><td class="">11400</td></tr><tr class=""><td class="">20221201</td><td class="">iltirreno.it</td><td class="">9.16</td><td class="">43.33</td><td class="">47.47</td><td class="">9500</td></tr><tr class=""><td class="">20221201</td><td class="">lanazione.it</td><td class="">14.87</td><td class="">34.37</td><td class="">50.78</td><td class="">12200</td></tr><tr class=""><td class="">20221201</td><td class="">lanuovasardegna.it</td><td class="">9.14</td><td class="">38.91</td><td class="">51.9</td><td class="">10900</td></tr><tr class=""><td class="">20221101</td><td class="">iltirreno.it</td><td class="">10.92</td><td class="">46.62</td><td class="">42.42</td><td class="">8700</td></tr><tr class=""><td class="">20221101</td><td class="">lanazione.it</td><td class="">19.34</td><td class="">34.34</td><td class="">46.33</td><td class="">11300</td></tr><tr class=""><td class="">20221101</td><td class="">lanuovasardegna.it</td><td class="">9.3</td><td class="">40.9</td><td class="">49.81</td><td class="">10300</td></tr>
</tbody>
</table>
</div>
</div>
<p>Cumulative Layout Shift sits at around 70% for everyone. This means there aren’t too many layout shifts in any of these websites.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption>CLS over time (% of users)</caption>
<thead class="">
<tr class=""><th scope="col" class="">yyyymmdd</th><th scope="col" class="">Domain</th><th scope="col" class="">CLS good</th><th scope="col" class="">CLS avg</th><th scope="col" class="">CLS bad</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">20230101</td><td class="">iltirreno.it</td><td class="">72.18</td><td class="">16.48</td><td class="">11.34</td></tr><tr class=""><td class="">20230101</td><td class="">lanazione.it</td><td class="">73.04</td><td class="">18.08</td><td class="">8.89</td></tr><tr class=""><td class="">20230101</td><td class="">lanuovasardegna.it</td><td class="">70.39</td><td class="">16.62</td><td class="">12.96</td></tr><tr class=""><td class="">20221201</td><td class="">iltirreno.it</td><td class="">73.17</td><td class="">15.98</td><td class="">10.85</td></tr><tr class=""><td class="">20221201</td><td class="">lanazione.it</td><td class="">70.66</td><td class="">19.42</td><td class="">9.93</td></tr><tr class=""><td class="">20221201</td><td class="">lanuovasardegna.it</td><td class="">67.22</td><td class="">18.63</td><td class="">14.17</td></tr><tr class=""><td class="">20221101</td><td class="">iltirreno.it</td><td class="">73.05</td><td class="">17.67</td><td class="">9.28</td></tr><tr class=""><td class="">20221101</td><td class="">lanazione.it</td><td class="">70.21</td><td class="">18.45</td><td class="">11.31</td></tr><tr class=""><td class="">20221101</td><td class="">lanuovasardegna.it</td><td class="">68.24</td><td class="">17.36</td><td class="">14.42</td></tr>
</tbody>
</table>
</div>
</div>
<p><a href="https://console.cloud.google.com/bigquery?sq=1051247446620:3e373ceeeeba47888f45305c2cc84555" target="_blank" rel="external noopener noreferrer">Here is the query.</a> Try it out!</p>
<h2 id="h-webpagetest" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-webpagetest"><span aria-hidden="true">#</span></a> WebPageTest</h2>
<div class="callout callout--tip">
<div class="callout__content">
<p>💡 —
This part of the article introduces WebPageTest and briefly describes its main sections. If you already know WebPageTest, you can probably skip this section.</p>
</div>
</div>
<p>We know that most visits to <a href="http://iltirreno.it/">iltirreno.it</a> during the month of January 2023 came from phones using a 4G connection.</p>
<p>The CrUX dataset provides us with performance metrics about those visits. In other words, it tells us <strong>how much</strong> a website is fast/slow <strong>overall</strong>.</p>
<p>However, CrUX doesn’t tell us <strong>why</strong> the website is fast/slow. In order to answer that question, we need to know:</p>
<ul>
<li>How many HTTP requests are made by the browser.</li>
<li>Which HTTP requests are problematic for web performance.</li>
<li>Which resources are cached, and for how long.</li>
<li>How much HTML, CSS, JS is on a given page.</li>
<li>How much time the main thread spends executing JS.</li>
<li>Which fonts are used, what’s their file format, how they are served.</li>
<li>How many images, audios, videos are on a page, where they are hosted, how big they are.</li>
</ul>
<p>The best tool to give us all this information is <a href="https://www.webpagetest.org/">WebPageTest</a>.</p>
<h3 id="h-launching-an-audit" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-launching-an-audit"><span aria-hidden="true">#</span></a> Launching an audit</h3>
<p>WebPageTest lets us choose the location and browser we want to use to test a website.<br />
Since we saw from CrUX that the majority of visits to <a href="http://iltirreno.it/">iltirreno.it</a> came from Italy and from phones, we pick Milan as the test location, and Chrome running on a low-mid tier phone as the browser.</p>
<p>In several articles about web performance, Moto G4 is often picked to run a performance audit. I argue that nowadays most users—or at least, most of the visitors of <a href="http://iltirreno.it/">iltirreno.it</a>—have a more powerful device. So I chose a Google Pixel 2 XL for this test (here is a <a href="https://www.gsmarena.com/compare.php3?idPhone1=8103&idPhone2=8720&idPhone3=10147">comparison</a> between Moto G4, Google Pixel 2 XL, and my current smartphone).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676822645/iltirreno-webpagetest-start-test_fgeb7g.png">https://res.cloudinary.com/jackdbd/image/upload/v1676822645/iltirreno-webpagetest-start-test_fgeb7g.png</a></p>
<p>As we can see from the word <code>EC2</code>, the test doesn’t really run on the Google Pixel 2 XL, but on an AWS EC2 instance. The phone is emulated by throttling the CPU of the EC2 instance (by a factor of 4, <a href="https://github.com/WPO-Foundation/webpagetest/blob/master/www/settings/mobile_devices.ini">see here</a>) and its connectivity (<a href="https://github.com/WPO-Foundation/webpagetest/blob/master/www/settings/connectivity.ini.sample">see here</a>).</p>
<p>It’s always a good idea to perform multiple runs (here 3, but the more the better), and to test the website both as a first-time visitor (First View) and as a repeat visitor (Repeat View). This way we can evaluate the effect of caching on the website’s performance. We can also assign a label to each test. This is particularly useful when we need to compare the results from two different tests.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676824767/webpagest-test-settings-3-runs_cq0txl.png">https://res.cloudinary.com/jackdbd/image/upload/v1676824767/webpagest-test-settings-3-runs_cq0txl.png</a></p>
<p>There are other tabs as we can see from the screenshot above. We’ll talk about them later.</p>
<h3 id="h-navigating-the-test-result" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-navigating-the-test-result"><span aria-hidden="true">#</span></a> Navigating the test result</h3>
<p>A WebPageTest test result contains many sections, dropdown menus, views, charts. It can be quite an intimidating experience for new users. The interface improved a lot in the last few months, but the problem is that assessing web performance requires a lot of information, so any web performance tool has to give us a lot of data to analyze.</p>
<p>I think the best way to start analyzing a WebPageTest test result is by selecting <strong>Performance Summary</strong> in the <strong>View</strong> dropdown. Here we can find a quick overview of a website’s performance, and links to pages that focus on individual details. We can also export the results of the test. For example, we can download the <a href="https://www.giacomodebidda.com/posts/generate-and-view-har-files/">HAR file</a> to analyze the HTTP requests using other tools.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676827660/webpagetest-result-performance-summary_bwwvtp.png">https://res.cloudinary.com/jackdbd/image/upload/v1676827660/webpagetest-result-performance-summary_bwwvtp.png</a></p>
<p>One of the first sections we see in Performance Summary is <strong>Real User Measurements</strong>. This contains data returned from CrUX. I think WebPageTest calls the <a href="https://developer.chrome.com/docs/crux/api/">CrUX API</a>, which might not return the exact same data of the queries I ran against the CrUX BigQuery dataset. This would explain why the numbers are slightly different.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676828779/iltirreno-webpagetest-crux_wfprad.png">https://res.cloudinary.com/jackdbd/image/upload/v1676828779/iltirreno-webpagetest-crux_wfprad.png</a></p>
<p>From the CrUX screenshot we can see that both FCP and LCP improve for repeat visits (light blue triangle). This suggests that some resources are cached. We’ll see later which ones.</p>
<p>The <strong>Individual Runs</strong> section contains the <strong>Waterfall</strong>, a chart visualizing all HTTP requests that the browser made when navigating to the URL we choose to test.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676830303/webpagetest-waterfall-thumbnail_tzzwly.png">https://res.cloudinary.com/jackdbd/image/upload/v1676830303/webpagetest-waterfall-thumbnail_tzzwly.png</a></p>
<p>In fact, there are several waterfall charts. Two for each run, given that we checked <strong>First View and Repeat View</strong> when we launched the test.</p>
<p>The waterfall charts here are just thumbnails. But if we click on one of them, WebPageTest takes us to the <strong>Details</strong> section. There we can interact with the <strong>Waterfall View</strong> of that particular test run and first/repeat view.</p>
<p>Next to each waterfall thumbnail we find a screenshot that WebPageTest took at the end of that particular test run. Then, next to the screenshot, two links, one to the <strong>Filmstrip View</strong>, another one to a video that shows how the browser loaded the page during the test run.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676833027/iltirreno-webpagetest-video_i0cloz.png">https://res.cloudinary.com/jackdbd/image/upload/v1676833027/iltirreno-webpagetest-video_i0cloz.png</a></p>
<h3 id="h-video-and-filmstrip-view" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-video-and-filmstrip-view"><span aria-hidden="true">#</span></a> Video and Filmstrip View</h3>
<p>Let’s start from the video and recap what it shows. It’s the experience that a user has when visiting <a href="http://iltirreno.it/versilia">iltirreno.it/versilia</a> for the first time, using a Google Pixel 2 XL, on a 4G connection from Milan.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1676833384/google-pixel2-xl-with-consent-banner_qdjm4w.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1676833384/google-pixel2-xl-with-consent-banner_qdjm4w.mp4</a></p>
<div class="callout callout--tip">
<div class="callout__content">
<p>💡 —
This is a video, not a GIF. You should <a href="https://robertcooper.me/post/stop-using-gifs">never use a GIF</a> for an animated clip.</p>
</div>
</div>
<p>The <strong>Filmstrip View</strong> shows a few frames of the video, and can be configured to highlight the Largest Contentful Paint (LCP) and any layout shifts.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676836761/iltirreno-webpagetest-filmstrip-view_d0kdeh.png">https://res.cloudinary.com/jackdbd/image/upload/v1676836761/iltirreno-webpagetest-filmstrip-view_d0kdeh.png</a></p>
<p>Even just by watching the video, we can start asking ourselves a few questions and formulating a few hypotheses:</p>
<ul>
<li>The page takes roughly 3.5 seconds to show any content. Remember, we are testing on a 4G connection from Milan, so this time seems a bit high for just a few lines of text and two images (one of which barely appears at the bottom of the viewport).</li>
<li>There are two text elements that are not immediately visible. The title of the newspaper (IL TIRRENO) and the subtitle of the only article that appears in the viewport (Corinne, Marzia, Priscilla…). Could this be a case of <a href="https://fonts.google.com/knowledge/glossary/foit">Flash Of Invisible Text (FOIT)</a>?</li>
<li>The images do not cause any layout shift. This is good. Layout shifts are bad for user experience.</li>
<li>An annoying consent banner appears and covers almost the entire viewport. This is expected. In order to comply with the <a href="https://edps.europa.eu/sites/edp/files/publication/dir_2002_58_en.pdf">2002 ePrivacy Directive</a>, any website that collects personal data from the user must obtain their explicit consent. We’ll spend a few words about consent banners, since they can be problematic when assessing the performance of a website.</li>
</ul>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
The ePrivacy Directive is often called EU Cookie Law. This, from a technical standpoint, is a bit imprecise. This EU directive cites browser cookies as an example, but it applies to all ways that a website—or any of its third party scripts—can use to track a user. For example, if a website tracks its users' behavior using Local Storage or IndexedDB, it still must obtain their explicit consent.</p>
</div>
</div>
<h3 id="h-detected-technologies" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-detected-technologies"><span aria-hidden="true">#</span></a> Detected Technologies</h3>
<p>We could now open Chrome DevTools, have a look at the <strong>Element</strong> panel and the <strong>Sources</strong> panel, and try to understand the tech stack behind this website. But WebPageTest has a <strong>Detected Technologies</strong> section that saves us a bit of time. It’s not always accurate and can be a bit misleading at times (even when it says <code>Detection confidence 100</code>), so it’s always a good idea to verify these findings in the Waterfall View.</p>
<p>WebPageTest detected that this website is built with Next.js, served by Nginx, and cached on AWS CloudFront.<br />
It also detected Webpack, React, Material UI, Emotion, Lodash, core-js.<br />
Finally, it detected these services:</p>
<ul>
<li>Iubenda</li>
<li>Google Analytics</li>
<li>Google Publisher Tag</li>
<li>Google Tag Manager</li>
</ul>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
<a href="https://www.iubenda.com/en/">Iubenda</a> is a service that deals with legal obligations. It generates cookie policies, privacy policies, terms of service, etc.</p>
</div>
</div>
<p>News websites make money by selling advertising space. So it’s no surprise that there are snippets for ads and trackers on this website. However, these third party scripts needs to be downloaded, parsed and executed. They can be detrimental to performance if they are too many.</p>
<h3 id="h-breakdown-by-domains" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-breakdown-by-domains"><span aria-hidden="true">#</span></a> Breakdown by domains</h3>
<p>WebPageTest has a section called <strong>Domains</strong> that shows us the requests breakdown, by domain. Down below we can see the number of requests on a first visit. I clipped the table to show only the first 5 domains by number of requests.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676844698/iltirreno-domains-requests_wxtwc0.png">https://res.cloudinary.com/jackdbd/image/upload/v1676844698/iltirreno-domains-requests_wxtwc0.png</a></p>
<p>In the same section we can also see the amount of bytes fetched, by domain. This pie chart tells us that, on a first visit, a bit more than 40% of the bytes come from domains other than <a href="http://www.iltirreno.it/">www.iltirreno.it</a>. Again, I clipped the table to show only the first 5 domains by number of bytes fetched.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676844697/iltirreno-domains-bytes_khpzlw.png">https://res.cloudinary.com/jackdbd/image/upload/v1676844697/iltirreno-domains-bytes_khpzlw.png</a></p>
<p>The situation improves on a repeat visit. The browser now makes only 11 requests.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676846145/iltirreno-domains-requests-repeat_oixt0w.png">https://res.cloudinary.com/jackdbd/image/upload/v1676846145/iltirreno-domains-requests-repeat_oixt0w.png</a></p>
<p>The amout of bytes downloaded is also significantly reduced and comes almost entirely from <a href="http://www.iltirreno.it/">www.iltirreno.it</a>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676846145/iltirreno-domains-bytes-repeat_idsnnk.png">https://res.cloudinary.com/jackdbd/image/upload/v1676846145/iltirreno-domains-bytes-repeat_idsnnk.png</a></p>
<h3 id="h-breakdown-by-mime-type" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-breakdown-by-mime-type"><span aria-hidden="true">#</span></a> Breakdown by MIME type</h3>
<p>A section of WebPageTest that can be quite revealing at times—and it is in this case—is the <strong>Content</strong> section, which breaks down requests and bytes by MIME type. Here I focus on the bytes, since they offer us a clear picture of what’s going on.</p>
<p>Here are the bytes downloaded on a first visit.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676846758/iltirreno-mime-type-bytes-first-view_mquod4.png">https://res.cloudinary.com/jackdbd/image/upload/v1676846758/iltirreno-mime-type-bytes-first-view_mquod4.png</a></p>
<p>And here are the bytes downloaded on a repeat visit.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676846758/iltirreno-mime-type-bytes-repeat-view_g26b5u.png">https://res.cloudinary.com/jackdbd/image/upload/v1676846758/iltirreno-mime-type-bytes-repeat-view_g26b5u.png</a></p>
<p>We can draw the following conclusions by looking at these two pie charts:</p>
<ol>
<li>A lot of JS is downloaded on a first visit. I find this a bit odd, since this website is built with Next.js, which can render pages on the server at runtime (<a href="https://nextjs.org/docs/basic-features/pages#server-side-rendering">server-side rendering</a>) or even pre-render them at build time (<a href="https://nextjs.org/docs/basic-features/pages#static-generation">static generation</a>). I would have expected to see more HTML and CSS.</li>
<li>Images and fonts images represent a significant share of the pie. This could be perfectly fine, it’s a news website after all. It’s reasonable to expect a lot of images and maybe some custom fonts.</li>
<li>Almost only HTML is downloaded on a repeat visit. This is not surprising. HTML is rarely cached (or if it is, not for much long), while immutable assets such as fonts and images are usually cached for a long time. The browser downloaded them on the first visit, so it doesn’t need to download them again on subsequent visits.</li>
</ol>
<h3 id="h-waterfall-view" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-waterfall-view"><span aria-hidden="true">#</span></a> Waterfall View</h3>
<p>Until now, all sections of WebPageTest have been quite straightforward to discuss. The <strong>Details</strong> section is a whole different beast though, since it contains the <strong>Waterfall View</strong>.</p>
<p>The Waterfall View is without a doubt the most difficult visualization to understand in a WebPageTest test result. But arguably the most important to analyze. It contains:</p>
<ul>
<li>On the left, the complete list of HTTP requests made by the browser when loading the website.</li>
<li>On the right, the timeline of all HTTP requests. Each request is represented by a horizontal bar. Each bar is colored according to the type of resource this HTTP request is made for (HTML in blue, CSS in green, font in red, etc).</li>
<li>At the bottom, two <a href="https://datavizproject.com/data-type/stepped-line-graph/">stepped line graphs</a> for <strong>CPU Utilization</strong> and <strong>Bandwidth In</strong>, and two other charts for the <strong>Browser Main Thread</strong> and <strong>Long Tasks</strong>.</li>
<li>At the top, a legend that encodes the meaning of each color and icon used in the waterfall.</li>
</ul>
<p>Down below we can see the Waterfall View of <a href="http://iltirreno.it/versilia">iltirreno.it/versilia</a> on a <strong>first visit</strong>, for the <strong>second run</strong> of the WebPageTest audit. I decided to show only the first and last ten HTTP requests to make it easier to comprehend. I blurred out all other requests in between.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676884072/iltirreno-waterfall-10-10_ko4r1b.png">https://res.cloudinary.com/jackdbd/image/upload/v1676884072/iltirreno-waterfall-10-10_ko4r1b.png</a></p>
<p>Throughout the rest of this article I will describe a few characteristics of <a href="http://iltirreno.it/versilia">iltirreno.it/versilia</a> by pointing them out in the waterfall. I will show only a few HTTP requests at a time, to avoid overwhelming you with too much information.</p>
<p>If you find yourself struggling making sense of this chart, definitely take a pause from reading this article and go study Matt Hobbs’ magnum opus <a href="https://nooshu.com/blog/2019/10/02/how-to-read-a-wpt-waterfall-chart/">How to read a WebPageTest Waterfall View chart</a>. Also, the knowledge you might gain from reading my article is limited to the characteristics of <a href="http://iltirreno.it/versilia">iltirreno.it/versilia</a>, while Matt’s article covers many more scenarios and web performance issues.</p>
<h2 id="h-html-css-js" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-html-css-js"><span aria-hidden="true">#</span></a> HTML, CSS, JS</h2>
<p>The first request made when visiting <a href="https://www.iltirreno.it/versilia">iltirreno.it/versilia</a> is for its HTML document. This file is hosted on the domain <code>www.iltirreno.it</code>.</p>
<p>Since this is the first resource requested, the browser has to first perform a <b><span style="color:#4ca2a8">DNS lookup</span></b>, establish an <b><span style="color:#ffa24c">HTTP connection</span></b>, complete a <b><span style="color:#dd66e8">TLS handshake (aka SSL negotiation)</span></b>.<br />
Then it can finally <b><span style="color:#c9d5e8">request the HTML</span></b> and <b><span style="color:#8cb2e6">download it</span></b>.</p>
<p>Down below we can see the request details on a <strong>first visit</strong>, on the <strong>second run</strong> of the WebPageTest audit.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676886070/iltirreno-waterfall-html-request-details_oexax3.png">https://res.cloudinary.com/jackdbd/image/upload/v1676886070/iltirreno-waterfall-html-request-details_oexax3.png</a></p>
<h3 id="h-cache-control-for-the-html-document" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-cache-control-for-the-html-document"><span aria-hidden="true">#</span></a> Cache-Control for the HTML document</h3>
<p>Down here we can see the request response (still for a first visit, on the second run of the WebPageTest audit).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676886070/iltirreno-waterfall-html-request-response_zjhwlk.png">https://res.cloudinary.com/jackdbd/image/upload/v1676886070/iltirreno-waterfall-html-request-response_zjhwlk.png</a></p>
<p>Why did I pick the <strong>second run</strong> and not the first one? Because it shows that the HTML, even on a first visit, it is served from a cache. To be more precise, this HTML file was cached 24 seconds ago on the AWS CloudFront CDN:</p>
<pre class="language-txt"><code class="language-txt">age: 24
x-cache: Hit from cloudfront</code></pre>
<p>A CDN acts as a <a href="https://learn.microsoft.com/en-us/azure/cdn/cdn-how-caching-works#introduction-to-caching">shared cache</a>, so a file that is requested by one user can be accessed later by other users. In this performance audit a “user” is any one of the WebPageTest test runs. The HTML document is cached on AWS CloudFront after the first run, so the second run was able to download it from the CDN without requesting it from the <a href="https://www.cloudflare.com/learning/cdn/glossary/origin-server/">origin server</a> (Nginx in this case).</p>
<p>We can verify this by looking at the HTTP response headers sent on the <strong>first run</strong>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676888170/iltirreno-waterfall-html-request-response-first-run_faog4s.png">https://res.cloudinary.com/jackdbd/image/upload/v1676888170/iltirreno-waterfall-html-request-response-first-run_faog4s.png</a></p>
<p>That <code>x-cache: Miss from cloudfront</code> tells us that on a <strong>first visit</strong>, on the <strong>first run</strong>, the browser had to:</p>
<ol>
<li>Do DNS lookup + HTTP connection + TLS handshake (as during a first visit, on the second run)</li>
<li>Look for the HTML in CloudFront and fail to find it</li>
<li>Request the HTML from Nginx</li>
<li>Wait for Nginx to <a href="https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/">pass the request</a> to a Node.js server</li>
<li>Wait for the Node.js server to server-side render the (dynamic) HTML with Next.js</li>
</ol>
<div class="callout callout--tip">
<div class="callout__content">
<p>💡 —
A nice tool for analyzing HTTP response headers is <a href="https://redbot.org/">REDbot</a>.</p>
</div>
</div>
<p>We can clearly see that during a <strong>first visit</strong>, on the <strong>first run</strong>, the DNS lookup, HTTP connection, TLS handshake take roughly the same amount of time than during a <strong>first visit</strong>, on the <strong>second run</strong>. The additional steps that the browser has to perform during the first run add ~1200ms to the Time To First Byte (TTFB).<br />
On the first run, the whole process (the combined length of the teal, orange, magenta, blue bars) takes 2368ms instead of the 905ms of the second run.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676888170/iltirreno-waterfall-html-request-details-first-run_jv9lsu.png">https://res.cloudinary.com/jackdbd/image/upload/v1676888170/iltirreno-waterfall-html-request-details-first-run_jv9lsu.png</a></p>
<p>Is there anything we can do to improve this? Maybe, but without knowing the source code of the backend it’s hard to tell.</p>
<p>A quick, but questionable fix, would be to increase the amount of time the HTML is considered fresh in the CloudFront cache. Right now, that time is set to 60 seconds:</p>
<pre class="language-txt"><code class="language-txt">Cache-Control: s-maxage=60</code></pre>
<p>The HTML pages of a news website are dynamic, and the front page particularly so. New or updated articles will not show up until the cached version of the HTML page is expired. So we can’t cache it for too long.</p>
<p>But maybe 60 seconds are too few. I guess we could cache the HTML document for 5-15 minutes, at the risk of serving a stale version of the page to a few users.</p>
<p>Also, we could tweak <a href="https://web.dev/stale-while-revalidate/">stale-while-revalidate</a>, so to serve a stale version of the page, while looking for a new version in the background.</p>
<p>For example the <code>Cache-Control</code> header below would instruct the browser to consider the HTML fresh for 300 seconds, either if it finds it in a shared cache (<code>s-maxage</code>) or in a private cache (<code>max-age</code>), and keep serving a stale version for an <strong>additional</strong> 600 seconds, while at the same time looking for a new version in the background (revalidating).</p>
<pre class="language-txt"><code class="language-txt">Cache-Control: max-age=300, s-maxage=300, stale-while-revalidate=600</code></pre>
<p>This means that in the worst case scenario a visitor would <strong>not</strong> receive the latest HTML page (generated by Next.js), and instead receive a HTML page which was cached 15 minutes ago (in CloudFront). This would be perfectly reasonable for a website that doesn’t update too many times during the day. But for a news website I’m not so sure.</p>
<p>Another thing we notice is that the page is not exactly light: 507.3 KB (uncompressed). Why is that? WebPageTest can’t really help to answer this question, but Chrome DevTools and a couple of other tools can.</p>
<h3 id="h-inspecting-the-html-with-chrome-devtools" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-inspecting-the-html-with-chrome-devtools"><span aria-hidden="true">#</span></a> Inspecting the HTML with Chrome DevTools</h3>
<div class="callout callout--tip">
<div class="callout__content">
<p>💡 —
When you want to inspect a web page in Chrome DevTools, always do it in <strong>incognito mode</strong>. This way no cookies or Chrome extensions will affect the page you're examining.</p>
</div>
</div>
<p>If we have a look at the <strong>Elements</strong> panel in Chrome DevTools we notice a lot of <code><script></code> and <code><style></code> tags in the <code><head></code> of the HTML document.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>head</span><span class="token punctuation">></span></span>
<span class="token comment"><!-- some meta tags, some preloads --></span>
<span class="token comment"><!-- a lot of these --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span> <span class="token attr-name">defer</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span><span class="token punctuation">"</span></span> <span class="token attr-name">src</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/_next/static/chunks/some-hash.js<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token script"></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span> <span class="token attr-name">data-emotion</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>css kqgxn0<span class="token punctuation">"</span></span> <span class="token attr-name">data-s</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span><span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<!-- CSS rulesets -->
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>head</span><span class="token punctuation">></span></span></code></pre>
<h3 id="h-unused-css-and-js" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-unused-css-and-js"><span aria-hidden="true">#</span></a> Unused CSS and JS</h3>
<p>All those <code><script></code> tags are created by <a href="https://nextjs.org/learn/foundations/how-nextjs-works/code-splitting">Next.js code splitting</a>, while the <code><style></code> tags by <a href="https://github.com/emotion-js/emotion">Emotion</a>, a CSS-in-JS library.</p>
<p>We can use the <strong>Coverage</strong> panel to view how much JS and CSS is actually used.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676899865/iltirreno-coverage-with-consent-banner_gmlrdc.png">https://res.cloudinary.com/jackdbd/image/upload/v1676899865/iltirreno-coverage-with-consent-banner_gmlrdc.png</a></p>
<p>The Next.js JS code chunks have <code>_next/static/chunks</code> in their URL. We can see they generate some code which is not actually used, but it doesn’t seem that much. Third party scripts such as securepubads, iubenda and googletagmanager waste many more bytes.</p>
<p><a href="http://iltirreno.it/">iltirreno.it</a> doesn’t ship source maps in its production build, so the only JavaScript code we can look at in the <strong>Sources</strong> panel in Chrome DevTools is something like this:</p>
<pre class="language-js"><code class="language-js"><span class="token punctuation">(</span>self<span class="token punctuation">.</span>webpackChunk_N_E <span class="token operator">=</span> self<span class="token punctuation">.</span>webpackChunk_N_E <span class="token operator">||</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">[</span><span class="token number">129</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
<span class="token number">21924</span><span class="token operator">:</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token parameter">t<span class="token punctuation">,</span> e<span class="token punctuation">,</span> r</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token string">"use strict"</span><span class="token punctuation">;</span>
<span class="token keyword">var</span> o <span class="token operator">=</span> <span class="token function">r</span><span class="token punctuation">(</span><span class="token number">40210</span><span class="token punctuation">)</span>
<span class="token punctuation">,</span> n <span class="token operator">=</span> <span class="token function">r</span><span class="token punctuation">(</span><span class="token number">55559</span><span class="token punctuation">)</span>
<span class="token punctuation">,</span> i <span class="token operator">=</span> <span class="token function">n</span><span class="token punctuation">(</span><span class="token function">o</span><span class="token punctuation">(</span><span class="token string">"String.prototype.indexOf"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
t<span class="token punctuation">.</span><span class="token function-variable function">exports</span> <span class="token operator">=</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token parameter">t<span class="token punctuation">,</span> e</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">var</span> r <span class="token operator">=</span> <span class="token function">o</span><span class="token punctuation">(</span>t<span class="token punctuation">,</span> <span class="token operator">!</span><span class="token operator">!</span>e<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token string">"function"</span> <span class="token operator">===</span> <span class="token keyword">typeof</span> r <span class="token operator">&&</span> <span class="token function">i</span><span class="token punctuation">(</span>t<span class="token punctuation">,</span> <span class="token string">".prototype."</span><span class="token punctuation">)</span> <span class="token operator">></span> <span class="token operator">-</span><span class="token number">1</span> <span class="token operator">?</span> <span class="token function">n</span><span class="token punctuation">(</span>r<span class="token punctuation">)</span> <span class="token operator">:</span> r
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p>If JavaScript source maps were available, we could also visualize the code with tools such as <a href="https://github.com/webpack-contrib/webpack-bundle-analyzer">Webpack Bundle Analyzer</a> or <a href="https://bundle-buddy.com/">Bundle Buddy</a>, or even inspect performance traces that show us <a href="https://developer.chrome.com/blog/new-in-devtools-109/#performance">the actual JS function names</a>.</p>
<h3 id="h-inlining-css-a-questionable-choice" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-inlining-css-a-questionable-choice"><span aria-hidden="true">#</span></a> Inlining CSS: a questionable choice</h3>
<p>The reason we don’t see any <code><link></code> to some external stylesheet, is that <a href="https://github.com/emotion-js/emotion/issues/2092#issuecomment-725978316">there are never external stylesheets involved when working with Emotion</a>. This is a rather questionable choice.<br />
Inlining a <strong>small</strong> amount of CSS <a href="https://www.11ty.dev/docs/quicktips/inline-css/">can be a good idea</a>, but inlining all CSS <a href="https://calendar.perfplanet.com/2011/why-inlining-everything-is-not-the-answer/">almost definitely isn’t</a>. One of the reasons this is bad for performance is because we can’t cache this CSS independently from the HTML. Another one is that it makes the HTML page heavier. We can see this using <a href="https://www.debugbear.com/html-size-analyzer">HTML Size Analyzer</a>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676904401/iltirreno-html-size-analyzer_atbmi8.png">https://res.cloudinary.com/jackdbd/image/upload/v1676904401/iltirreno-html-size-analyzer_atbmi8.png</a></p>
<p>These violet bars tell us that almost 15% of the HTML page is in fact inlined CSS. 76KB KB (uncompressed) which we could save by using external stylesheets.</p>
<p><a href="https://css-tricks.com/a-thorough-analysis-of-css-in-js/#aa-style-injected-dom-styles">Many CSS-in-JS libraries</a> inject <code><style></code> tags in the DOM at runtime. However, there are some that instead inject <code><link></code> tags that reference external CSS bundles. <a href="https://github.com/callstack/linaria">Linaria</a> is one of these.</p>
<h3 id="h-the-price-to-pay-for-client-side-hydration" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-the-price-to-pay-for-client-side-hydration"><span aria-hidden="true">#</span></a> The price to pay for client-side hydration</h3>
<p>In the <code><body></code> we can see the HTML for the final page (after server-side rendering and client-side hydration), some JSON data, some third-party scripts.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>body</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>__next<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token comment"><!-- the actual page --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>__NEXT_DATA__<span class="token punctuation">"</span></span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>application/json<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript">
<span class="token operator"><</span><span class="token operator">!</span><span class="token operator">--</span> pageProps here <span class="token keyword">for</span> hydration <span class="token operator">--</span><span class="token operator">></span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span>
<span class="token comment"><!-- third-party scripts: ads, analytics, etc --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>google-analytics-script<span class="token punctuation">"</span></span> <span class="token attr-name">data-nscript</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>lazyOnload<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript">
<span class="token operator"><</span><span class="token operator">!</span><span class="token operator">--</span> <span class="token constant">JS</span> code <span class="token keyword">for</span> Google Analytics <span class="token operator">--</span><span class="token operator">></span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>body</span><span class="token punctuation">></span></span></code></pre>
<p>The JSON contained in <code>__NEXT_DATA__</code> is required by Next.js to perform client-side hydration. We can extract it by running this code in the Chrome DevTools console.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token function">copy</span><span class="token punctuation">(</span><span class="token function">$</span><span class="token punctuation">(</span><span class="token string">'#__NEXT_DATA__'</span><span class="token punctuation">)</span><span class="token punctuation">.</span>innerHTML<span class="token punctuation">)</span></code></pre>
<p>If we then go to <a href="https://www.debugbear.com/json-size-analyzer">JSON Size Analyzer</a> and paste this JSON in the text box, we can see it’s quite big.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676903146/iltirreno-json-size-analyzer_oam55y.png">https://res.cloudinary.com/jackdbd/image/upload/v1676903146/iltirreno-json-size-analyzer_oam55y.png</a></p>
<p>There ways to <a href="https://www.smashingmagazine.com/2021/05/reduce-data-sent-client-nextjs/#the-problem">reduce the JSON data</a> injected by Next.js, but they involve reorganizing data fetching in some ways. Not something we can do without having access to the source code.</p>
<h3 id="h-external-style-sheets" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-external-style-sheets"><span aria-hidden="true">#</span></a> External style sheets</h3>
<p>As I wrote before, most of the CSS on this web page is inlined in the <code><head></code>. There are only three external stylesheets.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676853530/iltirreno-waterfall-css_rucoqb.png">https://res.cloudinary.com/jackdbd/image/upload/v1676853530/iltirreno-waterfall-css_rucoqb.png</a></p>
<p>The first stylesheet (request 9) is hosted on <code>www.iltirreno.it</code> and contains only a few <a href="https://css-tricks.com/css-ruleset-terminology/">CSS rulesets</a> and the <code>@font-face</code> rules for the fonts. It’s just 1.8KB (uncompressed). The browser takes 560ms to receive the first byte of this file, and two additional milliseconds to download it.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676997348/iltirreno-css-chunk-next_j6kmx7.png">https://res.cloudinary.com/jackdbd/image/upload/v1676997348/iltirreno-css-chunk-next_j6kmx7.png</a></p>
<p>This stylesheet is named <code>f4fc52ca4d1d33b9.css</code> and can be found in the <code>/_next/static/css</code> directory. It is served with this <code>Cache-Control</code> header, which is probably <a href="https://nextjs.org/docs/going-to-production#caching">set by Next.js</a>.</p>
<pre class="language-txt"><code class="language-txt">cache-control: public, max-age=31536000, immutable</code></pre>
<p>The other two stylesheets are hosted on <code>cdnjs.cloudflare.com</code> and contain the CSS for <a href="https://github.com/kenwheeler/slick">this carousel</a>.<br />
There seems to be nothing to prevent us from hosting these stylesheets on <code>www.iltirreno.it</code>, so <a href="https://csswizardry.com/2019/05/self-host-your-static-assets/">we should do it</a>.</p>
<p>The <code>Cache-Control</code> header sent alongside <code>slick.min.css</code> and <code>slick-theme.min.css</code> is the following:</p>
<pre class="language-txt"><code class="language-txt">cache-control: public, max-age=30672000</code></pre>
<h2 id="h-fonts" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-fonts"><span aria-hidden="true">#</span></a> Fonts</h2>
<p><a href="http://iltirreno.it/">iltirreno.it</a> uses several <a href="https://developer.mozilla.org/en-US/docs/Learn/CSS/Styling_text/Web_fonts">web fonts</a>, so the browser has to download them. In the WebPageTest waterfall, we can see the HTTP requests for these fonts in red. The <b><span style="color:#f6cac5">light red</span></b> portion of the bar represents the time the browser takes to request a font, while the <b><span style="color:#e96859">dark red</span></b> one is the time it takes to fetch it.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676972374/iltirreno-waterfall-fonts_abkbpk.png">https://res.cloudinary.com/jackdbd/image/upload/v1676972374/iltirreno-waterfall-fonts_abkbpk.png</a></p>
<p>We can count eight requests for fonts. Seven of these fonts are hosted on <code>www.iltirreno.it</code>. One is hosted on <code>cdnjs.cloudflare.com</code>.</p>
<h3 id="h-cache-control-for-fonts" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-cache-control-for-fonts"><span aria-hidden="true">#</span></a> Cache-Control for fonts</h3>
<p>Font files should never change and should be treated as immutable assets. This means we can tell the browser to cache them for a long time (tipically 31536000 seconds, i.e. one year).</p>
<p>In fact, this is the <code>Cache-Control</code> header returned with the fonts hosted on <code>www.iltirreno.it</code>:</p>
<pre class="language-txt"><code class="language-txt">cache-control: public, max-age=31536000, immutable</code></pre>
<p>The <code>Cache-Control</code> header returned with the font hosted on <code>cdnjs.cloudflare.com</code> is slightly different:</p>
<pre class="language-txt"><code class="language-txt">cache-control: public, max-age=30672000</code></pre>
<p>I’m not sure why Cloudflare is not using <code>immutable</code> and caching the font for 355 days instead of one year. <span class="shrug-emoticon">¯\_(ツ)_/¯</span></p>
<h3 id="h-most-fonts-are-preloaded" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-most-fonts-are-preloaded"><span aria-hidden="true">#</span></a> Most fonts are preloaded</h3>
<p>Web fonts must be declared using CSS <code>@font-face</code> rules like this:</p>
<pre class="language-css"><code class="language-css"><span class="token atrule"><span class="token rule">@font-face</span></span> <span class="token punctuation">{</span>
<span class="token property">font-family</span><span class="token punctuation">:</span> Utopia-Regular<span class="token punctuation">;</span>
<span class="token property">src</span><span class="token punctuation">:</span> <span class="token url"><span class="token function">url</span><span class="token punctuation">(</span>/fonts/Utopia/Utopia-Regular.otf<span class="token punctuation">)</span></span> <span class="token property">format</span><span class="token punctuation">:</span><span class="token punctuation">(</span><span class="token string">"otf"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token property">font-display</span><span class="token punctuation">:</span> swap<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>This means that by default a browser has to make a <a href="https://developer.chrome.com/docs/lighthouse/performance/critical-request-chains/">chain request</a> to download a web font: it needs to request the CSS, download it, parse the <code>@font-face</code> rule, find out it needs to fetch a font (e.g. <code>Utopia-Regular.otf</code>), and finally request the font file and download it.</p>
<p>The reason it doesn’t happen on this web page is that the font is <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel/preload">preloaded</a> thanks to a browser hint placed in the <code><head></code>.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>head</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>preload<span class="token punctuation">"</span></span>
<span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/fonts/Utopia/Utopia-Regular.otf<span class="token punctuation">"</span></span>
<span class="token attr-name">as</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>font<span class="token punctuation">"</span></span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>font/otf<span class="token punctuation">"</span></span> <span class="token attr-name">crossorigin</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>anonymous<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>head</span><span class="token punctuation">></span></span></code></pre>
<p>All seven self-hosted fonts (i.e. fonts hosted on <code>www.iltirreno.it</code>) are preloaded.</p>
<p>The font hosted on <code>cdnjs.cloudflare.com</code> is not preloaded. The browser finds out it has to request <code>slick.woff</code> (the font file) when it has finished parsing the <code>@font-face</code> rule contained in <code>slick-theme.min.css</code>. That’s why we see the request for this font much lower in the waterfall (HTTP request 41).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676976801/font-request-chain_pdtzcl.png">https://res.cloudinary.com/jackdbd/image/upload/v1676976801/font-request-chain_pdtzcl.png</a></p>
<h3 id="h-font-file-formats" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-font-file-formats"><span aria-hidden="true">#</span></a> Font file formats</h3>
<p>We can find three different file formats on this page: otf, ttf, woff. There is an important absence here: woff2.</p>
<p>Woff2 uses <a href="https://www.w3.org/TR/WOFF2/">Brotli for compression</a>, so we could save a few KB on each font if we convert otf and ttf fonts to woff2.</p>
<p>I downloaded a few <code>.otf</code>/<code>.ttf</code> files and converted them to <code>.woff2</code> using <a href="https://cloudconvert.com/">this online converter</a>. Here are the results:</p>
<pre class="language-sh"><code class="language-sh">ncdu <span class="token number">1.15</span>.1 ~ Use the arrow keys to navigate, press ? <span class="token keyword">for</span> <span class="token builtin class-name">help</span>
--- /home/jack/Downloads/iltirreno/fonts ---------------------------------------------------------------
<span class="token number">20,0</span> KiB <span class="token punctuation">[</span><span class="token comment">#### ] ITC_Franklin_Gothic_Book_Condensed.otf</span>
<span class="token number">12,0</span> KiB <span class="token punctuation">[</span><span class="token comment">## ] ITC_Franklin_Gothic_Book_Condensed.woff2</span>
<span class="token number">48,0</span> KiB <span class="token punctuation">[</span><span class="token comment">##########] PoynterOldstyleDisplay-SemiBold.ttf</span>
<span class="token number">20,0</span> KiB <span class="token punctuation">[</span><span class="token comment">#### ] PoynterOldstyleDisplay-SemiBold.woff2</span>
<span class="token number">36,0</span> KiB <span class="token punctuation">[</span><span class="token comment">####### ] Utopia-Regular.otf</span>
<span class="token number">16,0</span> KiB <span class="token punctuation">[</span><span class="token comment">### ] Utopia-Regular.woff2</span></code></pre>
<p>The <a href="https://caniuse.com/?search=woff2">browser support for woff2</a> is almost universal nowadays, so there is no reason not to use this file format.</p>
<h3 id="h-flash-of-invisible-text-foit" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-flash-of-invisible-text-foit"><span aria-hidden="true">#</span></a> Flash Of Invisible Text (FOIT)</h3>
<p>There are two areas on this page where text is not rendered for a while. I’m including the video once more, so you don’t have to scroll back up.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1676833384/google-pixel2-xl-with-consent-banner_qdjm4w.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1676833384/google-pixel2-xl-with-consent-banner_qdjm4w.mp4</a></p>
<p>Let’s review the definition of Flash Of Invisible Text (FOIT) <a href="https://fonts.google.com/knowledge/glossary/foit">according to Google</a>:</p>
<blockquote>
<p>A “flash of invisible text” (FOIT) is the phenomenon in which a web page loads without the type appearing at all, before rendering to the intended typeface(s). This delay is caused by the web fonts being downloaded to the user’s device.</p>
</blockquote>
<p>I had a look at the HTML in the Chrome DevTools <strong>Elements</strong> panel. There I found out that <strong>IL TIRRENO</strong> was in fact an SVG, not a text. SVG is not a web font, so I don’t think this case qualifies as FOIT.</p>
<p>As for the subtitle starting with <em>Corinne, Marzia, Priscilla…</em>, I think it could be one of these cases:</p>
<ol>
<li>The subtitle is present in the initial, server-side rendered HTML. This means this <strong>is</strong> a FOIT, since the text is there but nothing is rendered.</li>
<li>The subtitle is not present in the initial HTML, and is added to the page after client-side hydration. This means this <strong>is not</strong> a FOIT, since in this test any one of the JS chunks gets executed after all fonts have finished downloading (see waterfall below, JS execution in pink).</li>
</ol>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1677003678/iltirreno-js-execution_f0p2af.png">https://res.cloudinary.com/jackdbd/image/upload/v1677003678/iltirreno-js-execution_f0p2af.png</a></p>
<h2 id="h-images" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-images"><span aria-hidden="true">#</span></a> Images</h2>
<p>Images are probably the biggest files on a web page. They are obviously important in a news website, so it makes sense to try and optimize them as much as possible.</p>
<p>The problem is that image optimization is hard and involves all of these steps:</p>
<ol>
<li>Serving the best available image <strong>format</strong> (e.g. AVIF, WebP) supported by your website visitors’ browser.</li>
<li>Picking the best <strong>resolution</strong> for <a href="https://archive.fosdem.org/2021/schedule/event/webperf_making_rum_responsive/">each device</a> used by your website visitors. This means taking into account: screen dimensions, screen quality, device pixel ratio, CSS breakpoints.</li>
<li>Defining appropriate <strong>caching policies</strong> for your images.</li>
<li>Optimizing images for <strong>SEO</strong> and <strong>accessibility</strong>.</li>
<li>Loading and decoding images <strong>lazily</strong>.</li>
</ol>
<h3 id="h-a-lot-of-jpegs" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-a-lot-of-jpegs"><span aria-hidden="true">#</span></a> A lot of JPEGs</h3>
<p>Most of the images on <a href="http://iltirreno.it/">iltirreno.it</a> are JPEG and are included in the page with <code><img></code> tags like this:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>img</span>
<span class="token attr-name">alt</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span><span class="token punctuation">"</span></span>
<span class="token attr-name">src</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558<span class="token entity named-entity" title="&">&amp;</span>h=720<span class="token entity named-entity" title="&">&amp;</span>w=1280<span class="token entity named-entity" title="&">&amp;</span>$p$f$h$w=999f9ab<span class="token punctuation">"</span></span>
<span class="token attr-name">decoding</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>async<span class="token punctuation">"</span></span>
<span class="token attr-name">data-nimg</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>responsive<span class="token punctuation">"</span></span>
<span class="token special-attr"><span class="token attr-name">style</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span><span class="token value css language-css">SOME-INLINE-STYLES</span><span class="token punctuation">"</span></span></span>
<span class="token attr-name">sizes</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>100vw<span class="token punctuation">"</span></span>
<span class="token attr-name">srcset</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558<span class="token entity named-entity" title="&">&amp;</span>h=720<span class="token entity named-entity" title="&">&amp;</span>w=1280<span class="token entity named-entity" title="&">&amp;</span>$p$f$h$w=999f9ab 640w,
https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558<span class="token entity named-entity" title="&">&amp;</span>h=720<span class="token entity named-entity" title="&">&amp;</span>w=1280<span class="token entity named-entity" title="&">&amp;</span>$p$f$h$w=999f9ab 750w,
https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558<span class="token entity named-entity" title="&">&amp;</span>h=720<span class="token entity named-entity" title="&">&amp;</span>w=1280<span class="token entity named-entity" title="&">&amp;</span>$p$f$h$w=999f9ab 828w,
https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558<span class="token entity named-entity" title="&">&amp;</span>h=720<span class="token entity named-entity" title="&">&amp;</span>w=1280<span class="token entity named-entity" title="&">&amp;</span>$p$f$h$w=999f9ab 1080w,
https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558<span class="token entity named-entity" title="&">&amp;</span>h=720<span class="token entity named-entity" title="&">&amp;</span>w=1280<span class="token entity named-entity" title="&">&amp;</span>$p$f$h$w=999f9ab 1200w,
https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558<span class="token entity named-entity" title="&">&amp;</span>h=720<span class="token entity named-entity" title="&">&amp;</span>w=1280<span class="token entity named-entity" title="&">&amp;</span>$p$f$h$w=999f9ab 1920w,
https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558<span class="token entity named-entity" title="&">&amp;</span>h=720<span class="token entity named-entity" title="&">&amp;</span>w=1280<span class="token entity named-entity" title="&">&amp;</span>$p$f$h$w=999f9ab 2048w,
https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558<span class="token entity named-entity" title="&">&amp;</span>h=720<span class="token entity named-entity" title="&">&amp;</span>w=1280<span class="token entity named-entity" title="&">&amp;</span>$p$f$h$w=999f9ab 3840w<span class="token punctuation">"</span></span>
<span class="token punctuation">></span></span></code></pre>
<p>We can notice a few things about this <code><img></code> tag:</p>
<ul>
<li>It has an empty <code>alt</code> attribute. This means the image is not accessible. It offers no alternative text that screen readers can read aloud to blind users.</li>
<li>The <code>data-nimg</code> attribute seems to suggest that the website is using the <code><Image></code> component provided by Next.js. More precisely, since WebPageTest detected a version of Next.js previous to 13, we could speculate this is the <a href="https://nextjs.org/docs/api-reference/next/legacy/image">next/legacy/image</a> component.</li>
<li>The image is decoded asynchronously</li>
<li>There is no <code>loading="lazy"</code>. This means the browser loads this image eagerly. This is <a href="https://web.dev/browser-level-image-lazy-loading/#avoid-lazy-loading-images-that-are-in-the-first-visible-viewport">fine for images that are in the viewport</a>, but images below the viewport could be lazily loaded. See <a href="https://caniuse.com/?search=loading%3Dlazy">here for today’s browser support for loading=“lazy”</a>.</li>
<li>There is a <code>srcset</code> attribute, but the URL specified for each viewport width (e.g. 750w, 3840w) is the same (yep, double check those query strings). Also, no media queries are specified in the <code>sizes</code> attribute. This means <a href="https://cloudfour.com/thinks/responsive-images-the-simple-way/#the-sizes-attribute">resolution switching</a> is actually <strong>not</strong> done: the browser will always load the same image, regardless of the viewport width.</li>
</ul>
<p>Most images on this web page are JPEG and are served by a <a href="https://github.com/strapi/strapi">Strapi</a> server hosted at <a href="https://api-sites-prd.saegroup.abinsula.com/">api-sites-prd.saegroup.abinsula.com</a>. We know this because if we have a look at the Chrome DevTools <strong>Network</strong> panel (or at <a href="https://redbot.org/?uri=https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg">REDbot</a>), we see that the HTTP request for <a href="https://api-sites-prd.saegroup.abinsula.com/api/media/image/contentid/policy:1.100245629:1676969458/TR.jpg?f=detail_558&h=720&w=1280&$p$f$h$w=999f9ab">this image</a> returns this custom header:</p>
<pre class="language-txt"><code class="language-txt">x-powered-by: Strapi <strapi.io></code></pre>
<p>Another response header returned by the image service is <code>Content-Security-Policy</code>. Sending a CSP for <a href="https://stackoverflow.com/questions/69747541/is-content-security-policy-header-applicable-only-for-text-html-content-type">other than HTML</a> is probably not necessary. I think it is <a href="https://github.com/strapi/strapi/issues/11637">Strapi the one sending it</a>.</p>
<h3 id="h-a-few-data-urls" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-a-few-data-urls"><span aria-hidden="true">#</span></a> A few data URLs</h3>
<p>A few other images on this website are <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs">data URLs</a> and are included in the HTML using a strange <code><msdiv></code> tag. My guess is that these data URLs are generated by the streaming media service used by <a href="http://iltirreno.it/">iltirreno.it</a>.</p>
<pre class="language-txt"><code class="language-txt"><msdiv
class="player-poster"
msp-id="SOME-ID"
style="SOME-INLINE-STYLES"
background-image: "url(&quot;data:image/jpeg;base64,DATA-URI-HERE);"
>
</msdiv></code></pre>
<p>This <code><msdiv></code> tag appears only in pages that have one or more videos, and since front pages like <a href="https://www.iltirreno.it/versilia">www.iltirreno.it/versilia</a> or <a href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/www.iltirreno.it/firenze">www.iltirreno.it/firenze</a> change every day (often multiple times a day), we might find this tag today, and not find it tomorrow.</p>
<p>Data URLs that embed images in the HTML have several issues:</p>
<ul>
<li>The image cannot be cached independently from the HTML.</li>
<li>The data URL increases the page size. Also, the image is just Base64-encoded, not compressed.</li>
<li>Data URL in general pose some <a href="https://blog.mozilla.org/security/2017/11/27/blocking-top-level-navigations-data-urls-firefox-59/">security concerns</a>.</li>
</ul>
<h3 id="h-no-low-quality-image-placeholders-lqip" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-no-low-quality-image-placeholders-lqip"><span aria-hidden="true">#</span></a> No Low Quality Image Placeholders (LQIP)</h3>
<p>Take a look at the video below, that I recorded after having enabled <strong>Paint flashing</strong> (green) and <strong>Layout Shift Regions</strong> (blue) in Chrome DevTools.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1677066634/iltirreno-no-lqip_iffu6t.mkv">https://res.cloudinary.com/jackdbd/video/upload/v1677066634/iltirreno-no-lqip_iffu6t.mkv</a></p>
<p>As you can see, there is a small layout shift in the top right area, while images trigger only repaints, no layout shifts. This is good news.</p>
<p>But no image is displayed on the screen until it is fully loaded. Images suddenly appear on the page, replacing the blank space that was previously there.</p>
<p>This is not the best user experience, especially when the connectivity isn’t great and the user has to stare at the screen for a few seconds until the browser has finished loading the images.</p>
<p>An improvement would be to show a <a href="https://developer.fastly.com/solutions/tutorials/low-quality-image-placeholders/">Low Quality Image Placeholder (LQIP)</a> initially, and then swap it for the image when the image is ready.</p>
<h2 id="h-consent-banner" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-consent-banner"><span aria-hidden="true">#</span></a> Consent banner</h2>
<p>When a user visits <a href="http://iltirreno.it/">iltirreno.it</a>, their browser sets some cookies for analytics and ads. In order to comply with the ePrivacy Directive, the website must display a consent banner to its visitors. The banner looks like this:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676501957/iltirreno-consent-banner_mqggc9.png">https://res.cloudinary.com/jackdbd/image/upload/v1676501957/iltirreno-consent-banner_mqggc9.png</a></p>
<p>On <a href="http://iltirreno.it/">iltirreno.it</a> this banner is generated by the script <code>iubenda_cs.js</code>. In fact, if we block that script using the <strong>Network request blocking</strong> tab in Chrome DevTools, the banner does not appear at all.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676502187/iltirreno-block-iubenda-consent-banner_wzktgc.png">https://res.cloudinary.com/jackdbd/image/upload/v1676502187/iltirreno-block-iubenda-consent-banner_wzktgc.png</a></p>
<p>In performance audits, consent banners can be problematic for various reasons:</p>
<ol>
<li>They are often the biggest element in the page, so they can falsify <a href="https://web.dev/lcp/">Largest Contentful Paint (LCP)</a>.</li>
<li>They can cause layout shifts (aka reflows).</li>
<li>They might block HTTP requests until the banner is closed or dismissed (i.e. they delay the browser load event).</li>
</ol>
<p>We can use the ChromeDevTools <strong>Rendering</strong> tab to understand when this banner triggers a repaint, and whether it causes a layout shift. We can do this by enabling the checkboxes <strong>Paint flashing</strong> and <strong>Layout Shift Regions</strong> in Chrome DevTools.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676503511/paint-flashing-and-layout-shift-regions_bw3svb.png">https://res.cloudinary.com/jackdbd/image/upload/v1676503511/paint-flashing-and-layout-shift-regions_bw3svb.png</a></p>
<p>As we can see from the video below (I recorded this video on a different day, but the banner is the same), the banner pops up in the page a few instant after navigation. The browser needs to render it, so this causes a paint (see the green flash). Luckily, this causes no layout shifts (no blue flashes associated with the banner).</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1677008185/iltirreno-google-pixel2xl-paint-layout-shifts_etr836.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1677008185/iltirreno-google-pixel2xl-paint-layout-shifts_etr836.mp4</a></p>
<h3 id="h-accessing-cookies-and-local-storage-with-a-chrome-snippet" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-accessing-cookies-and-local-storage-with-a-chrome-snippet"><span aria-hidden="true">#</span></a> Accessing cookies and Local Storage with a Chrome snippet</h3>
<p>When a website visitor accepts or refuses the cookie policy, consent banners store this choice in a cookie or in localStorage. We can find out which cookie is used to track the user’s consent by examining the <strong>Application</strong> tab in Chrome DevTools.</p>
<p>If we navigate to <a href="https://www.iltirreno.it/versilia">www.iltirreno.it/versilia</a>, we can see that the browser sets two cookies for Google Analytics.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676551437/iltirreno-cookies_d68lfq.png">https://res.cloudinary.com/jackdbd/image/upload/v1676551437/iltirreno-cookies_d68lfq.png</a></p>
<p>If we click <code>Accetta</code> (i.e. accept) in the consent banner, a few other cookies are set.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676551445/iltirreno-cookies-after-consent-accepted_qpgyoa.png">https://res.cloudinary.com/jackdbd/image/upload/v1676551445/iltirreno-cookies-after-consent-accepted_qpgyoa.png</a></p>
<p>The cookie we are interested in is called <code>euconsent-v2</code>. This is the one that stores the user’s consent, and it’s set by the <code>iubenda_cs.js</code> script.</p>
<p>Since <code>euconsent-v2</code> is set by many other consent banners, I created a <a href="https://developer.chrome.com/docs/devtools/javascript/snippets/">Chrome snippet</a> to extract it. The code below is based on a script that Andy Davies describes in his article <a href="https://andydavies.me/blog/2021/03/25/bypassing-cookie-consent-banners-in-lighthouse-and-webpagetest/">Bypassing Cookie Consent Banners in Lighthouse and WebPageTest</a>.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// Bypass consent banner. Useful for WebPageTest</span>
<span class="token comment">// Adapted from this Andy Davies' script</span>
<span class="token comment">// https://github.com/andydavies/webpagetest-cookie-consent-scripts/blob/main/scripts/quantcast-choice.md</span>
<span class="token comment">// 1. visit a website (e.g. https://www.iltirreno.it/)</span>
<span class="token comment">// 2. click accept/reject in the consent banner</span>
<span class="token comment">// 3. run this Chrome snippet. It will generate a JS script and it will copy it to the clipboard</span>
<span class="token comment">// 4. go to WebPageTest https://www.webpagetest.org/</span>
<span class="token comment">// 5. In WebPageTest Advanced Configuration, paste the generated script into the Inject Script text box</span>
<span class="token comment">// We use let instead of const so we can run this Chrome snippet multiple times on the same page.</span>
<span class="token keyword">let</span> local_storage_keys <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token string">'noniabvendorconsent'</span><span class="token punctuation">,</span>
<span class="token comment">// these 2 define storage duration</span>
<span class="token string">'ope_fpid'</span><span class="token punctuation">,</span>
<span class="token string">'ope_uid'</span>
<span class="token punctuation">]</span>
<span class="token keyword">let</span> local_storage_entries <span class="token operator">=</span> local_storage_keys<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">key</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span> key<span class="token punctuation">,</span> <span class="token literal-property property">value</span><span class="token operator">:</span> localStorage<span class="token punctuation">.</span><span class="token function">getItem</span><span class="token punctuation">(</span>key<span class="token punctuation">)</span> <span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">let</span> local_storage_snippets <span class="token operator">=</span> local_storage_entries<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">e</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>e<span class="token punctuation">.</span>value<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">localStorage.setItem('</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>e<span class="token punctuation">.</span>key<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">', '</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>e<span class="token punctuation">.</span>value<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">');</span><span class="token template-punctuation string">`</span></span>
<span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token string">''</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token parameter">s</span> <span class="token operator">=></span> s <span class="token operator">!==</span> <span class="token string">''</span><span class="token punctuation">)</span>
<span class="token keyword">let</span> cookies <span class="token operator">=</span> document<span class="token punctuation">.</span>cookie<span class="token punctuation">.</span><span class="token function">split</span><span class="token punctuation">(</span><span class="token string">'; '</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token parameter">s</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> s<span class="token punctuation">.</span><span class="token function">startsWith</span><span class="token punctuation">(</span><span class="token string">'_iub_cs'</span><span class="token punctuation">)</span> <span class="token operator">||</span> s<span class="token punctuation">.</span><span class="token function">startsWith</span><span class="token punctuation">(</span><span class="token string">'euconsent-v2'</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">let</span> cookie_snippets <span class="token operator">=</span> cookies<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">cookie</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">document.cookie = '</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>cookie<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">;';</span><span class="token template-punctuation string">`</span></span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">// Generate JS script and copy it to the system clipboard</span>
<span class="token keyword">let</span> snippets <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token operator">...</span>local_storage_snippets<span class="token punctuation">,</span> <span class="token operator">...</span>cookie_snippets<span class="token punctuation">]</span>
<span class="token keyword">let</span> output <span class="token operator">=</span> snippets<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span><span class="token string">'\n'</span><span class="token punctuation">)</span>
<span class="token function">copy</span><span class="token punctuation">(</span>output<span class="token punctuation">)</span>
<span class="token comment">// Be sure to click accept/reject in the consent banner before running this snippet</span>
console<span class="token punctuation">.</span><span class="token function">group</span><span class="token punctuation">(</span><span class="token string">'bypass consent banner'</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Instructions:</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">1. go to WebPageTest https://www.webpagetest.org/</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">2. enter </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>document<span class="token punctuation">.</span><span class="token constant">URL</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> as the Website URL to test</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">3. go to Advanced > Inject Script, then paste the generated JS script (it's already copied to the clipboard)</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">groupEnd</span><span class="token punctuation">(</span><span class="token string">'bypass consent banner'</span><span class="token punctuation">)</span></code></pre>
<h3 id="h-bypassing-the-consent-banner-in-webpagetest" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-bypassing-the-consent-banner-in-webpagetest"><span aria-hidden="true">#</span></a> Bypassing the consent banner in WebPageTest</h3>
<p>If we click accept in the consent banner, run the Chrome snippet, copy the JavaScript code into the WebPageTest Inject Script text box and launch a new audit, we can see that the consent banner no longer appears in the video.</p>
<p>But we notice an issue: a layout shift caused by the top ad.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1677185873/iltirreno-layout-shift-caused-by-top-ad_fhnzsh.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1677185873/iltirreno-layout-shift-caused-by-top-ad_fhnzsh.mp4</a></p>
<p>This is also apparent from the Filmstrip View. A <strong>dashed</strong> yellow frame signals us that a layout shift occurred.</p>
<p>Notice also how the percentage drops from 95% to 84%, and then to 91%. That’s the <strong>Visual Progress</strong> of the page. The ad does not completely fills the space reserved to it.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1677185882/iltirreno-layout-shift-due-to-topad_ax8lqs.png">https://res.cloudinary.com/jackdbd/image/upload/v1677185882/iltirreno-layout-shift-due-to-topad_ax8lqs.png</a></p>
<p>A visit where no consent banner is shown is <strong>one</strong> of the possible scenarios when visiting most news websites for the <strong>second</strong> time. I can think of these situations for a second-time visitor:</p>
<ol>
<li>They are <strong>subscribed</strong> to the newspaper, and they have accepted/rejected the website policy: cookies and Local Storage are set, <strong>no consent banner</strong> appears, <strong>no ads</strong> appear.</li>
<li>They are <strong>not subscribed</strong> to the newspaper, and they have accepted/rejected the website policy: cookies and Local Storage are set, <strong>no consent banner</strong> appears, <strong>some ads</strong> appear.</li>
<li>They are <strong>subscribed</strong> to the newspaper, but they have not accepted/rejected the website policy: cookies and Local Storage are not set, the <strong>consent banner</strong> keeps showing up, <strong>no ads</strong> appear.</li>
<li>They are <strong>not subscribed</strong> to the newspaper, but they have not accepted/rejected the website policy: cookies and Local Storage are not set, the <strong>consent banner</strong> keeps showing up, <strong>no ads</strong> appear.</li>
</ol>
<p>Scenarios 1 and 2 are far more likely to occurr, so they are more important to test. But 3 and 4 happen from time to time. I know it’s strange to keep visiting a website without ever dismissing the consent banner. But I’ve seen it done, so we can’t rule it out.</p>
<h2 id="h-third-party-requests" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-third-party-requests"><span aria-hidden="true">#</span></a> Third party requests</h2>
<p>When a user visits <a href="http://iltirreno.it/">iltirreno.it</a> and they have <strong>not yet</strong> expressed their consent, the browser makes roughly <strong>60 requests</strong>. A few of them are first party requests, namely they fetch resources from the website itself (e.g. JS files generated by Next.js, web fonts). The rest are third party requests.</p>
<p>When a user has <strong>already</strong> given their consent and visits the website again, the browser now makes roughly <strong>140 requests</strong>.</p>
<p>Why is that?</p>
<p>It’s because only <strong>after consent</strong> is <strong>freely given</strong> the website can execute those third party scripts that process the user’s personal data. In fact, in the <a href="https://edpb.europa.eu/sites/default/files/files/file1/edpb_guidelines_202005_consent_en.pdf">European Data Protection Board (EDPB) Guidelines 05/2020 on consent under Regulation 2016/679</a> we can read the following:</p>
<blockquote>
<p>[…] consent must always be obtained before the controller starts processing personal data<br />
for which consent is needed.</p>
</blockquote>
<p>A WebPageTest Waterfall View with 140 requests would be long to include, but there is another one which is equally effective in showcasing this fact: the <strong>Connection View</strong>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1677252516/iltirreno-tcp-waterfall_i7kcvx.png">https://res.cloudinary.com/jackdbd/image/upload/v1677252516/iltirreno-tcp-waterfall_i7kcvx.png</a></p>
<p>As we can see from the image above, most of the resources actually needed to render useful content on the screen take less than 7 seconds to load. I’m talking about the HTML, CSS and JS hosted on <code>www.iltirreno.it</code> (request 1), and the images hosted on <code>api-sites-prd.saegroup.abinsula.com</code> (request 10). Almost all other resources are small files (<code>Bandwidth In</code> stays rather low), but they cause the execution of a lot of JavaScript (<code>CPU Utilization</code> stays high and they cause many <a href="https://developer.mozilla.org/en-US/docs/Glossary/Long_task">long tasks</a> in the browser’s main thread).</p>
<p>We can also use the <a href="https://developer.chrome.com/docs/devtools/performance-insights/">Performance Insights</a> panel in Chrome DevTools to understand where this JS comes from, and when it is executed.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1677258706/iltirreno-performance-insights-third-party-scripts_pwocat.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1677258706/iltirreno-performance-insights-third-party-scripts_pwocat.mp4</a></p>
<p>But the most useful tool is probably <a href="https://requestmap.pages.dev/">Request Mapper</a>, that takes the WebPageTest Test ID as an input (e.g. <code>230223_BiDcV9_GGY</code>) and generates a force-directed graph of all requests.</p>
<p><a href="https://res.cloudinary.com/jackdbd/video/upload/v1677259935/iltirreno-request-map-generator_wzjnve.mp4">https://res.cloudinary.com/jackdbd/video/upload/v1677259935/iltirreno-request-map-generator_wzjnve.mp4</a></p>
<h3 id="h-what-if-we-block-all-third-party-requests%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-what-if-we-block-all-third-party-requests%3F"><span aria-hidden="true">#</span></a> What if we block all third party requests?</h3>
<p>The impact of third party requests on the web performance of <a href="http://iltirreno.it/">iltirreno.it</a> is apparent if we block them.</p>
<p>We can configure WebPageTest to block requests by URL or domain. We then launch a test and compare the results against a baseline where we don’t block any request.</p>
<p>First, we have to identify all third party requests. There are several ways to do it. I’m going to use <a href="https://github.com/BurntSushi/xsv">xsv</a> for this task.</p>
<p>From the WebPageTest test results of the baseline (no requests blocked), click <code>Export</code> and then <code>Download Request CSV</code>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1676827660/webpagetest-result-performance-summary_bwwvtp.png">https://res.cloudinary.com/jackdbd/image/upload/v1676827660/webpagetest-result-performance-summary_bwwvtp.png</a></p>
<p>Rename the CSV to <code>requests.csv</code>, so we have less typing to do.</p>
<p>Each row in <code>requests.csv</code> represents a request. We can partition these requests by their <code>host</code> column, and create a CSV file for each host. We will store these CSV files in a folder we call <code>request-hosts</code>.</p>
<pre class="language-sh"><code class="language-sh">xsv partition <span class="token string">"host"</span> <span class="token string">"request-hosts"</span> requests.csv</code></pre>
<p>We want to keep all requests to <code>www.iltirreno.it</code> and to <code>api-sites-prd-saegroup.abinsula.com</code> (the website’s image service).</p>
<pre class="language-sh"><code class="language-sh"><span class="token function">mv</span> request-hosts/wwwiltirrenoit.csv <span class="token builtin class-name">.</span>
<span class="token function">mv</span> request-hosts/apisitesprdsaegroupabinsulacom.csv <span class="token builtin class-name">.</span></code></pre>
<p>We create a single CSV file containing all requests to block. For this we can use a combination of <a href="https://github.com/Peltoche/lsd">lsd</a>, xargs and xsv.</p>
<pre class="language-sh"><code class="language-sh"><span class="token builtin class-name">cd</span> request-hosts
lsd <span class="token operator">|</span> <span class="token function">xargs</span> <span class="token parameter variable">--verbose</span> xsv <span class="token function">cat</span> rows <span class="token parameter variable">-o</span> third-parties.csv</code></pre>
<p>We then select the <code>host</code> column from <code>third-parties.csv</code> and create a space-separated list of hosts.</p>
<pre class="language-sh"><code class="language-sh">xsv <span class="token keyword">select</span> <span class="token string">'host'</span> third-parties.csv <span class="token operator">|</span> <span class="token function">sort</span> <span class="token operator">|</span> <span class="token function">uniq</span> <span class="token operator">|</span> sd <span class="token string">'\n'</span> <span class="token string">' '</span> <span class="token operator">></span> hosts-to-block.txt</code></pre>
<p>The <code>hosts-to-block.txt</code> file should look like this:</p>
<pre class="language-txt"><code class="language-txt">0620948da3a653d1d34a6c8ef45dd69a.safeframe.googlesyndication.com 338db68053e8c4561c8b22bf7e1a3ce2.safeframe.googlesyndication.com 39dc14b003a17f497df29ffc64355b51.safeframe.googlesyndication.com 98a701c2a89109232cfbba6f0b43d7d1.safeframe.googlesyndication.com aax2scn2lo312f2xv6kn54eiqz9sr1677185495.nuid.imrworldwide.com ad.doubleclick.net adservice.google.com adservice.google.it cdn-gl.imrworldwide.com cdn.iubenda.com cdnjs.cloudflare.com cdn.jsdelivr.net cdn.opecloud.com cm.g.doubleclick.net euasync01.admantx.com gedi.profiles.tagger.opecloud.com gedi.tagger.opecloud.com gum.criteo.com hits-i.iubenda.com host pagead2.googlesyndication.com region1.analytics.google.com s1.adform.net secure-it.imrworldwide.com securepubads.g.doubleclick.net sohsjmkfg3km8ds8rwmqgbqz03pbn1677185540.nuid.imrworldwide.com static.criteo.net stats.g.doubleclick.net tagger.opecloud.com token.rubiconproject.com tpc.googlesyndication.com track.adform.net www.google.com www.google.it www.googletagmanager.com www.googletagservices.com www.iubenda.com</code></pre>
<p>We copy this space-separated list of hosts, go to WebPageTest, <code>Advanced Configuration</code> > <code>Block</code>, and paste it the <code>Block Domains</code> text box. We can then launch the test.</p>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
There is no need to set the <code>euconsent-v2</code> cookie when we block third party requests. It's the script <code>iubenda_cs.js</code> the one creating the consent banner. But <code>iubenda_cs.js</code> is hosted on <code>cdn.iubenda.com</code>, which we are blocking. This means no consent banner will appear.</p>
</div>
</div>
<p>In WebPageTest we can select two tests and compare them.</p>
<p>Here down below, in blue, we can see the test in which all 3rd party requests are blocked. In red, the baseline where no requests are blocked.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1677323260/iltirreno-comparison-requests-with-vs-without-third-parties_larr1u.png">https://res.cloudinary.com/jackdbd/image/upload/v1677323260/iltirreno-comparison-requests-with-vs-without-third-parties_larr1u.png</a></p>
<p>We can notice a few things:</p>
<ul>
<li>When 3rd parties are blocked, the browser makes only 49 requests instead of 141.</li>
<li>When 3rd parties are not blocked, there are 13 HTML entries. One is the main HTML document. Others are mostly <code><iframe></code>s created by ads and trackers.</li>
<li>There is almost double the JavaScript when 3rd parties are not blocked.</li>
<li>There are many more images when 3rd parties are not blocked. I suspect these ones are 1-pixel images used for tracking.</li>
</ul>
<p>Overall, the browser has to download roughly 40% more bytes when third parties are not blocked. And it has 171% more JavaScript to download, parse, execute.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1677323260/iltirreno-comparison-bytes-with-vs-without-third-parties_ccur0p.png">https://res.cloudinary.com/jackdbd/image/upload/v1677323260/iltirreno-comparison-bytes-with-vs-without-third-parties_ccur0p.png</a></p>
<p>The impact of JavaScript is apparent when we look at CPU Utilization, Bandwidth In, Browser Main Thread, Long Tasks.</p>
<p>Here are the charts for the baseline:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1677323260/iltirreno-cpu-utilization-js-execution-with-third-parties_ulosp0.png">https://res.cloudinary.com/jackdbd/image/upload/v1677323260/iltirreno-cpu-utilization-js-execution-with-third-parties_ulosp0.png</a></p>
<p>And here are the charts when third party requests are blocked:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1677323260/iltirreno-cpu-utilization-js-execution-without-third-parties_qeocow.png">https://res.cloudinary.com/jackdbd/image/upload/v1677323260/iltirreno-cpu-utilization-js-execution-without-third-parties_qeocow.png</a></p>
<p>Does this mean that we should remove all third party requests? Of course not. Newspapers make money selling advertising space, either on their print edition, or the digital one. Removing all ads is technically feasible, but it would be economically impossible. What we could do is carrying out a cost-benefit analysis on each third party, and decide whether it justifies its cost in terms of browser performance.</p>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ —
The Pew Research Center reports that in 2020 almost 40% of the advertising revenue of US newspapers (published by a publicly traded company) came from digital ads (<a href="https://www.pewresearch.org/journalism/chart/sotnm-newspapers-percentage-of-newspaper-advertising-revenue-coming-from-digital/">source</a>). This percentage might be different for an Italian newspaper though, I couldn't find any data on this.</p>
</div>
</div>
<h2 id="h-prototyping-changes-without-touching-the-source-code" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-prototyping-changes-without-touching-the-source-code"><span aria-hidden="true">#</span></a> Prototyping changes without touching the source code</h2>
<p>A typical <a href="https://en.wikipedia.org/wiki/OODA_loop">OODA loop</a> when trying to improve the performance of a website could be something like this:</p>
<ol>
<li>Measure the website performance in a <strong>staging environment</strong>, to establish a baseline.</li>
<li>Come up with a hypothesis.</li>
<li>Modify the source code of the <strong>website</strong>.</li>
<li>Deploy the changes to a <strong>staging environment</strong>.</li>
<li>Measure the website performance in a <strong>staging environment</strong>, to evaluate the effect of changes.</li>
<li>Accept the changes or roll them back.</li>
</ol>
<p>This is a very slow process. We could save some time if we deploy directly to production. But this of course introduces risks.</p>
<p>It would be nice if we could quickly prototype our changes without touching the source code of the website and without messing with the production environment. Well, turns out we can: we can place a <strong>proxy server</strong> between the browser and the website. The proxy server in question can be a worker on Cloudflare Workers.</p>
<p>With this approach the steps of the process are basically the same, but they happen on the worker, not on the staging/production environment:</p>
<ol>
<li>Measure the website performance on <strong>Cloudflare Workers</strong>, to establish a baseline.</li>
<li>Come up with a hypothesis.</li>
<li>Modify the source code of the <strong>worker</strong>.</li>
<li>Deploy the changes to <strong>Cloudflare Workers</strong>.</li>
<li>Measure the website performance on <strong>Cloudflare Workers</strong>, to evaluate the effect of changes.</li>
<li>Accept the changes or roll them back.</li>
</ol>
<p>So, what code should we write in the worker? We should write code that fetches the original website, adds/removes/alters the HTTP response headers, and adds/removes/alters the HTML. We can do this using the Cloudflare Workers <a href="https://developers.cloudflare.com/workers/runtime-apis/html-rewriter/">HTMLRewriter</a> API.</p>
<h3 id="h-performance-proxy" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-performance-proxy"><span aria-hidden="true">#</span></a> Performance Proxy</h3>
<p>This blog post is already much longer than I had anticipated, so I won’t include any code for the worker here. Instead, I’ll use <a href="https://perfproxy.com/">Performance Proxy</a> by <a href="https://twitter.com/SimonHearne">Simon Hearne</a> to showcase a few things we can do with <code>HTMLRewriter</code>.</p>
<p>First of all, we need to establish a new baseline. When we put a proxy between the browser and the website, the resources (HTML, CSS, JS, images, etc.) are served by the proxy. We cannot compare a WebPageTest test result obtained when the website is served by the original infrastructure (AWS CloudFront + nginx + Node.js) with a WebPageTest test result obtained when the website is served by this proxy. It would be an apples to oranges comparison.</p>
<p>So, before trying out any optimization, we enter <a href="https://www.iltirreno.it/versilia">https://www.iltirreno.it/versilia</a> as the <code>Target URL</code> in Performance Proxy, and leave everything else blank.</p>
<p>Performance Proxy gives us a <code>Test URL</code> that we can enter directly in the browser’s address bar. This URL is the website proxied by Cloudflare.</p>
<pre class="language-txt"><code class="language-txt">https://www_iltirreno_it.perfproxy.com/versilia</code></pre>
<p>Performance Proxy also gives us a <code>WebPageTest Script</code> containing a few istructions written in the <a href="https://docs.webpagetest.org/scripting/">WebPageTest scripting DSL</a>. We can copy these instructions and paste them in WebPageTest.</p>
<pre class="language-txt"><code class="language-txt">overrideHost www.iltirreno.it www_iltirreno_it.perfproxy.com
blockDomains <space-separated-list-of-domains-to-block>
navigate https://www.iltirreno.it/versilia</code></pre>
<p>Performance Proxy, <code>Request Blocking</code>, <code>Block Hosts containing</code>.</p>
<p>stiamo passando da un proxy. E poi Cloudflare gia’ di suo potrebbe migliorare la performance.</p>
<p>Before launching the test in WebPageTest, give this WebPageTest audit a <code>Label</code>, such as <code>cf-no-third-parties</code> (where <code>cf</code> stands for Cloudflare).</p>
<p>And also Request Blocking, Block Hosts containing.</p>
<p>Then in <code>Advanced Configuration</code> > <code>Script</code>. WebPageTest DSL, not JS.</p>
<p>No need to enter the website URL, since the <code>navigate</code> instruction will visit the page.</p>
<h3 id="h-adding-response-headers" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-adding-response-headers"><span aria-hidden="true">#</span></a> Adding response headers</h3>
<p>One of the things we can do with the <code>HTMLRewriter</code> API is adding some response headers. For example I added a <a href="https://www.w3.org/TR/server-timing/">Server-Timing</a> header:</p>
<pre class="language-txt"><code class="language-txt">Server-Timing: request-validation;desc="Request validation";dur=150, cms;desc="Fetch articles from CMS";dur=300, ssr;desc="Next.js SSR";dur=750,analytics;desc="Send data to analytics";dur=180,monitoring;desc="Send data to monitoring";dur=200</code></pre>
<p>Note that the timestamps I included are completely fake. In reality these timestamps would be computed on the backend and would reflect what is actually going on there.</p>
<p>The cool thing about <code>Server-Timing</code> is that the measurements it contains can be visualized in the Chrome DevTools <strong>Network</strong> panel (select the main HTML document, then the <code>Timing</code> tab to see all headers, Server-Timing included).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1677343074/server-timing_rbohk0.png">https://res.cloudinary.com/jackdbd/image/upload/v1677343074/server-timing_rbohk0.png</a></p>
<h3 id="h-adding-resource-hints" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-adding-resource-hints"><span aria-hidden="true">#</span></a> Adding resource hints</h3>
<p>We can add a <a href="https://www.w3.org/TR/resource-hints/#dfn-preconnect">preconnect hint</a> for the image service (Strapi). If the browser doesn’t decide to ignore the hint, it performs DNS lookup + TCP connection + TLS handshake towards that host.</p>
<p>We can also add a <a href="https://developer.mozilla.org/en-US/docs/Web/Performance/dns-prefetch">dns-prefetch hint</a> as a “fallback”, just in case <a href="https://caniuse.com/?search=preconnect">preconnect is not supported</a> (dns-prefetch only performs the DNS lookup, so calling it fallback of preconnect is a bit of a stretch).</p>
<p>In <a href="https://perfproxy.com/">Performance Proxy</a>, we enter this in the text box that says <code>Top of <head></code>:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>preconnect<span class="token punctuation">"</span></span>
<span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://api-sites-prd.saegroup.abinsula.com<span class="token punctuation">"</span></span>
<span class="token attr-name">crossorigin</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>dns-prefetch<span class="token punctuation">"</span></span>
<span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://api-sites-prd.saegroup.abinsula.com<span class="token punctuation">"</span></span><span class="token punctuation">></span></span></code></pre>
<p>We could also add a few other resource hints for Google Tag Manager:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>preconnect<span class="token punctuation">"</span></span>
<span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://www.googletagmanager.com<span class="token punctuation">"</span></span>
<span class="token attr-name">crossorigin</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>dns-prefetch<span class="token punctuation">"</span></span>
<span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://www.googletagmanager.com<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>preconnect<span class="token punctuation">"</span></span>
<span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://www.googletagservices.com<span class="token punctuation">"</span></span>
<span class="token attr-name">crossorigin</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>dns-prefetch<span class="token punctuation">"</span></span>
<span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://www.googletagservices.com<span class="token punctuation">"</span></span><span class="token punctuation">></span></span></code></pre>
<h3 id="h-adding-timestamps-using-the-user-timing-api" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-adding-timestamps-using-the-user-timing-api"><span aria-hidden="true">#</span></a> Adding timestamps using the User Timing API</h3>
<p>Another thing we could do is to use the <code>performance.mark()</code> method of the <a href="https://developer.mozilla.org/en-US/docs/Web/API/User_Timing_API">User Timing API</a> to track when a certain event occurs. And the <code>performance.measure()</code> method to measure the time between two performance marks.</p>
<p>For example, in Performance Proxy we can insert this code at the top of <code><head></code>:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript">
performance<span class="token punctuation">.</span><span class="token function">mark</span><span class="token punctuation">(</span><span class="token string">'head-begin'</span><span class="token punctuation">)</span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span></code></pre>
<p>This code at the bottom of <code><head></code>:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript">
performance<span class="token punctuation">.</span><span class="token function">mark</span><span class="token punctuation">(</span><span class="token string">'head-end'</span><span class="token punctuation">)</span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript">
performance<span class="token punctuation">.</span><span class="token function">measure</span><span class="token punctuation">(</span>
<span class="token string">'head-processing'</span><span class="token punctuation">,</span>
<span class="token string">'head-begin'</span><span class="token punctuation">,</span>
<span class="token string">'head-end'</span>
<span class="token punctuation">)</span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span></code></pre>
<p>This at the top of <code><body></code>:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript">
performance<span class="token punctuation">.</span><span class="token function">mark</span><span class="token punctuation">(</span><span class="token string">'body-begin'</span><span class="token punctuation">)</span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span></code></pre>
<p>And this at the bottom of <code><body></code>:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript">
performance<span class="token punctuation">.</span><span class="token function">mark</span><span class="token punctuation">(</span><span class="token string">'body-end'</span><span class="token punctuation">)</span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript">
performance<span class="token punctuation">.</span><span class="token function">measure</span><span class="token punctuation">(</span>
<span class="token string">'body-processing'</span><span class="token punctuation">,</span>
<span class="token string">'body-begin'</span><span class="token punctuation">,</span>
<span class="token string">'body-end'</span>
<span class="token punctuation">)</span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span></code></pre>
<p>When we add these performance marks, a WebPageTest Waterfall View will show them in purple.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1677338989/iltirreno-waterfall-with-markers_ypuqxm.png">https://res.cloudinary.com/jackdbd/image/upload/v1677338989/iltirreno-waterfall-with-markers_ypuqxm.png</a></p>
<h2 id="h-summing-up" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/performance-audit-of-an-italian-news-website/#h-summing-up"><span aria-hidden="true">#</span></a> Summing up</h2>
<p>Overall, the main issue of <a href="https://www.iltirreno.it/versilia">www.iltirreno.it/versilia</a> is represented by the many third party scripts on the page. They are the most detrimental to page performance, <strong>even before</strong> accepting the consent banner (thus adding a few more third party scripts to load and execute).</p>
<p>To improve the situation we could put all third party scripts in Google Tag Manager (GTM) and load them using a <a href="https://support.google.com/tagmanager/answer/7679319">Window Loaded trigger</a>.<br />
In alternative, we could execute them in a web worker. For example using <a href="https://partytown.builder.io/">partytown</a> (there is already an integration for Next.js). This would translate into less work to do for the main thread.<br />
But even before that, we could perform a cost-benefit analysis about each third party script used in the page and ask ourselved these questions: How much does it cost in term of performance? How much revenue does it generate for the website?</p>
<p>There are several other things we could try:</p>
<ul>
<li>Self-host the slick carousel (both the CSS and the <code>slick.woff</code> font) instead of leaving it on <code>cdnjs.cloudflare.com</code>.</li>
<li>Review what data is returned by <code>getServerSideProps()</code> and try decreasing what gets injected into the <code>__NEXT_DATA__</code> JSON script. Maybe inline some page data into the HTML.</li>
<li>Use tools like <a href="https://bundle-buddy.com/">Bundle Buddy</a> and <a href="https://github.com/webpack-contrib/webpack-bundle-analyzer">Webpack Bundle Analyzer</a> in combination with the Chrome DevTools <strong>Coverage</strong> panel to identify unused JavaScript and reduce the JS code chunks.</li>
<li>Use a service worker to cache third party scripts and other assets.</li>
<li>Use AVIF or WebP as the image format, instead of JPEG.</li>
<li>Configure resolution switching correctly, so big images will be downloaded only when viewing the website on wide viewports (i.e. desktops, not mobile phones).</li>
<li>Use an image CDN like Cloudinary to serve images optimized for each device.</li>
<li>Add a <code>preconnect</code> and a <code>dns-prefetch</code> in the <code><head></code> to improve the first connection to the image service (Strapi or Cloudinary).</li>
<li>Use Woff2 for web fonts.</li>
<li>Avoid inlining all the CSS in the <code><head></code>. Maybe keep inlining some critical CSS.</li>
</ul>
<p><em>Thanks to <a href="https://twitter.com/tkadlec">Tim Kadlec</a> and <a href="https://twitter.com/TimVereecke">Tim Vereecke</a> for suggestions on how to interpret a few WebPageTest test results</em>.</p>
https://www.giacomodebidda.com/posts/inspect-container-images-with-dive/Inspect container images with dive2022-02-04T10:30:00Z2022-02-04T10:30:00ZDive is a CLI tool that can be used to inspect container images and understand how the layers contribute to size of the container image.
<p>Container images can be created using <code>docker build</code> with a <code>Dockerfile</code>, or with other tools such as <a href="https://github.com/GoogleContainerTools/jib">jib</a> or <a href="https://github.com/buildpacks/pack">pack</a>. Regardless of which tool was used to create the container image, you can use <a href="https://github.com/wagoodman/dive">dive</a> to inspect all layers that make up the final image.</p>
<h2 id="h-container-image-and-container-image-layer" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/inspect-container-images-with-dive/#h-container-image-and-container-image-layer"><span aria-hidden="true">#</span></a> Container image and container image layer</h2>
<p>A <strong>container image</strong> is a read-only, immutable file that contains the source code, libraries, dependencies, environment configuration, tools, and other files needed for an application to run. It has a human-readable name, a human-assigned tag, and can be used to create <strong>containers</strong>.</p>
<p>A <strong>container image layer</strong> is basically just an intermediate container image that has neither a name, nor a tag. You can think of a container image just as a diff of several container image layers.</p>
<p>The <a href="https://opencontainers.org/">Open Container Initiative (OCI)</a> came up with a <a href="https://github.com/opencontainers/image-spec">specification</a> for a standard format for container images. This means that you can use any tool to create a container image, as long as such tool produces a container image that adheres to the OCI specification.</p>
<h2 id="h-dive-vs-other-tools" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/inspect-container-images-with-dive/#h-dive-vs-other-tools"><span aria-hidden="true">#</span></a> Dive vs other tools</h2>
<p>If you are just interested in knowing what’s inside a <strong>container</strong>—which, remember, was built using a container image, i.e. the final container image layer—you can enter the container and have a look around.</p>
<p>For example, let’s say that you want to know what’s inside an Alpine Linux container.</p>
<p>First, you pull the container image from a container registry like <a href="https://hub.docker.com/_/alpine">Docker Hub</a>:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">docker</span> pull alpine:latest</code></pre>
<p>Second, you create a container and obtain a shell:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">docker</span> run <span class="token parameter variable">--rm</span> <span class="token parameter variable">--interactive</span> <span class="token parameter variable">--tty</span> alpine:latest
<span class="token comment"># or, for short</span>
<span class="token function">docker</span> run <span class="token parameter variable">--rm</span> <span class="token parameter variable">-it</span> alpine:latest</code></pre>
<p>Third, inside the Alpine container, you list all the files:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">ls</span> <span class="token parameter variable">-1aR</span></code></pre>
<p>But what if want to know how many layers there are, and how big they are? Well, for that there is <a href="https://docs.docker.com/engine/reference/commandline/history/">docker history</a>.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">docker</span> <span class="token function">history</span> alpine:latest</code></pre>
<p>However, as far as I know, there is only one tool that allows you to inspect <strong>each</strong> layer that make up the final image, and that tool is <a href="https://github.com/wagoodman/dive">dive</a>.</p>
<h2 id="h-installation" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/inspect-container-images-with-dive/#h-installation"><span aria-hidden="true">#</span></a> Installation</h2>
<p>You can obtain dive by downloading and installing the binary from <a href="https://github.com/wagoodman/dive/releases">its GitHub releases</a>, or by pulling its container image with <code>docker pull wagoodman/dive:latest</code>.</p>
<h2 id="h-how-to-use-dive%3F-a-few-examples" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/inspect-container-images-with-dive/#h-how-to-use-dive%3F-a-few-examples"><span aria-hidden="true">#</span></a> How to use dive? A few examples</h2>
<h3 id="h-example-1-alpine-linux" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/inspect-container-images-with-dive/#h-example-1-alpine-linux"><span aria-hidden="true">#</span></a> Example 1: Alpine Linux</h3>
<p>If you don’t have a local Alpine image, you can pull the latest one from Docker Hub with this command:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">docker</span> pull alpine:latest</code></pre>
<p>You are now ready to inspect the container image layers with dive:</p>
<pre class="language-shell"><code class="language-shell">dive alpine:latest</code></pre>
<p>You can switch panes with <code>Tab</code>, collapse/expand the file tree with the space bar, and exit the program with <code>Ctrl</code>+<code>c</code>.<br />
You can read the <a href="https://github.com/wagoodman/dive#keybindings">documentation</a> for the other keybindings.</p>
<p>Alpine is a Linux distro designed to be used in containers and embedded devices. Two of the main reasons of its small size (~5MB) are that Alpine packs 300+ UNIX tools in a <strong>single binary</strong>, thanks to <a href="https://www.busybox.net/">BusyBox</a>, and <a href="https://youtu.be/Cc1rBayMnVI">decided</a> to pick <a href="https://musl.libc.org/">musl</a> as the implementation for the standard C library (see <a href="https://www.etalabs.net/compare_libcs.html">here</a> how musl compares with glibc).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1643976403/alpine_linux_container_image_with_dive.png">https://res.cloudinary.com/jackdbd/image/upload/v1643976403/alpine_linux_container_image_with_dive.png</a></p>
<p>Using dive you can see that the “binaries” in Alpine are just symlinks to the single binary <code>/bin/busybox</code>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1643976375/busybox_bin.png">https://res.cloudinary.com/jackdbd/image/upload/v1643976375/busybox_bin.png</a></p>
<p>If you are curious you can also inspect the container image of BusyBox (or <a href="https://landley.net/toybox/">Toybox</a>, an alternative to BusyBox) and see how small it is. I’ll leave you the commands here so you can copy and paste them.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">docker</span> pull busybox:latest
dive busybox:latest</code></pre>
<pre class="language-shell"><code class="language-shell"><span class="token function">docker</span> pull tianon/toybox:latest
dive tianon/toybox:latest</code></pre>
<h3 id="h-example-2-jdk-vs-jre-in-a-clojure-app" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/inspect-container-images-with-dive/#h-example-2-jdk-vs-jre-in-a-clojure-app"><span aria-hidden="true">#</span></a> Example 2: JDK vs JRE in a Clojure app</h3>
<p>If you are developing a Java/Clojure/Kotlin/Scala application (or any language which compiles to Java bytecode and runs on the Java Virtual Machine), you need the Java Development Kit (JDK). However, once that application is compiled and packaged into an uberjar, it can be run either with the JDK or with the Java Runtime Environment (JRE).</p>
<p>There are many ways to create a container image which can run an uberjar. I like to use <a href="https://github.com/buildpacks/pack">pack</a>. With pack you don’t need a <code>Dockerfile</code> to create a container image. You simply specify a path to your source code, a <a href="https://buildpacks.io/docs/concepts/components/builder/">builder</a>, and let the builder build the container image for you.</p>
<p>I had a Clojure application and I decided to use pack and the <a href="https://github.com/paketo-buildpacks/tiny-builder">Paketo Tiny Builder</a> to create a container image. This is the command I used:</p>
<pre class="language-shell"><code class="language-shell">pack build my_clojure_app:latest <span class="token punctuation">\</span>
<span class="token parameter variable">--path</span> <span class="token builtin class-name">.</span> <span class="token punctuation">\</span>
<span class="token parameter variable">--builder</span> paketobuildpacks/builder:tiny</code></pre>
<p>The generated container image is roughly 180 MB, mostly due to the layer containing the Java Runtime Environment (JRE), which is 148 MB.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1643979796/clojure_app_jre.png">https://res.cloudinary.com/jackdbd/image/upload/v1643979796/clojure_app_jre.png</a></p>
<p>You don’t really need to know pack, builders and <a href="https://buildpacks.io/docs/concepts/components/buildpack/">buildpacks</a> to understand this example. The only thing you need to know is that a builder is itself a container image, and can be configured with environment variables <strong>at build-time</strong>. These environment variables control the buildpacks used by the builder.</p>
<p>Two of the buildpacks used by Paketo Tiny Builder are <a href="https://github.com/paketo-buildpacks/bellsoft-liberica#configuration">Paketo BellSoft Liberica Buildpack</a> and the <a href="https://github.com/paketo-buildpacks/clojure-tools/blob/main/README.md#configuration">Paketo Clojure Tools Buildpack</a>, and can be configured with many environment variables. The most important one is <code>BP_JVM_TYPE</code>, which controls whether the container image will contain the JDK or the JRE (the default).</p>
<p>You can generate a container image containing the JDK (instead of the JRE) with this command:</p>
<pre class="language-shell"><code class="language-shell">pack build my_clojure_app:latest <span class="token punctuation">\</span>
<span class="token parameter variable">--path</span> <span class="token builtin class-name">.</span> <span class="token punctuation">\</span>
<span class="token parameter variable">--builder</span> paketobuildpacks/builder:tiny <span class="token punctuation">\</span>
<span class="token parameter variable">--env</span> <span class="token assign-left variable">BP_JVM_TYPE</span><span class="token operator">=</span>JDK <span class="token punctuation">\</span>
<span class="token parameter variable">--env</span> <span class="token assign-left variable">BP_JVM_VERSION</span><span class="token operator">=</span><span class="token number">11</span> <span class="token punctuation">\</span>
<span class="token parameter variable">--env</span> <span class="token assign-left variable">BP_CLJ_TOOLS_BUILD_ARGUMENTS</span><span class="token operator">=</span><span class="token string">"-T:build uber"</span></code></pre>
<p>The layer containing the Java Development Kit (JDK) is 350 MB, more than double the size of the layer containing the JRE (148 MB).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1643979801/clojure_app_jdk.png">https://res.cloudinary.com/jackdbd/image/upload/v1643979801/clojure_app_jdk.png</a></p>
<h3 id="h-example-3-nodejs-express-app" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/inspect-container-images-with-dive/#h-example-3-nodejs-express-app"><span aria-hidden="true">#</span></a> Example 3: Node.js Express app</h3>
<p>I have a simple Express app which I use to receive and handle webhook events from Stripe. The app is written in TypeScript and bundled with <a href="https://esbuild.github.io/">esbuild</a>. I compile the TS and generate the <code>bundle.js</code> on my laptop, and let the builder <a href="https://github.com/GoogleCloudPlatform/buildpacks">gcr.io/buildpacks/builder:v1</a> build the container image (which I then deploy to Cloud Run). Here is the command I use to create the container image:</p>
<pre class="language-shell"><code class="language-shell">pack build my_nodejs_app:latest <span class="token punctuation">\</span>
<span class="token parameter variable">--path</span> ./dist <span class="token punctuation">\</span>
<span class="token parameter variable">--builder</span> gcr.io/buildpacks/builder:v1</code></pre>
<p><em>Note</em>: <code>dist/</code> contains <code>bundle.js</code> and <code>package.json</code></p>
<p>Using dive we can see that the container image layer corresponding to the Node.js runtime is called <code>google.nodejs.runtime</code>. It’s roughly 101 MB in size and contains the <code>node</code> binary (75 MB) and two symlinks for npm and npx.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1643985444/google_nodejs_runtime_layer.png">https://res.cloudinary.com/jackdbd/image/upload/v1643985444/google_nodejs_runtime_layer.png</a></p>
<p>The container image layer corresponding to the Express app is 42 MB, mostly due to its dependencies in <code>node_modules</code> (37 MB).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1643987511/nodejs_app_source_code_layer.png">https://res.cloudinary.com/jackdbd/image/upload/v1643987511/nodejs_app_source_code_layer.png</a></p>
<p>Another layer is the <a href="https://buildpacks.io/docs/concepts/components/lifecycle/launch/">Cloud Native Buildpack launcher</a>. Here <code>cnb</code> is the <strong>non-root</strong> user that executes the commands in the container.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1643987650/cloud_native_buildpack_launcher.png">https://res.cloudinary.com/jackdbd/image/upload/v1643987650/cloud_native_buildpack_launcher.png</a></p>
<p>The second-to-last layer contains the configuration <a href="https://github.com/buildpacks/spec/blob/platform/0.7/platform.md#metadatatoml-toml">metadata.toml</a> of the buildpack.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1643989213/cloud_native_buildpack_config_metadata.png">https://res.cloudinary.com/jackdbd/image/upload/v1643989213/cloud_native_buildpack_config_metadata.png</a></p>
<p>The last layer is just a symlink for the buildpack launcher.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1643989193/cloud_native_buildpack_launcher_symlink.png">https://res.cloudinary.com/jackdbd/image/upload/v1643989193/cloud_native_buildpack_launcher_symlink.png</a></p>
https://www.giacomodebidda.com/posts/test-your-javascript-on-multiple-engines-with-eshost-cli-and-jsvu/Test your JavaScript on multiple engines with eshost-cli and jsvu2021-05-07T17:00:03Z2021-05-07T17:00:03ZTesting your code on multiple JavaScript engines can be an effective way to understand how to write more performant libraries and applications.
<p>JavaScript engines are complex software components that optimize and run your JavaScript. Testing your code on multiple engines can be an effective way to understand how to write more performant libraries and applications. This is true no matter whether you are developing for the browser, Node.js, Deno, or for any other JavaScript runtime.</p>
<h2 id="h-install-js-engines-with-jsvu" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/test-your-javascript-on-multiple-engines-with-eshost-cli-and-jsvu/#h-install-js-engines-with-jsvu"><span aria-hidden="true">#</span></a> Install JS engines with jsvu</h2>
<p>Since most engines are written in languages like C, C++ and Rust, you’ll have to download the source files and compile them. However, you don’t have to do it yourself. A tool like <a href="https://github.com/GoogleChromeLabs/jsvu">jsvu</a> can download and compile the most popular JS engines for you.</p>
<p>You can install the jsvu CLI with npm (or yarn):</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">npm</span> <span class="token function">install</span> jsvu <span class="token parameter variable">-g</span></code></pre>
<p>and run it with:</p>
<pre class="language-shell"><code class="language-shell">jsvu</code></pre>
<p>On the first run, <code>jsvu</code> asks you for the list of JavaScript engines you wish to manage through <code>jsvu</code>, then it creates:</p>
<ol>
<li>a <code>.jsvu</code> directory in your home directory;</li>
<li>a <code>status.json</code> file inside of the <code>.jsvu</code> directory</li>
</ol>
<p>Every time <code>jsvu</code> runs, including the first time, it installs the latest version of the engines specified in <code>~/.jsvu/status.json</code>.</p>
<p><code>jsvu</code> then compiles the engines’ source files and stores their binaries in <code>~/.jsvu/engines</code>. It also creates either a symlink or a small shell script to execute each engine’s shell.</p>
<div class="callout callout--tip">
<div class="callout__content">
<p>💡 — You may want to add the <code>~/.jsvu</code> directory to your PATH, as they suggest in the <a href="https://github.com/GoogleChromeLabs/jsvu#installation">project's README</a>.</p>
</div>
</div>
<p>Having all JS engines in <code>~/.jsvu</code> (through a symlink or a shell script) is quite convenient for setting up <a href="https://github.com/bterlson/eshost-cli">eshost-cli</a>, the tool that runs your code on all of these engines.</p>
<h2 id="h-run-js-code-on-multiple-engines-with-eshost-cli" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/test-your-javascript-on-multiple-engines-with-eshost-cli-and-jsvu/#h-run-js-code-on-multiple-engines-with-eshost-cli"><span aria-hidden="true">#</span></a> Run JS code on multiple engines with eshost-cli</h2>
<p>You can install eshost-cli with npm (or yarn):</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">npm</span> <span class="token function">install</span> <span class="token parameter variable">-g</span> eshost-cli</code></pre>
<p>You <strong>add</strong> an ECMAScript host with the following syntax:</p>
<pre class="language-shell"><code class="language-shell">eshost <span class="token parameter variable">--add</span> <span class="token operator"><</span>host name<span class="token operator">></span> <span class="token operator"><</span>host type<span class="token operator">></span> <span class="token operator"><</span>host path<span class="token operator">></span></code></pre>
<p>Let’s say that you have previously installed (with jsvu) <a href="https://github.com/oracle/graaljs">GraalJS</a>, <a href="https://spidermonkey.dev/">SpiderMonkey</a> and <a href="https://github.com/v8/v8">V8</a>. Here is how you can add them as hosts:</p>
<pre class="language-shell"><code class="language-shell">eshost <span class="token parameter variable">--add</span> <span class="token string">'GraalJS'</span> graaljs ~/.jsvu/graaljs
eshost <span class="token parameter variable">--add</span> <span class="token string">'SpiderMonkey'</span> jsshell ~/.jsvu/spidermonkey
eshost <span class="token parameter variable">--add</span> <span class="token string">'V8'</span> d8 ~/.jsvu/v8
eshost <span class="token parameter variable">--add</span> <span class="token string">'V8-debug'</span> d8 ~/.jsvu/v8-debug</code></pre>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ — That <code>d8</code> is not a typo. It's the <a href="https://v8.dev/docs/d8">V8 developer shell</a>.</p>
</div>
</div>
<p>Now that you have configured some hosts, you can evaluate your code on multiple engines:</p>
<pre class="language-shell"><code class="language-shell">eshost <span class="token parameter variable">--eval</span> <span class="token string">'1 + 2'</span> <span class="token comment"># or just -e</span></code></pre>
<pre class="language-shell"><code class="language-shell"><span class="token comment">#### GraalJS</span>
<span class="token number">3</span>
<span class="token comment">#### SpiderMonkey</span>
<span class="token number">3</span>
<span class="token comment">#### V8</span>
<span class="token number">3</span>
<span class="token comment">#### V8-debug</span>
<span class="token number">3</span></code></pre>
<p>If you prefer a tabular output, just pass the <code>-t / --table</code> flag:</p>
<pre class="language-shell"><code class="language-shell">eshost <span class="token parameter variable">-e</span> <span class="token string">'1 + 2'</span> <span class="token parameter variable">--table</span> <span class="token comment"># or just -t</span></code></pre>
<pre class="language-shell"><code class="language-shell">┌────────────────┬───┐
│ GraalJS │ <span class="token number">3</span> │
├────────────────┼───┤
│ SpiderMonkey │ <span class="token number">3</span> │
├────────────────┼───┤
│ V8 │ <span class="token number">3</span> │
├────────────────┼───┤
│ V8-debug │ <span class="token number">3</span> │
└────────────────┴───┘</code></pre>
<p>You can evaluate your code on a <strong>single host</strong> with <code>-h / --host</code>:</p>
<pre class="language-shell"><code class="language-shell">eshost <span class="token parameter variable">-e</span> <span class="token string">'1 + 2'</span> <span class="token parameter variable">-t</span> <span class="token parameter variable">-h</span> GraalJS</code></pre>
<pre class="language-shell"><code class="language-shell">┌────────────────┬───┐
│ GraalJS │ <span class="token number">3</span> │
└────────────────┴───┘</code></pre>
<p>You can also evaluate your code on <strong>multiple hosts</strong>, but first you need to group some hosts together, using a common tag.</p>
<p>Here is how to <strong>edit</strong> existing hosts:</p>
<pre class="language-shell"><code class="language-shell">eshost <span class="token parameter variable">--edit</span> <span class="token string">'V8'</span> d8 ~/.jsvu/v8 <span class="token parameter variable">--tags</span> v8,latest
eshost <span class="token parameter variable">--edit</span> <span class="token string">'V8-debug'</span> d8 ~/.jsvu/v8-debug <span class="token parameter variable">--tags</span> v8,debug</code></pre>
<p>and how to select the hosts by tag:</p>
<pre class="language-shell"><code class="language-shell">eshost <span class="token parameter variable">-e</span> <span class="token string">'1 + 2'</span> <span class="token parameter variable">-t</span> <span class="token parameter variable">--tags</span> v8</code></pre>
<pre class="language-shell"><code class="language-shell">┌────────────────┬───┐
│ V8 │ <span class="token number">3</span> │
├────────────────┼───┤
│ V8-debug │ <span class="token number">3</span> │
└────────────────┴───┘</code></pre>
<p>You can also pass arguments specific to the underlying JavaScript engine. This is useful if you want to understand how a particular engine works. You can output profiling data, enable tracing, generate debugging information from the engine garbage collector/s, optimizing compiler/s, etc.</p>
<p>For example, the flag <code>--trace-gc-verbose</code> can be useful if you want to understand how garbage collection works in V8.</p>
<pre class="language-shell"><code class="language-shell">eshost <span class="token parameter variable">--edit</span> <span class="token string">'V8-debug'</span> <span class="token parameter variable">--args</span> <span class="token string">'--trace-gc-verbose'</span></code></pre>
<p>Keep in mind that every JS engine has its own flags, and there might be a lot of them. I think V8 has more than 500 flags!</p>
<p>Until now we have evaluated inline JavaScript, but eshost-cli allows you to pass a JavaScript file too.</p>
<p>Write the JavaScript you want to test…</p>
<pre class="language-js"><code class="language-js"><span class="token comment">// test.js</span>
<span class="token keyword">const</span> x <span class="token operator">=</span> <span class="token number">42</span><span class="token punctuation">;</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">The answer is </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>x<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p>…then run eshost-cli to evaluate it:</p>
<pre class="language-shell"><code class="language-shell">eshost test.js</code></pre>
<p>Finally, when you no longer need a host, here is how to <strong>delete</strong> it:</p>
<pre class="language-shell"><code class="language-shell">eshost <span class="token parameter variable">--delete</span> <span class="token string">'JavaScriptCore'</span></code></pre>
https://www.giacomodebidda.com/posts/generate-and-view-har-files/Generate and view HAR files2021-05-06T17:30:03Z2021-05-06T17:30:03ZA HAR (Http ARchive) file is important when troubleshooting performance issues because it contains data related to the HTTP transactions that occur between a web browser and a website or web app.
<p>A HAR (Http ARchive) file contains data related to the HTTP transactions that occur between a web browser and a website or web app. Generating and analyzing a HAR file is important when troubleshooting performance issues.</p>
<h2 id="h-what%E2%80%99s-inside-a-har-file%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/generate-and-view-har-files/#h-what%E2%80%99s-inside-a-har-file%3F"><span aria-hidden="true">#</span></a> What’s inside a HAR file?</h2>
<p>The HAR file format is based on JSON, and the current version of the specification (1.2) can be found <a href="https://www.softwareishard.com/blog/har-12-spec/">here</a>.</p>
<p>Inside a HAR file you can find both coarse data regarding every single page visited—like timings for the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/DOMContentLoaded_event">DOMContentLoaded</a> and <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/load_event">load</a> events—and granular data describing every single HTTP request made by the browser.</p>
<p>Every <a href="https://www.softwareishard.com/blog/har-12-spec/#entries">entry</a> in a HAR file is tied to a <a href="https://www.softwareishard.com/blog/har-12-spec/#pages">page</a> via a <code>pageref</code> field. Thanks to this relationship the browser—or any tool that can process and show HAR files—can reconstruct a <strong>waterfall</strong> of HTTP requests.</p>
<p>Down below you can see how <a href="https://developer.chrome.com/docs/devtools/network/reference/">Chrome DevTools</a> displays the waterfall chart of the Google homepage. The blue vertical line represents the DOMContentLoaded event, while the red one the Load event.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1620314435/waterfall_google_chrome_bp672m.png">https://res.cloudinary.com/jackdbd/image/upload/v1620314435/waterfall_google_chrome_bp672m.png</a></p>
<p>And here you can see the waterfall of the same webpage using <a href="https://compare.sitespeed.io/">Compare</a>. The lavender vertical line represents the load event, while the violet one the startRender event (which is a metric calculated by <a href="https://github.com/WPO-Foundation/visualmetrics/blob/22d152978ac5a3007603d1dca374011874cbf49f/visualmetrics.py#L542">visualmetrics</a>).</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1620314435/waterfall_google_compare_hqyhcf.png">https://res.cloudinary.com/jackdbd/image/upload/v1620314435/waterfall_google_compare_hqyhcf.png</a></p>
<p>Note how Chrome DevTools and Compare share the same color scheme for the horizontal bars. This is usually the case with HAR file viewers, even if there are some tools that use a different color scheme.</p>
<p>The colors of the horizontal bars represent:</p>
<ul>
<li><strong><span style="color:#aaaaaa">Blocked</span></strong>: the request is blocked for some reason: maybe the browser is fetching higher priority requests, there are no TCP connections available, there are too many TCP connections already open (applies only to HTTP/1.0 and HTTP/1.1). Chrome DevTools refers to this case with the terms <em>Queueing</em> and <em>Stalled</em>.</li>
<li><strong><span style="color:#149588">DNS</span></strong>: the browser is performing a DNS lookup, i.e. translating <em>your-website.com</em> into <em>your-host-ip-address</em>. DNS requests are cached, so DNS lookup times may differ in subsequent tests. That’s why most tools perform at least three runs before generating a waterfall.</li>
<li><strong><span style="color:#FE9726">Connect</span></strong>: the browser is establishing a TCP connection, including TCP handshakes/retries.</li>
<li><strong><span style="color:#C140CD">SSL (TLS)</span></strong>: browser and server are negotiating an SSL certificate with the <a href="https://howhttps.works/the-handshake/">SSL/TLS handshake</a>.</li>
<li><strong><span style="color:#AFBFC5">Send</span></strong>: the browser is sending the request to the server. If it’s a PUT or POST request, then this will also include the time spent uploading any data with that request.</li>
<li><strong><span style="color:#1EC659">Wait</span></strong>: the server received the request and it is generating a response, while the browser is waiting for the first byte of a response. Chrome DevTools calls this time <em>TTFB</em> (Time To First Byte). Note that <a href="https://developer.mozilla.org/en-US/docs/Glossary/time_to_first_byte">other definitions of TTFB</a> include the DNS lookup time.</li>
<li><strong><span style="color:#1DAAF2">Receive</span></strong>: the browser is receiving the response. Chrome DevTools uses the term <em>Content Download</em>.</li>
</ul>
<p>By looking at these waterfalls we can understand <strong>a lot</strong> about the performance of a web page, but first we need to generate a HAR file. So let’s see which tools we can use for that.</p>
<h2 id="h-generate-a-har-file" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/generate-and-view-har-files/#h-generate-a-har-file"><span aria-hidden="true">#</span></a> Generate a HAR file</h2>
<p>You can use several tools to create a HAR file:</p>
<ol>
<li>a web browser like Chrome or Firefox. This is convenient, but don’t forget to warm up the DNS cache by making multiple runs. Also, <a href="https://www.youtube.com/watch?v=dCThwpglIeE&t=108s&ab_channel=sitespeed.io">HAR files generated by browsers are not perfect</a>;</li>
<li>a browser automation tool like <a href="https://github.com/Everettss/puppeteer-har">Puppeteer</a>, <a href="https://github.com/NeuraLegion/cypress-har-generator">Cypress</a> or <a href="https://github.com/sitespeedio/browsertime">Browsertime</a>;</li>
<li>a web performance tool like <a href="https://www.webpagetest.org/">WebPageTest</a> or <a href="https://github.com/sitespeedio/sitespeed.io">sitespeed.io</a>;</li>
<li>a library that extracts the HTTP transactions from the Chrome DevTools Protocol, like <a href="https://leonardofaria.net/2020/11/30/creating-har-files-with-lighthouse/">chrome-har-capturer</a> or <a href="https://github.com/sitespeedio/chrome-har">chrome-har</a>.</li>
</ol>
<p>I like to use WebPageTest to generate a HAR file because of its many test locations and the many options for device emulation. Replicating the same options in Chrome would mean having to implement <a href="https://developer.chrome.com/docs/devtools/device-mode/">custom profiles</a> for CPU throttling, network throttling and location.</p>
<div class="callout callout--warn">
<div class="callout__content">
<p>⚠️ — You should always create a HAR file in an incognito tab, to avoid having requests made by Chrome extensions show up in your waterfall. I find this quite annoying and easy to forget. That's another reason why I prefer a tool like WebPageTest for this task.</p>
</div>
</div>
<h2 id="h-view-and-analyze-a-har-file" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/generate-and-view-har-files/#h-view-and-analyze-a-har-file"><span aria-hidden="true">#</span></a> View and analyze a HAR file</h2>
<p>You have several options to view a HAR file:</p>
<ol>
<li>a web browser;</li>
<li>an online tool like <a href="https://www.webpagetest.org/">WebPageTest</a>, <a href="https://compare.sitespeed.io/">Compare</a>, <a href="https://toolbox.googleapps.com/apps/har_analyzer/">HAR Analyzer</a><br />
or <a href="https://www.softwareishard.com/blog/har-viewer/">HAR Viewer</a>;</li>
<li>a UI component like <a href="https://opensource.saucelabs.com/blog/react_network_viewer/">Network-Viewer</a> for React.</li>
</ol>
<p>I like Compare because it lets me load either one or two HAR files, and it lets me… well… compare them. But the waterfall chart generated by WebPageTest is by far the one with the most details.</p>
<p>The waterfall chart contains a lot of valuable information, and it’s important to have a look both at individual rows and at the overall picture.</p>
<p>When looking at a single row, you may want to focus on the response size (body and headers), Cache-Control directives, cookies, and the TTFB.</p>
<p>When looking at the overall picture you can take note of how many parallel requests there are, how many TCP connections, how many HTTP redirects, and whether there are wide horizontal gaps in the chart. The waterfall generated by WebPageTest is particularly useful here, because it shows the JavaScript execution in pink. So for example a wide horizontal gap in the network activity, with many bursts of pink, indicates a CPU bottleneck: low-end mobile devices might have a hard time processing this particular request.</p>
<p>For a thorough analysis of visual patterns that can be found in a waterfall chart, I recommend the talk <a href="https://www.youtube.com/watch?v=THmJwZPGAuQ&ab_channel=LondonWebPerformanceGroup">How to read a WebPageTest waterfall chart</a> by Matt Hobbs or the <a href="https://nooshu.github.io/blog/2019/10/02/how-to-read-a-wpt-waterfall-chart/">associated blog post</a>.</p>
<h2 id="h-other-tools" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/generate-and-view-har-files/#h-other-tools"><span aria-hidden="true">#</span></a> Other tools</h2>
<p>HAR files are mostly used to detect performance issues, but they can also be useful when developing and (stress) testing web apps and websites. Here are a couple of tools that make use of HAR files for these use-cases:</p>
<ul>
<li><a href="https://github.com/Stuk/server-replay">server-replay</a>: a proxy server that replays HTTP responses. I have never used it, but it seems to me a better solution that mocking HTTP responses with something like <a href="https://github.com/nock/nock">nock</a>.</li>
<li><a href="https://github.com/acastaner/harhar/wiki">harhar</a>: a HTTP benchmark tool. It takes all requests recorded in a HAR file and replays them a thousand fold.</li>
</ul>
https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/How I setup my laptop with Ansible2020-09-18T12:00:00Z2020-09-18T12:00:00ZHere is how I (re)configure my laptop with Ansible
<p>I’ve recently started using Ansible to configure my laptop. Having a development machine setup with a couple of command is pretty great, but there are still some manual steps I need to take care in the process.</p>
<p>Here is what I do every time I (re)configure my laptop.</p>
<h2 id="h-preliminary-operations" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-preliminary-operations"><span aria-hidden="true">#</span></a> Preliminary operations</h2>
<ol>
<li>Create a bootable USB stick for my Linux distro of choice, Xubuntu 20.04 (see step 0 below).</li>
<li>Save all my SSH keys and config files on a USB stick or external hard disk.</li>
<li>Print the backup 2FA codes for my password manager (Lastpass).</li>
<li>Export the feedlist from my RSS reader (Liferea).</li>
<li>Check that I can connect to a WiFi network.</li>
</ol>
<h2 id="h-0---create-a-bootable-usb-stick-for-a-linux-distro" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-0---create-a-bootable-usb-stick-for-a-linux-distro"><span aria-hidden="true">#</span></a> 0 - Create a bootable USB stick for a Linux distro</h2>
<p>There are many articles that explain how to do it, but I think they are way too verbose. Here is how I do it:</p>
<ol>
<li>Download the ISO of your Linux distro of choice.</li>
<li>Find a 8GB USB stick and format it with GParted. Choose a FAT32 file system. In alternative, you can format it from the terminal using <code>fdisk</code> and <code>mkfs</code> (see <a href="https://www.redips.net/linux/create-fat32-usb-drive/">this tutorial</a>), but in my opinion GParted is way easier and faster.</li>
<li>Check the device name that your system assigned to the USB stick. You can use either <code>lsblk</code> or <code>sudo fdisk -l</code> to find it out. Let’s say it’s <code>/dev/sda</code></li>
<li>Use <code>ddrescue</code> to write the ISO to the USB stick. This should take ~5 minutes.</li>
</ol>
<p>Here is how I wrote a bootable Xubuntu 20.04 USB stick with <code>ddrescue</code>:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">sudo</span> <span class="token function">ddrescue</span> xubuntu-20.04-desktop-amd64.iso /dev/sda <span class="token parameter variable">--force</span> <span class="token parameter variable">-D</span></code></pre>
<h2 id="h-1---install-xubuntu-with-full-disk-encryption" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-1---install-xubuntu-with-full-disk-encryption"><span aria-hidden="true">#</span></a> 1 - Install Xubuntu with full-disk encryption</h2>
<p>As far as I know, Full Disk Encryption is the single most important thing you can do to ensure your peace of mind. You need LVM (Logical Volume Management) to use it, and you have to encrypt the disk when you install a new system. You can’t do it later.</p>
<h2 id="h-2---install-git-and-gnu-stow" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-2---install-git-and-gnu-stow"><span aria-hidden="true">#</span></a> 2 - Install GIT and GNU Stow</h2>
<p>I don’t know why, but Ubuntu doesn’t come with git already installed. <a href="https://www.gnu.org/software/stow/">GNU Stow</a> is a program I use to symlink my dotfiles from a git repo to my home directory.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">sudo</span> <span class="token function">apt</span> <span class="token function">install</span> <span class="token parameter variable">-y</span> <span class="token function">git</span> stow</code></pre>
<h2 id="h-3---clone-my-dotfiles-repo-with-https" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-3---clone-my-dotfiles-repo-with-https"><span aria-hidden="true">#</span></a> 3 - Clone my dotfiles repo with HTTPS</h2>
<p>I keep all my git repos in a single directory.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">mkdir</span> ~/repos</code></pre>
<p>I can’t yet clone a git repo via SSH because I haven’t copied the SSH keys, so I use HTTPS for now.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> clone https://github.com/jackdbd/dotfiles.git</code></pre>
<h2 id="h-4---setup-bash" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-4---setup-bash"><span aria-hidden="true">#</span></a> 4 - setup bash</h2>
<p>I keep my bash files (<code>.bashrc</code>, <code>.bash_logout</code>, <code>bash_aliases</code>) in <a href="https://github.com/jackdbd/dotfiles">my dotfiles repo</a>, so I remove the ones that come with the distro:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">rm</span> ~/.bashrc
<span class="token function">rm</span> ~/.bash_logout</code></pre>
<p>I create symlinks from the dotfiles in <code>mkdir ~/repos/dotfiles/bash</code> to <code>~</code> with Stow.</p>
<pre class="language-shell"><code class="language-shell"><span class="token builtin class-name">cd</span> ~/repos/dotfiles/bash
stow <span class="token builtin class-name">.</span> <span class="token parameter variable">--target</span> ~</code></pre>
<h2 id="h-5---setup-ssh" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-5---setup-ssh"><span aria-hidden="true">#</span></a> 5 - setup SSH</h2>
<p>I keep some of my SSH config files—obviously neither the SSH keys nor the config files for private servers—in my dotfiles repo.</p>
<p>Here is how I create the symlinks with Stow.</p>
<pre class="language-shell"><code class="language-shell"><span class="token builtin class-name">cd</span> ~/repos/dotfiles/ssh
stow <span class="token builtin class-name">.</span> —target ~</code></pre>
<p>I can now copy my SSH keys from an USB stick. It’s going to be something like this:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">cp</span> <span class="token parameter variable">-r</span> /media/jack/<span class="token environment constant">PATH</span>-TO-SSH-KEYS-ON-USB-STICK/ ~/.ssh</code></pre>
<p>If I now try to connect to one of my remote servers with SSH, the SSH agent would probably complain and give me a permission denied error. This is because the file permissions for the SSH keys are too open. To fix this, I change the file permissions to be read-only.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">chmod</span> <span class="token number">400</span> ~/.ssh/keys/*
<span class="token function">chmod</span> <span class="token number">400</span> ~/.ssh/conf.d/*
<span class="token function">chmod</span> <span class="token number">400</span> ~/.ssh/config</code></pre>
<p>I then need to add the SSH keys to ssh-agent, the program that keeps track of user’s identity keys and passphrases. The command for that is simply <code>ssh-add PATH-TO-SSH-KEY</code>, but I don’t type it because I keep it in my <code>.bashrc</code>. So all I have to do is this:</p>
<pre class="language-shell"><code class="language-shell"><span class="token builtin class-name">source</span> ~.bashrc</code></pre>
<p>I can now test that I can connect to remote services using the aliases I defined in my SSH config.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">ssh</span> github
<span class="token function">ssh</span> gitlab
<span class="token comment"># etc</span></code></pre>
<h2 id="h-6---clone-my-ansible-laptop-repo-with-https" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-6---clone-my-ansible-laptop-repo-with-https"><span aria-hidden="true">#</span></a> 6 - Clone my ansible-laptop repo with HTTPS</h2>
<p>At this point I still haven’t installed the password manager extension (Lastpass) for my favorite browser (Chromium), so I can’t yet use SSH to clone a git repo from Github (because I use a super long password for Github and I don’t remember it). So I use HTTPS once again:</p>
<pre class="language-shell"><code class="language-shell"><span class="token builtin class-name">cd</span> ~/repos
<span class="token function">git</span> clone https://github.com/jackdbd/ansible-laptop.git
<span class="token builtin class-name">cd</span> ~/repos/ansible-laptop</code></pre>
<h2 id="h-7---install-ansible" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-7---install-ansible"><span aria-hidden="true">#</span></a> 7 - Install Ansible</h2>
<p>There are several ways to install Ansible on Ubuntu-based distros. I do as they say in the <a href="https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible-on-ubuntu">official documentation</a>:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">sudo</span> <span class="token function">apt</span> update
<span class="token function">sudo</span> <span class="token function">apt</span> <span class="token function">install</span> software-properties-common
<span class="token function">sudo</span> apt-add-repository <span class="token parameter variable">--yes</span> <span class="token parameter variable">--update</span> ppa:ansible/ansible
<span class="token function">sudo</span> <span class="token function">apt</span> <span class="token function">install</span> ansible</code></pre>
<h2 id="h-8---install-all-the-things" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-8---install-all-the-things"><span aria-hidden="true">#</span></a> 8 - Install all the things!</h2>
<p>Some of the tools I install with Ansible are just binaries. I need to place them in a single directory so I can add them all to my <code>$PATH</code>.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">mkdir</span> ~/bin</code></pre>
<p>Finally I can setup my laptop with the configuration I keep in my <a href="https://github.com/jackdbd/ansible-laptop">ansible-laptop</a> repo. I need two commands for this: <code>ansible-galaxy</code> and <code>ansible-playbook</code>.</p>
<p>I use a couple of Ansible roles from <a href="https://galaxy.ansible.com/docs/">Ansible Galaxy</a>. If you don’t know Ansible Galaxy, it’s a public repository for Ansible roles. Think of Docker Hub but for Ansible. I install <a href="https://galaxy.ansible.com/geerlingguy/java">java</a> and <a href="https://galaxy.ansible.com/geerlingguy/docker">docker</a> this way.</p>
<pre class="language-shell"><code class="language-shell">ansible-galaxy <span class="token function">install</span> <span class="token parameter variable">-r</span> requirements.yml</code></pre>
<p>All other software I need is configured in my Ansible playbook, that I run with this command:</p>
<pre class="language-shell"><code class="language-shell">ansible-playbook playbook.yml <span class="token parameter variable">-K</span></code></pre>
<p><em>Note</em>: I use <a href="https://www.nerdfonts.com/">Nerd Fonts</a> to install custom fonts on my machine. At the moment I’m cloning the entire repo, which is probably way too many fonts I will ever need. Since cloning the entire repo takes a long time, you might want to skip it if you are in a hurry. You can do it with <code>--skip-tags</code>.</p>
<pre class="language-shell"><code class="language-shell">ansible-playbook playbook.yml <span class="token parameter variable">-K</span> --skip-tags <span class="token string">"fonts"</span></code></pre>
<h2 id="h-9---install-lastpass-on-chromium" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-9---install-lastpass-on-chromium"><span aria-hidden="true">#</span></a> 9 - Install Lastpass on Chromium</h2>
<p>I use a bunch of browser extensions, and I keep them synced across devices. So just I install Lastpass and sync all the other extensions.</p>
<h2 id="h-10---re-clone-my-dotfiles-and-ansible-laptop-repos-via-ssh" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-i-setup-my-laptop-with-ansible/#h-10---re-clone-my-dotfiles-and-ansible-laptop-repos-via-ssh"><span aria-hidden="true">#</span></a> 10 - Re-clone my dotfiles and ansible-laptop repos via SSH</h2>
<p>Now that I can login to my Github account with Lastpass and that I can connect via SSH, I re-clone my repos with SSH. This way I don’t have to type my password every time I pull from a git remote or push code to it.</p>
https://www.giacomodebidda.com/posts/flexible-runtime-polymorphism-with-clojure-multimethods/Flexible runtime polymorphism with Clojure multimethods2020-09-12T12:00:00Z2020-09-12T12:00:00ZPolymorphism is a generic concept that means providing many implementations while retaining a single interface. Clojure multimethods allow us to use multiple dynamic dispatch.
<p>Polymorphism is a generic concept that means providing many implementations while retaining a single interface.</p>
<p>There are many kinds of polymorphism in computer science. There is static polymorphism, where the implementation is chosen at compile time, and then there is with dynamic polymorphism, where it is chosen at run time.<br />
This article will only discuss the latter.</p>
<p>The mechanism used by dynamic polymorphism to select the function/method to invoke at run time is called <a href="https://en.wikipedia.org/wiki/Dynamic_dispatch">dynamic dispatch</a>, or simply dispatch.</p>
<h2 id="h-single-dispatch" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/flexible-runtime-polymorphism-with-clojure-multimethods/#h-single-dispatch"><span aria-hidden="true">#</span></a> Single dispatch</h2>
<p>Many languages can dispatch on the <em>type</em> of the <em>first</em> argument.</p>
<p>Take for example this Python class:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">Animal</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">sound</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">"foo"</span></code></pre>
<p>We can create a subclass like this one, and <em>override</em> the <code>sound</code> method:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">Cat</span><span class="token punctuation">(</span>Animal<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">sound</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">"meow"</span></code></pre>
<p>If we then create an instance of <code>Cat</code> and call <code>sound()</code> on it, we get <code>"meow"</code> as the return value.</p>
<pre class="language-python"><code class="language-python">cat <span class="token operator">=</span> Cat<span class="token punctuation">(</span><span class="token punctuation">)</span>
cat<span class="token punctuation">.</span>sound<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token string">"meow"</span></code></pre>
<p>This is <em>runtime subtype-based polymorphism</em> in action: the behavior—what the <code>sound</code> method is supposed to do—is chosen dynamically based on the runtime type of the object. When we call <code>cat.sound()</code>, the <code>sound</code> method receives the instance of <code>Cat</code> it is bound to. This one argument (i.e. <code>self</code>) is used to select which implementation of <code>sound</code> to run.<br />
To put it in other terms, there can be many implementations of <code>sound</code>, but when we call <code>cat.sound()</code> it’s the <code>sound</code> method in the <code>Cat</code> class the one who gets called.</p>
<p>Many other programming languages support this single dispatch mechanism. On the other hand, only a few ones have a multiple dispatch mechanism.</p>
<h2 id="h-multiple-dispatch-aka-multimethods" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/flexible-runtime-polymorphism-with-clojure-multimethods/#h-multiple-dispatch-aka-multimethods"><span aria-hidden="true">#</span></a> Multiple dispatch (aka multimethods)</h2>
<p>In languages that support a multiple dispatch mechanism, one or more of the arguments passed to a method are used to select the method to run. Clojure and ClojureScript are two of these languages.</p>
<p>In particular, Clojure/ClojureScript multimethods support dispatching on:</p>
<ul>
<li>types of any argument passed to the dispatch function;</li>
<li>values of any argument;</li>
<li>attributes and metadata;</li>
<li>relationships between one or more arguments.</li>
</ul>
<p>When using multimethods in Clojure, we associate <strong>one symbol</strong> with <strong>multiple implementations</strong> (methods) by defining a dispatch function. The dispatch function returns a <strong>dispatch value</strong> that is used to determine which method to use.</p>
<h3 id="h-multimethods-a-basic-example" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/flexible-runtime-polymorphism-with-clojure-multimethods/#h-multimethods-a-basic-example"><span aria-hidden="true">#</span></a> Multimethods: a basic example</h3>
<p>Let’s see an example. If you want to follow along, I suggest you type the code in a ClojureScript REPL like <a href="https://github.com/planck-repl/planck">Planck</a>.</p>
<p>First of all, we need to define a multimethod with the <a href="https://cljs.github.io/api/cljs.core/defmulti">defmulti</a> macro.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token keyword">defmulti</span> draw <span class="token symbol">:shape</span><span class="token punctuation">)</span></code></pre>
<p>Here the symbol <code>draw</code> represents the unique interface, and the dispatch function is <code>:shape</code> (Clojure/ClojureScript can call keywords because they implement the <a href="https://clojure.org/reference/data_structures#Keywords">same interface as functions</a>).</p>
<p>We then provide an implementation with the <a href="https://cljs.github.io/api/cljs.core/defmethod">defmethod</a> macro.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token keyword">defmethod</span> draw <span class="token symbol">:triangle</span> <span class="token punctuation">[</span>options<span class="token punctuation">]</span>
<span class="token punctuation">(</span><span class="token keyword">str</span> <span class="token string">"Drawing a "</span> <span class="token punctuation">(</span><span class="token symbol">:color</span> options<span class="token punctuation">)</span> <span class="token string">" triangle"</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>And we call the <code>draw</code> (multi) function.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token function">draw</span> <span class="token punctuation">{</span><span class="token symbol">:shape</span> <span class="token symbol">:triangle</span> <span class="token symbol">:color</span> <span class="token string">"red"</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">; "Drawing a red triangle"</span></code></pre>
<p>Here are the steps involved in the few lines of code written above.</p>
<ol>
<li>We call the <code>draw</code> multimethod.</li>
<li><code>draw</code> is a <a href="https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/MultiFn.java">MultiFn</a>, so it can have several implementations. In order to decide which implementation to call, it asks the <code>:shape</code> dispatch function to come with a dispatch value that will determine the chosen implementation.</li>
<li>The <code>:shape</code> dispatch function (remember, keywords behave as functions in Clojure/ClojureScript) extracts the value associated with the <code>:shape</code> key from the data structure we passed: <code>:triangle</code>. That’s the dispatch value.</li>
<li>We previously <em>installed</em> (it’s the term used by <code>defmethod</code>) a method associated with the dispatch value <code>:triangle</code>, so <code>draw</code> calls that method.</li>
</ol>
<p>What if the dispatch function returns a dispatch value for which we didn’t provide an implementation?</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token function">draw</span> <span class="token punctuation">{</span><span class="token symbol">:shape</span> <span class="token symbol">:square</span> <span class="token symbol">:color</span> <span class="token string">"red"</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">; Execution error (Error) at (<cljs repl>:1).</span>
<span class="token comment">; No method in multimethod 'cljs.user/draw' for dispatch value: :square</span></code></pre>
<p>Here the error message is pretty clear: we didn’t provide an implementation for when the dispatch value is <code>:square</code>. Let’s provide it now.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token keyword">defmethod</span> draw <span class="token symbol">:square</span> <span class="token punctuation">[</span>options<span class="token punctuation">]</span>
<span class="token punctuation">(</span><span class="token keyword">str</span> <span class="token string">"Drawing a "</span> <span class="token punctuation">(</span><span class="token symbol">:color</span> options<span class="token punctuation">)</span> <span class="token string">" square"</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>Now if we try to call <code>draw</code> we get no errors.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token function">draw</span> <span class="token punctuation">{</span><span class="token symbol">:shape</span> <span class="token symbol">:square</span> <span class="token symbol">:color</span> <span class="token string">"red"</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">; "Drawing a red square"</span></code></pre>
<p>I hope this basic example was helpful in understanding how this dynamic dispatch mechanism works. However, here we are only dispatching on the first argument, so it’s just single dispatch, not multiple dispatch.</p>
<h3 id="h-multimethods-a-more-involved-example" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/flexible-runtime-polymorphism-with-clojure-multimethods/#h-multimethods-a-more-involved-example"><span aria-hidden="true">#</span></a> Multimethods: a more involved example</h3>
<p>Let’s see an example of an actual multiple dispatch where we really need the flexibility that Clojure multimethods have to offer.</p>
<p>This time, let’s first define the dispatch function by itself. I think it makes the example a bit easier to understand.</p>
<p>The dispatch function is like any other function in Clojure. We can give it any name we want and we can add as many parameters as we want in its signature.</p>
<p>Let’s say that we are developing a graphics engine and we want to provide our users with a unique interface to call, the <code>draw</code> function. They can call it with a shape (e.g. <code>"triangle"</code>), the number of shapes they want to draw (e.g. 10), and some options. We are at the early stages of development, so our graphics engine can’t deal with too many shapes (especially if anti-aliasing is on).</p>
<p>Our dispatch function would look like this:</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token keyword">defn</span> engine-dispatch-fn <span class="token punctuation">[</span>shape quantity options<span class="token punctuation">]</span>
<span class="token punctuation">(</span><span class="token keyword">cond</span>
<span class="token punctuation">(</span><span class="token keyword">>=</span> quantity <span class="token number">300</span><span class="token punctuation">)</span> <span class="token symbol">:too-many-shapes</span>
<span class="token punctuation">(</span><span class="token keyword">and</span> <span class="token punctuation">(</span><span class="token keyword">></span> quantity <span class="token number">100</span><span class="token punctuation">)</span> <span class="token punctuation">(</span><span class="token keyword">get</span> options <span class="token symbol">:anti-aliasing</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token symbol">:too-many-shapes-with-anti-aliasing-on</span>
<span class="token punctuation">(</span><span class="token keyword">></span> quantity <span class="token number">0</span><span class="token punctuation">)</span> shape
<span class="token symbol">:else</span> <span class="token symbol">:default</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>We create a new multimethod with the associated dispatch function.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token keyword">defmulti</span> draw engine-dispatch-fn<span class="token punctuation">)</span></code></pre>
<p>And we install the method responsible for drawing triangles on the screen.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token keyword">defmethod</span> draw <span class="token string">"triangle"</span> <span class="token punctuation">[</span>_ quantity options<span class="token punctuation">]</span>
<span class="token punctuation">(</span><span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token keyword">=</span> <span class="token boolean">true</span> <span class="token punctuation">(</span><span class="token keyword">get</span> options <span class="token symbol">:anti-aliasing</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token keyword">str</span> <span class="token string">"Drawing "</span> quantity <span class="token string">" triangles with anti aliasing"</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token keyword">str</span> <span class="token string">"Drawing "</span> quantity <span class="token string">" triangles"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token keyword">defmethod</span> draw <span class="token symbol">:too-many-shapes</span> <span class="token punctuation">[</span>shape quantity _<span class="token punctuation">]</span>
<span class="token punctuation">(</span><span class="token keyword">throw</span> <span class="token punctuation">(</span><span class="token function">js/Error.</span> <span class="token punctuation">(</span><span class="token keyword">str</span> <span class="token string">"Can't draw "</span> quantity <span class="token string">" "</span> shape <span class="token string">"s"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token keyword">defmethod</span> draw <span class="token symbol">:too-many-shapes-with-anti-aliasing-on</span> <span class="token punctuation">[</span>shape quantity _<span class="token punctuation">]</span>
<span class="token punctuation">(</span><span class="token keyword">throw</span> <span class="token punctuation">(</span><span class="token function">js/Error.</span> <span class="token punctuation">(</span><span class="token keyword">str</span> <span class="token string">"Can't draw "</span> quantity <span class="token string">" "</span> shape <span class="token string">"s when anti-aliasing is on"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token keyword">defmethod</span> draw <span class="token symbol">:default</span> <span class="token punctuation">[</span>shape quantity options<span class="token punctuation">]</span>
<span class="token punctuation">(</span><span class="token keyword">let</span> <span class="token punctuation">[</span>msg <span class="token punctuation">(</span><span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token keyword">nil?</span> options<span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token keyword">str</span> <span class="token string">"Cannot draw "</span> quantity <span class="token string">" "</span> shape <span class="token string">"s"</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token keyword">str</span> <span class="token string">"Cannot draw "</span> quantity <span class="token string">" "</span> shape <span class="token string">"s with options "</span> options<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">]</span>
<span class="token punctuation">(</span><span class="token keyword">throw</span> <span class="token punctuation">(</span><span class="token function">js/Error.</span> msg<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>When our users try to draw some triangles, here is what they would get:</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token function">draw</span> <span class="token string">"triangle"</span> <span class="token number">101</span><span class="token punctuation">)</span>
<span class="token comment">; "Drawing 101 triangles"</span>
<span class="token punctuation">(</span><span class="token function">draw</span> <span class="token string">"triangle"</span> <span class="token number">101</span> <span class="token punctuation">{</span><span class="token symbol">:anti-aliasing</span> <span class="token boolean">true</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">; "Can't draw 101 triangles when anti-aliasing is on"</span></code></pre>
<h2 id="h-conclusion" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/flexible-runtime-polymorphism-with-clojure-multimethods/#h-conclusion"><span aria-hidden="true">#</span></a> Conclusion</h2>
<p>Multimethods allow us to extend a system without modifying existing code (if we don’t have to touch the dispatch function). With their multiple dispatch mechanism, they offer us the highest degree of runtime polymorphism. This article showed only a couple of examples, but if you want to master the subject I recommend Eli Bendersky’s series of blog posts <a href="https://eli.thegreenplace.net/tag/multiple-dispatch">A polyglot’s guide to multiple dispatch</a>.</p>
<p>The “problem” with multiple dispatch is that we often don’t need this flexibility in our code. For most use-case single dispatch is enough. In that case we are better off using another Clojure feature: protocols.</p>
https://www.giacomodebidda.com/posts/event-storming-simplified/Event Storming, simplified2020-09-11T12:00:00Z2020-09-11T12:00:00Z
<p>The first time I’ve heard about Event Storming was when I watched the excellent talk <a href="https://www.infoq.com/presentations/microservices-events-first-design/">Designing event-first microservices</a> by Jonas Bonér, creator of the <a href="https://github.com/akka/akka">Akka</a> project.</p>
<p>Event Storming is a workshop-based method that you can employ when designing a new system or product. You run this workshop before developing any feature. You gather engineers, domain experts and decision makers, and you make them write post-it notes where they write <strong>domain events</strong>.<br />
A domain event is something meaningful that <strong>happened</strong> in the domain. The emphasys on happened is because domain events must be expressed with the <strong>past tense</strong>.</p>
<h2 id="h-why-starting-from-events%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/event-storming-simplified/#h-why-starting-from-events%3F"><span aria-hidden="true">#</span></a> Why starting from events?</h2>
<p>Event Storming starts from events because it recognizes that a system should be designed with its domain in mind (Domain-Driven Design), and in In DDD events are first-class citizens: you view the world (i.e. the business domain) through events.</p>
<p>As Greg Young—creator of the Event Sourcing architectural pattern—puts it:</p>
<blockquote>
<p>When you start modeling events, it forces you to think about the behavior of the system. As opposed to thinking about the structure of the system.</p>
</blockquote>
<h2 id="h-commands-and-actors" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/event-storming-simplified/#h-commands-and-actors"><span aria-hidden="true">#</span></a> Commands and Actors</h2>
<p>Events do not appear from nothing. They are the product of some process executed by some person or some program.<br />
Following the Event Storming terminology, the process is called <strong>command</strong>, and the person or thing that executed the command is called <strong>actor</strong>.</p>
<pre class="language-text"><code class="language-text">Actor --> |executes| --> Command --> |causes| --> Domain Event</code></pre>
<h2 id="h-other-components" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/event-storming-simplified/#h-other-components"><span aria-hidden="true">#</span></a> Other components</h2>
<p>Event Storming introduces additional components:</p>
<ul>
<li><strong>Aggregates</strong>: a cluster of domain objects that can be treated as a single unit.</li>
<li><strong>External System</strong>: some third-party service provider such as a payment gateway or shipping company.</li>
<li><strong>Business Process</strong>: it processes a command according to business rules and logic. Creates one or more domain events.</li>
<li><strong>User</strong>: the person who executes a command through a view.</li>
<li><strong>View</strong>: the view the users interact with to carry out a task in the system.</li>
</ul>
<p>I don’t think that these components are really necessary though. Or at least they are far less important than domain events, commands and actors. Here is why.</p>
<p><strong>Aggregates</strong>. Since we are using Event Storming before developing any feature, it can be hard or impossible to group domain events in a single unit. Another issue is that there can be some overlap between aggregates: the same domain event could belong to more than one aggregate. As far as I know, Event Storming doesn’t propose a solution on how to deal with this issue. Also, I am not sure about the real value of knowing the aggregates of the business domain we are modeling. I think it can be useful from an engineering standpoint, but less so from a business standpoint.</p>
<p><strong>External System</strong>. I think we should consider third-party service providers as just another actor. In fact, in a microservice architecture there should be no difference between a service implemented internally in our company and a service from a third-party provider: both services are opaque to us and we can communicate with them only via their public APIs.</p>
<p><strong>Business Process</strong>. Again, a business process <em>processes commands</em>, so I think it’s fair to view it as just another actor.</p>
<p><strong>User</strong>. Sure, it can be valuable to identify users of our system. But users execute commands, so they are just a particular kind of actors.</p>
<p><strong>View</strong>. First of all, there can be no view. If the actor is a cron process that executes a command, and that command creates some domain event, then there is no view involved. Second, even when there is a user operating a portion of the system, I don’t think that identifying the view at this stage of the requirements gathering is that important.</p>
<h2 id="h-conclusion" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/event-storming-simplified/#h-conclusion"><span aria-hidden="true">#</span></a> Conclusion</h2>
<p>Event Storming can be a great way to gather business and engineering requirements. I agree that viewing the world through events—namely modeling our business domain starting from domain events—is a valid approach. At the moment I just don’t see the need of introducing additional concepts when actors, command and events suffice. It’s possible that I’m just missing something important though. I know that Alberto Brandolini is writing a <a href="https://en.wikipedia.org/wiki/Event_storming">book about Event Storming</a>, and I plan to read it when it’s ready. I might change my mind if I find something interesting in it.</p>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/event-storming-simplified/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<ul>
<li><a href="https://ziobrando.blogspot.com/2013/11/introducing-event-storming.html">Introducing Event Storming (by Alberto Brandolini)</a></li>
<li><a href="https://en.wikipedia.org/wiki/Event_storming">Event Storming (Wikipedia)</a></li>
</ul>
https://www.giacomodebidda.com/posts/3-ways-to-solve-the-expression-problem/3 ways to solve the expression problem2020-09-10T12:00:00Z2020-09-10T12:00:00ZThe expression problem is about adding new capabilities to existing code without modifying it. In this article we discuss a few approaches to solve the expression problem.
<p>The expression problem—also known as the extensibility problem—is about adding new capabilities to existing code without modifying it.</p>
<p>The term was coined by Philip Wadler at Bell Labs in the late 90s. As he put it:</p>
<blockquote>
<p>The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety (e.g., no casts).</p>
</blockquote>
<p>Given that we might use a dynamic language instead of a language that require a compilation step to introduce new functionality, I suggest we view the <em>existing code</em> Wadler talks about, as code that comes from a library we don’t have access to and we can’t modify directly. Let’s image we can’t read the source code of the original library and we can only use its public API.</p>
<p>To recap, here is what we want to achieve:</p>
<ol>
<li>use the <strong>existing types</strong> provided by a third-party library, and <strong>introduce new operations</strong> on such types;</li>
<li>use the <strong>existing operations</strong> provided by the library, and <strong>introduce new types</strong> that can be acted upon by such operations.</li>
</ol>
<p>Cool. How do we go about solving goals 1 and 2?</p>
<h2 id="h-different-approaches-to-solve-the-expression-problem" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/3-ways-to-solve-the-expression-problem/#h-different-approaches-to-solve-the-expression-problem"><span aria-hidden="true">#</span></a> Different approaches to solve the expression problem</h2>
<p>In <strong>object-oriented languages goal 1 is hard, goal 2 is easy</strong>: introducing new operations on existing types is hard, because we would need to modify all existing types to support the new operations (and we might not be able to do it, if we don’t control the code which defines such types); on the other hand, introducing new types is easy, because we can just subclass the existing types.</p>
<p>In <strong>functional languages goal 1 is easy, goal 2 is hard</strong>: introducing new operations on existing types is easy, because we can just create a new function; but introducing new types is hard, because we would need to edit every function that accepts the new types we want to support.</p>
<p>So, neither with object-oriented languages nor with functional languages we can achieve goal 1 and 2 easily, and we need to rely on additional language features or design patterns to do that.</p>
<p>Here are three approaches we can take to solve the expression problem:</p>
<ul>
<li>open classes (aka monkey patching)</li>
<li>multiple dynamic dispatch (aka multimethods)</li>
<li>single dynamic dispatch</li>
</ul>
<p>Not all languages support all of these features. Let’s see a few examples.</p>
<h2 id="h-open-classes-aka-monkey-patching" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/3-ways-to-solve-the-expression-problem/#h-open-classes-aka-monkey-patching"><span aria-hidden="true">#</span></a> Open classes (aka monkey patching)</h2>
<p>Dynamic languages such as Ruby, Python and Javascript can redefine at runtime any code they need to extend.</p>
<p>Let’s say there is a class that we want to extend. Here is what we have to do:</p>
<ol>
<li>import the class we want to extend;</li>
<li>define new methods or redefine existing ones;</li>
<li>attach them to the class we want to extend.</li>
</ol>
<p>Step 1: import the class we want to extend.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">MyClass</span><span class="token punctuation">:</span>
a <span class="token operator">=</span> <span class="token number">1</span>
b <span class="token operator">=</span> <span class="token string">'2'</span>
<span class="token keyword">def</span> <span class="token function">get_value</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>a</code></pre>
<p>Step 2: define a new method.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">get_another_value</span><span class="token punctuation">(</span>cls<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> cls<span class="token punctuation">.</span>b</code></pre>
<p>Step 3: attach the new method to the existing class.</p>
<pre class="language-python"><code class="language-python">MyClass<span class="token punctuation">.</span>get_another_value <span class="token operator">=</span> get_another_value</code></pre>
<p>With these 3 steps we achieved goal 1: introduce new operations on existing types.</p>
<p>What about goal 2, namely introduce new types that can be acted upon by existing operations?</p>
<p>This is a non-issue in dynamic languages like Python and Javascript. Since in these languages functions are first-class citizens, we can pass them around and bind them to our objects at runtime. And thanks to duck typing we can use existing functions with our new objects.</p>
<p>In Python we can even allow our classes to define their own behavior with respect to language operators. This can be done with <a href="https://docs.python.org/3/reference/datamodel.html#special-method-names">special method names</a>, commonly known as magic methods.</p>
<p>So if we have an existing class like this:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">FooParent</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">bar</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">"baz"</span></code></pre>
<p>and we want to introduce new functionality when instances of this class are garbage collected, we can extend <code>FooParent</code> and overload <code>__del__</code>:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">FooChild</span><span class="token punctuation">(</span>FooParent<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__del__</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"I am garbage collected!"</span><span class="token punctuation">)</span></code></pre>
<p>So when we write:</p>
<pre class="language-python"><code class="language-python">foo <span class="token operator">=</span> FooChild<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">del</span> foo</code></pre>
<p>we <a href="https://stackoverflow.com/questions/1481488/what-is-the-del-method-and-how-do-i-call-it/1481512#1481512">might</a> get <code>"I am garbage collected!"</code>.</p>
<p>JavaScript is based on prototypes rather than classes, so monkey patching involves extending the prototype of the object we want to extend. If we have an application and plan to extend the prototype of a library that no other library uses, that might be ok, but extending native objects such as <code>String</code> is a <a href="https://www.reddit.com/r/javascript/comments/5ch66r/why_is_extending_native_objects_such_as_string/">big no-no</a>.</p>
<p>Keep in mind that the problem here is that these native Javascript objects have a <a href="https://softwareengineering.stackexchange.com/questions/287827/whats-wrong-about-extending-a-class-with-prototype-methods">global scope</a>. ClojureScript is able to bypass this issue and extend native objects prototypes safely because it extends the JS prototypes per namespace (if you want to know more about it, watch the talk “ClojureScript Anatomy” at around 19’25").</p>
<p><a href="https://www.youtube.com/watch?v=lC39ifspIf4">https://www.youtube.com/watch?v=lC39ifspIf4</a></p>
<p>So, monkey patching can solve the expression problem. It’s convenient and easy to understand. It has several problems though. First of all, only dynamic languages can use it. Second, it’s easy to make a mess and forget what code we monkey patched and why. There are some ways to mitigate these issues. For example, in Ruby we can <a href="https://www.justinweiss.com/articles/3-ways-to-monkey-patch-without-making-a-mess/">scope our monkey patches in a module</a> or use <a href="https://blog.alex-miller.co/ruby/2017/07/22/scope-the-monkey.html">Ruby refinements</a>.</p>
<h2 id="h-multiple-dynamic-dispatch-aka-multimethods" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/3-ways-to-solve-the-expression-problem/#h-multiple-dynamic-dispatch-aka-multimethods"><span aria-hidden="true">#</span></a> Multiple dynamic dispatch (aka multimethods)</h2>
<p>Some—though not many—programming languages support multiple dispatch. In these languages a function uses more than one piece of information to determine which function to actually call (runtime polymorphism). Usually the pieces of information are the types of the arguments passed to the function.</p>
<p>A language <a href="https://nbviewer.org/gist/StefanKarpinski/b8fe9dbb36c1427b9f22">designed with multiple dispatch in mind</a> is Julia. In fact in Julia multiple dispatch is so at the core of the language that <code>+</code> is a generic function with 96 implementations. And since generic functions are open, functions are more like protocols which users can also implement.</p>
<p>Let’s say we have a function <code>f</code> which comes from an existing library we don’t control (if you want you can try this code in a <a href="https://docs.julialang.org/en/v1/stdlib/REPL/">Julia REPL</a>).</p>
<pre class="language-julia"><code class="language-julia">f<span class="token punctuation">(</span>x<span class="token punctuation">::</span>Float64<span class="token punctuation">,</span> y<span class="token punctuation">::</span>Float64<span class="token punctuation">)</span> <span class="token operator">=</span> <span class="token number">2</span>x <span class="token operator">+</span> y</code></pre>
<p>If we call this function with <code>f(2.0, 3.0)</code> we get <code>7.0</code>. That’s fine and dandy, but what if we write <code>f(2.0, 3)</code>? If we do, we get this error.</p>
<pre class="language-shell"><code class="language-shell">julia<span class="token operator">></span> f<span class="token punctuation">(</span><span class="token number">2.0</span>, <span class="token number">3</span><span class="token punctuation">)</span>
ERROR: MethodError: no method matching f<span class="token punctuation">(</span>::Float64, ::Int32<span class="token punctuation">)</span>
Closest candidates are:
f<span class="token punctuation">(</span>::Float64, ::Float64<span class="token punctuation">)</span> at REPL<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span>:1
Stacktrace:
<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> top-level scope at none:0</code></pre>
<p><em>On a sidenote, I think that Julia error messages are pretty great, maybe on par with <a href="https://elm-lang.org/">Elm</a> ones.</em></p>
<p>We would really like to call <code>f</code> with an integer as its second argument. So what do we do? Well, in Julia we can simply define a new version of the function <code>f</code>:</p>
<pre class="language-julia"><code class="language-julia">f<span class="token punctuation">(</span>x<span class="token punctuation">::</span>Float64<span class="token punctuation">,</span> y<span class="token punctuation">::</span>Integer<span class="token punctuation">)</span> <span class="token operator">=</span> <span class="token number">2</span>x <span class="token operator">+</span> y</code></pre>
<p>Now if we call <code>(2.0, 3)</code> we get <code>7.0</code>.</p>
<p>Another language which supports multimethods is Clojure, but I’ll write about Clojure multimethods in a future blog post.</p>
<h2 id="h-single-dynamic-dispatch" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/3-ways-to-solve-the-expression-problem/#h-single-dynamic-dispatch"><span aria-hidden="true">#</span></a> Single dynamic dispatch</h2>
<p>In some cases the type of the first argument in a function or method is enough to determine which function to call at runtime. A language that supports this flavor of runtime polymorphism in an elegant and performant way is Clojure.</p>
<p>Clojure methods live outside of types. They don’t have to be part of a class like in Java or C++.</p>
<p>Have look at <a href="https://gist.github.com/elnygren/e34368a86d62f0cb75f04ba903f7834a">this clojure gist</a>.</p>
<p><code>Triangle</code> and <code>Square</code> are two data structures that obey the <code>Areable</code> protocol and the <code>SelfAware</code> protocol. If you are not familiar with Clojure, think of them as Java classes <code>Triangle</code> and <code>Square</code> that implement both the <code>Areable</code> and the <code>SelfAware</code> interfaces. These clojure protocols (or Java interfaces) define the <code>area</code> method and the <code>whoami</code> method.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token comment">; data structures ("shapes")</span>
<span class="token punctuation">(</span><span class="token keyword">defrecord</span> Triangle <span class="token punctuation">[</span>a b c<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token keyword">defrecord</span> Square <span class="token punctuation">[</span>edge<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token comment">; protocols</span>
<span class="token punctuation">(</span><span class="token keyword">defprotocol</span> Areable
<span class="token punctuation">(</span><span class="token function">area</span> <span class="token punctuation">[</span>shape<span class="token punctuation">]</span> <span class="token string">"calculates the shape's area"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token keyword">defprotocol</span> SelfAware
<span class="token punctuation">(</span><span class="token function">whoami</span> <span class="token punctuation">[</span>shape<span class="token punctuation">]</span> <span class="token string">"returns the name of the shape"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment">; implementations</span>
<span class="token punctuation">(</span><span class="token function">extend-type</span> Triangle
Areable
<span class="token punctuation">(</span><span class="token function">area</span> <span class="token punctuation">[</span><span class="token punctuation">{</span><span class="token symbol">:keys</span> <span class="token punctuation">[</span>a b c<span class="token punctuation">]</span><span class="token punctuation">}</span><span class="token punctuation">]</span>
<span class="token string">"use Heron's formula to calculate area"</span>
<span class="token punctuation">(</span><span class="token keyword">let</span> <span class="token punctuation">[</span>s <span class="token punctuation">(</span><span class="token keyword">/</span> <span class="token punctuation">(</span><span class="token keyword">+</span> a b c<span class="token punctuation">)</span> <span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">]</span>
<span class="token punctuation">(</span><span class="token function">Math/sqrt</span> <span class="token punctuation">(</span><span class="token keyword">*</span> s <span class="token punctuation">(</span><span class="token keyword">-</span> s a<span class="token punctuation">)</span> <span class="token punctuation">(</span><span class="token keyword">-</span> s b<span class="token punctuation">)</span> <span class="token punctuation">(</span><span class="token keyword">-</span> s c<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
SelfAware
<span class="token punctuation">(</span><span class="token function">whoami</span> <span class="token punctuation">[</span>this<span class="token punctuation">]</span> <span class="token string">"Triangle"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token function">extend-type</span> Square
Areable
<span class="token punctuation">(</span><span class="token function">area</span> <span class="token punctuation">[</span>this<span class="token punctuation">]</span> <span class="token punctuation">(</span><span class="token keyword">*</span> <span class="token punctuation">(</span><span class="token symbol">:edge</span> this<span class="token punctuation">)</span> <span class="token punctuation">(</span><span class="token symbol">:edge</span> this<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
SelfAware
<span class="token punctuation">(</span><span class="token function">whoami</span> <span class="token punctuation">[</span>this<span class="token punctuation">]</span> <span class="token string">"Square"</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>Let’s say that we want to extend <code>Triangle</code> and <code>Square</code> to provide new functionality that computes the perimeter.<br />
Without modyfiying existing code, in Clojure we are able to define a new <code>protocol</code> that contains the abstract definition of perimeter and provide a concrete implementation for the <code>Triangle</code> and the <code>Square</code> types.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token keyword">defprotocol</span> Perimeterable
<span class="token punctuation">(</span><span class="token function">perimeter</span> <span class="token punctuation">[</span>shape<span class="token punctuation">]</span> <span class="token string">"calculates the perimeter of the shape"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token function">extend-protocol</span> Perimeterable
Triangle
<span class="token punctuation">(</span><span class="token function">perimeter</span> <span class="token punctuation">[</span><span class="token punctuation">{</span><span class="token symbol">:keys</span> <span class="token punctuation">[</span>a b c<span class="token punctuation">]</span><span class="token punctuation">}</span><span class="token punctuation">]</span> <span class="token punctuation">(</span><span class="token keyword">+</span> a b c<span class="token punctuation">)</span><span class="token punctuation">)</span>
Square
<span class="token punctuation">(</span><span class="token function">perimeter</span> <span class="token punctuation">[</span>square<span class="token punctuation">]</span> <span class="token punctuation">(</span><span class="token keyword">*</span> <span class="token punctuation">(</span><span class="token symbol">:edge</span> square<span class="token punctuation">)</span> <span class="token number">4</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>By doing so, we gained new functionality for the existing <code>Triangle</code> and <code>Square</code> types.</p>
<pre class="language-clojure"><code class="language-clojure"><span class="token punctuation">(</span><span class="token keyword">let</span> <span class="token punctuation">[</span>triangle <span class="token punctuation">(</span><span class="token function">->Triangle</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">]</span>
<span class="token comment">; existing functionality</span>
<span class="token punctuation">(</span><span class="token function">area</span> triangle<span class="token punctuation">)</span>
<span class="token punctuation">(</span><span class="token function">whoami</span> triangle<span class="token punctuation">)</span>
<span class="token comment">; new functionality</span>
<span class="token punctuation">(</span><span class="token function">perimeter</span> triangle<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<div class="callout callout--info">
<div class="callout__content">
<p>ℹ️ — <a href="https://news.ycombinator.com/item?id=1285039">Clojure protocols can also extend final Java classes</a>, even if I still don't know how they are able to do it.</p>
</div>
</div>
<h2 id="h-other-approaches" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/3-ways-to-solve-the-expression-problem/#h-other-approaches"><span aria-hidden="true">#</span></a> Other approaches</h2>
<p>Open classes and dynamic dispatch (single or multiple) are not the only approaches we can take to solve the expression problem. Here are a few approaches I haven’t talked about in this article:</p>
<ul>
<li><a href="https://eli.thegreenplace.net/2018/more-thoughts-on-the-expression-problem-in-haskell/">Typeclasses</a></li>
<li><a href="https://i.cs.hku.hk/~bruno/oa/">Object algebras</a> (only available in languages that support generics)</li>
<li><a href="https://okmij.org/ftp/tagless-final/index.html">Tagless final</a></li>
</ul>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/3-ways-to-solve-the-expression-problem/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<p>This blog post was fairly short and introductive, but I hope it taught you a couple of things. If you want to know more about the expression problem—and especially if you are interested in Clojure—have a look at these articles:</p>
<ul>
<li><a href="https://eli.thegreenplace.net/2016/the-expression-problem-and-its-solutions/">The Expression Problem and its solutions</a></li>
<li><a href="https://www.infoq.com/presentations/Clojure-Expression-Problem/">Clojure’s Solutions to the Expression Problem</a></li>
<li><a href="https://max.computer/blog/solving-the-expression-problem-in-clojure/">Solving the expression problem in Clojure</a></li>
<li><a href="https://www.ibm.com/developerworks/library/j-clojure-protocols/">Solving the Expression Problem with Clojure 1.2</a></li>
</ul>
https://www.giacomodebidda.com/posts/easy-terminal-multiplexing-with-byobu/Easy terminal multiplexing with Byobu2019-03-20T00:00:00Z2019-03-20T00:00:00Z
<p>I spend a lot of time in the linux terminal, because it’s usually the fastest and most effective way to perform a task.</p>
<p>On average I use the terminal to navigate the file system, launch both a backend server and a frontend server, and commit code with git.</p>
<p>One obvious way to perform many tasks is to open many terminal windows. But then, switching between them with <code>Alt + Tab</code> on my Xubuntu system gets tedious pretty fast.</p>
<p>Another option, if you use VS code like me, is to open several terminal windows and perform your tasks there. The problem is that I often work with a 13.3’ laptop, and I don’t have that much screen real estate to throw away.</p>
<p>A better alternative is to use a terminal multiplexer like <a href="https://github.com/tmux/tmux/wiki">tmux</a> or <a href="https://en.wikipedia.org/wiki/GNU_Screen">GNU Screen</a>.</p>
<p>I tried tmux a couple of times, but it didn’t really stick with me. I watched a few videos from <a href="https://www.youtube.com/watch?v=ZNM1KfqpyGo&t=5s&list=PL5BE1545D8486D66D&index=2">this YouTube playlist</a>, but in the end I didn’t invest enough time to learn it properly.</p>
<p>A colleague at <a href="https://www.develer.com/">work</a> recommended me <a href="https://help.ubuntu.com/community/Byobu">Byobu</a>, so I decided to try it out.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599302017/byobu-screenshot_sbkhm6.png">https://res.cloudinary.com/jackdbd/image/upload/v1599302017/byobu-screenshot_sbkhm6.png</a></p>
<h2 id="h-installation-and-first-steps" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/easy-terminal-multiplexing-with-byobu/#h-installation-and-first-steps"><span aria-hidden="true">#</span></a> Installation and first steps</h2>
<p>Byobu is a script that launches either tmux (default) or GNU screen. You can install it with <code>sudo apt-get install byobu</code> and launch it with <code>byobu</code>.</p>
<p>If you want to see Byobu in action and learn a few basic commands, I think your best option is to watch the 10-minute tutorial by its author:</p>
<p><a href="https://www.youtube.com/watch?v=NawuGmcvKus&ab_channel=DustinKirkland">https://www.youtube.com/watch?v=NawuGmcvKus&ab_channel=DustinKirkland</a></p>
<h2 id="h-keybindings-i-use-all-the-time" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/easy-terminal-multiplexing-with-byobu/#h-keybindings-i-use-all-the-time"><span aria-hidden="true">#</span></a> Keybindings I use all the time</h2>
<p>I am sure I will keep using Byobu and maybe configure some custom keybindings in the future. For now I use these ones:</p>
<ul>
<li><code>Shift + F1</code>: show all Byobu’s keybindings (press <code>q</code> to exit).</li>
<li><code>Shift + F2</code>: split the current panel vertically.</li>
<li><code>Ctrl + F2</code>: split the current panel horizontally. <em>Note</em> Xubuntu’s Window Manager has already a keybinding for this key combination: <code>Move to Workspace 2</code>. Since I don’t use workspaces, I disabled the keybinding in Window Manager.</li>
<li><code>Shift + arrow (LEFT/RIGHT/UP/DOWN)</code>: switch between panels.</li>
<li><code>Shift + Alt + arrow (LEFT/RIGHT/UP/DOWN)</code>: resize panel.</li>
<li><code>F7</code>: enter <em>scrollback</em> mode. You can navigate past output using <em>vi-like</em> commands, like see <a href="https://help.ubuntu.com/lts/serverguide/byobu.html">here</a>. Exit with <code>Enter</code>.</li>
<li><code>Alt + PageUp/PageDown</code>: scroll back/forward.</li>
</ul>
<p>When you type <code>exit</code>, Byobu closes the current panel and resizes all other panels in the terminal window.</p>
https://www.giacomodebidda.com/posts/reading-large-excel-files-with-pandas/Reading large Excel files with Pandas2018-09-01T08:00:03Z2018-09-01T08:00:03Z
<p>Last week I took part in a <a href="https://www.reddit.com/r/dataisbeautiful/comments/950j3n/battle_dataviz_battle_for_the_month_of_august/">Dataviz Battle on the dataisbeautiful subreddit</a>, where we had to create a visualization from the <a href="https://www.dhs.gov/tsa-claims-data">TSA claims dataset</a>. I like these kind of competitions because most of the time you end up learning a lot of useful things along the way.</p>
<p>This time the data was quite clean, but it was scattered across several PDF files and Excel files. In the process of extracting data from PDFs I got to know some tools and libraries, and in the end I used <a href="https://github.com/chezou/tabula-py">tabula-py</a>, a Python wrapper for the Java library <a href="https://tabula.technology/">tabula</a>. As for the Excel files, I found out that a one-liner - a simple <code>pd.read_excel</code> - wasn’t enough.</p>
<p>The biggest Excel file was ~7MB and contained a single worksheet with ~100k lines. I though Pandas could read the file in one go without any issue (I have 10GB of RAM on my computer), but apparently I was wrong.</p>
<p>The solution was to read the file in chunks. The <code>pd.read_excel</code> function doesn’t have a cursor like <code>pd.read_sql</code>, so I had to implement this logic manually. Here is what I did:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> os
<span class="token keyword">import</span> pandas <span class="token keyword">as</span> pd
HERE <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>abspath<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>dirname<span class="token punctuation">(</span>__file__<span class="token punctuation">)</span><span class="token punctuation">)</span>
DATA_DIR <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>abspath<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>HERE<span class="token punctuation">,</span> <span class="token string">'..'</span><span class="token punctuation">,</span> <span class="token string">'data'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">make_df_from_excel</span><span class="token punctuation">(</span>file_name<span class="token punctuation">,</span> nrows<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Read from an Excel file in chunks and make a single DataFrame.
Parameters
----------
file_name : str
nrows : int
Number of rows to read at a time. These Excel files are too big,
so we can't read all rows in one go.
"""</span>
file_path <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>abspath<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>DATA_DIR<span class="token punctuation">,</span> file_name<span class="token punctuation">)</span><span class="token punctuation">)</span>
xl <span class="token operator">=</span> pd<span class="token punctuation">.</span>ExcelFile<span class="token punctuation">(</span>file_path<span class="token punctuation">)</span>
<span class="token comment"># In this case, there was only a single Worksheet in the Workbook.</span>
sheetname <span class="token operator">=</span> xl<span class="token punctuation">.</span>sheet_names<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
<span class="token comment"># Read the header outside of the loop, so all chunk reads are</span>
<span class="token comment"># consistent across all loop iterations.</span>
df_header <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_excel<span class="token punctuation">(</span>file_path<span class="token punctuation">,</span> sheetname<span class="token operator">=</span>sheetname<span class="token punctuation">,</span> nrows<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string-interpolation"><span class="token string">f"Excel file: </span><span class="token interpolation"><span class="token punctuation">{</span>file_name<span class="token punctuation">}</span></span><span class="token string"> (worksheet: </span><span class="token interpolation"><span class="token punctuation">{</span>sheetname<span class="token punctuation">}</span></span><span class="token string">)"</span></span><span class="token punctuation">)</span>
chunks <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
i_chunk <span class="token operator">=</span> <span class="token number">0</span>
<span class="token comment"># The first row is the header. We have already read it, so we skip it.</span>
skiprows <span class="token operator">=</span> <span class="token number">1</span>
<span class="token keyword">while</span> <span class="token boolean">True</span><span class="token punctuation">:</span>
df_chunk <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_excel<span class="token punctuation">(</span>
file_path<span class="token punctuation">,</span> sheetname<span class="token operator">=</span>sheetname<span class="token punctuation">,</span>
nrows<span class="token operator">=</span>nrows<span class="token punctuation">,</span> skiprows<span class="token operator">=</span>skiprows<span class="token punctuation">,</span> header<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">)</span>
skiprows <span class="token operator">+=</span> nrows
<span class="token comment"># When there is no data, we know we can break out of the loop.</span>
<span class="token keyword">if</span> <span class="token keyword">not</span> df_chunk<span class="token punctuation">.</span>shape<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">:</span>
<span class="token keyword">break</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string-interpolation"><span class="token string">f" - chunk </span><span class="token interpolation"><span class="token punctuation">{</span>i_chunk<span class="token punctuation">}</span></span><span class="token string"> (</span><span class="token interpolation"><span class="token punctuation">{</span>df_chunk<span class="token punctuation">.</span>shape<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">}</span></span><span class="token string"> rows)"</span></span><span class="token punctuation">)</span>
chunks<span class="token punctuation">.</span>append<span class="token punctuation">(</span>df_chunk<span class="token punctuation">)</span>
i_chunk <span class="token operator">+=</span> <span class="token number">1</span>
df_chunks <span class="token operator">=</span> pd<span class="token punctuation">.</span>concat<span class="token punctuation">(</span>chunks<span class="token punctuation">)</span>
<span class="token comment"># Rename the columns to concatenate the chunks with the header.</span>
columns <span class="token operator">=</span> <span class="token punctuation">{</span>i<span class="token punctuation">:</span> col <span class="token keyword">for</span> i<span class="token punctuation">,</span> col <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>df_header<span class="token punctuation">.</span>columns<span class="token punctuation">.</span>tolist<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">}</span>
df_chunks<span class="token punctuation">.</span>rename<span class="token punctuation">(</span>columns<span class="token operator">=</span>columns<span class="token punctuation">,</span> inplace<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span>
df <span class="token operator">=</span> pd<span class="token punctuation">.</span>concat<span class="token punctuation">(</span><span class="token punctuation">[</span>df_header<span class="token punctuation">,</span> df_chunks<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> df
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
df <span class="token operator">=</span> make_df_from_excel<span class="token punctuation">(</span><span class="token string">'claims-2002-2006_0.xls'</span><span class="token punctuation">,</span> nrows<span class="token operator">=</span><span class="token number">10000</span><span class="token punctuation">)</span></code></pre>
<p>Another thing to keep in mind. When working with <a href="https://www.python-excel.org/">Excel files in Python</a>, you might need to use different packages whether you need to read/write data from/to <code>.xls</code> and <code>.xlsx</code> files.</p>
<p>This dataset contained both <code>.xls</code> and <code>.xlsx</code> files, so I had to use <a href="https://github.com/python-excel/xlrd">xlrd</a> to read them. Please <a href="https://groups.google.com/forum/#!msg/python-excel/P6TjJgFVjMI/g8d0eWxTBQAJ">be aware</a> that if your only concern is reading <code>.xlsx</code> files, then <a href="https://openpyxl.readthedocs.io/en/stable/">openpyxl</a> is the way to go, even if xlrd <a href="https://stackoverflow.com/questions/35823835/reading-excel-file-is-magnitudes-slower-using-openpyxl-compared-to-xlrd">could still be faster</a>.</p>
<p>This time I didn’t have to write any Excel files, but if you need to, then you want <a href="https://xlsxwriter.readthedocs.io/">xlsxwriter</a>. I remember having used it to create workbooks (i.e. Excel files) with many complex worksheets and cell comments. You can even use it to create worksheets with <a href="https://xlsxwriter.readthedocs.io/working_with_sparklines.html">sparklines</a> and <a href="https://xlsxwriter.readthedocs.io/working_with_macros.html">VBA macros</a>!</p>
https://www.giacomodebidda.com/posts/12-years-of-fires-in-sardinia/12 Years of Fires in Sardinia2018-08-22T19:00:03Z2018-08-22T19:00:03Z
<p>This summer I was looking for some data visualization challenges and I came across <a href="https://mauromelis.gitlab.io/sardinia-on-fire/">this cool project</a> by Mauro Melis. Mauro created it for a contest organized by <a href="https://contest.formez.it/">Open Data Sardegna</a>, and the jury found it so cool that he won the first prize in the data visualization category.</p>
<p>It’s basically a <a href="https://flowingdata.com/tag/scrollytelling/">scrollytelling</a> visualization, a type of visualization popularized - among others - by <a href="https://www.nytimes.com/interactive/2016/12/07/world/asia/rodrigo-duterte-philippines-drugs-killings.html">The New York Times</a> and the guys at <a href="https://pudding.cool/">The Pudding</a>.</p>
<p>There were no links to the data that Mauro used, but it was pretty easy to find the datasets from <a href="http://dati.regione.sardegna.it/dataset/cfva-perimetrazioni-aree-percorse-dal-fuoco-2005">2005</a> to <a href="http://dati.regione.sardegna.it/dataset/cfva-perimetrazioni-aree-percorse-dal-fuoco-2016">2016</a>, namely 12 years of wild fires in Sardinia.</p>
<p>I like scrollytelling, but I wanted to do something quick this time. I also wanted to try an online tool (it’s also a library, but I used the online tool) developed by Uber: <a href="https://kepler.gl/">Kepler.gl</a>.</p>
<h2 id="h-shapefiles%3F-geopandas" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/12-years-of-fires-in-sardinia/#h-shapefiles%3F-geopandas"><span aria-hidden="true">#</span></a> Shapefiles? GeoPandas!</h2>
<p>The datasets from 2005 to 2016 contain <a href="https://en.wikipedia.org/wiki/Shapefile">shapefiles</a>, a popular geospatial vector data format. I know that there are several <a href="https://www.sitepoint.com/javascript-geospatial-advanced-maps/">geospatial libraries in Javascript</a>, and of course <a href="https://medium.com/@mbostock/command-line-cartography-part-1-897aa8f8ca2c">D3 is awesome for creating maps</a>, but I think that Python is so much better at data wrangling than Javascript, so I decided to go with it.</p>
<p>In Python, if you need to work with data, you pick Pandas.</p>
<p>If you need to work with Geospatial data, you pick GeoPandas.</p>
<p>It’s that simple!</p>
<h2 id="h-not-much-data-wrangling" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/12-years-of-fires-in-sardinia/#h-not-much-data-wrangling"><span aria-hidden="true">#</span></a> Not much Data Wrangling</h2>
<p>Turns out that these datasets were actually pretty good, so I didn’t have to do too much data wrangling. Of course there were differences from year to year, but nothing major. As an example, this is what I did to clean the 2016 dataset:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> os
<span class="token keyword">import</span> geopandas <span class="token keyword">as</span> gpd
gdf2016 <span class="token operator">=</span> gpd<span class="token punctuation">.</span>read_file<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>data_dir<span class="token punctuation">,</span> <span class="token string">'areeIncendiatePerim2016'</span><span class="token punctuation">,</span> <span class="token string">'Perimetri_Superfici_Bruciate_2016.shp'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
gdf2016 <span class="token operator">=</span> gdf2016\
<span class="token punctuation">.</span>reset_index<span class="token punctuation">(</span>drop<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span>\
<span class="token punctuation">.</span>drop<span class="token punctuation">(</span>columns<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">'BASE_FID'</span><span class="token punctuation">,</span> <span class="token string">'ID_INCE'</span><span class="token punctuation">,</span> <span class="token string">'ISTAT'</span><span class="token punctuation">,</span> <span class="token string">'ID_PROV'</span><span class="token punctuation">,</span> <span class="token string">'STIR'</span><span class="token punctuation">,</span> <span class="token string">'STAZIONE'</span><span class="token punctuation">,</span>
<span class="token string">'COMUNE'</span><span class="token punctuation">,</span> <span class="token string">'TIPOLOGIE'</span><span class="token punctuation">,</span> <span class="token string">'M2_BOSCO'</span><span class="token punctuation">,</span> <span class="token string">'M2_PASCOLO'</span><span class="token punctuation">,</span> <span class="token string">'M2_ALTRO'</span><span class="token punctuation">,</span> <span class="token string">'SUP_TOT_M2'</span><span class="token punctuation">,</span>
<span class="token string">'TIPO_INCE'</span><span class="token punctuation">,</span> <span class="token string">'dist_ins'</span><span class="token punctuation">,</span> <span class="token string">'ID_RILIEVO'</span><span class="token punctuation">,</span> <span class="token string">'MODIFICHE'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>\
<span class="token punctuation">.</span>rename<span class="token punctuation">(</span>columns<span class="token operator">=</span><span class="token punctuation">{</span><span class="token string">'TOPONIMO'</span><span class="token punctuation">:</span> <span class="token string">'toponym'</span><span class="token punctuation">,</span> <span class="token string">'DATA_INCE'</span><span class="token punctuation">:</span> <span class="token string">'date'</span><span class="token punctuation">,</span> <span class="token string">'N_INCE'</span><span class="token punctuation">:</span> <span class="token string">'num_fires'</span><span class="token punctuation">,</span> <span class="token string">'SUP_TOT_HA'</span><span class="token punctuation">:</span> <span class="token string">'hectars'</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
cols <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'toponym'</span><span class="token punctuation">,</span> <span class="token string">'hectars'</span><span class="token punctuation">,</span> <span class="token string">'date'</span><span class="token punctuation">,</span> <span class="token string">'num_fires'</span><span class="token punctuation">,</span> <span class="token string">'geometry'</span><span class="token punctuation">]</span>
gdf2016 <span class="token operator">=</span> gdf2016<span class="token punctuation">[</span>cols<span class="token punctuation">]</span></code></pre>
<p>Basically I harmonized the datasets from 2005 to 2016, so they had the same structure.</p>
<pre class="language-python"><code class="language-python">gdf2016<span class="token punctuation">.</span>head<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599200654/gdf-head_gqjp24.png">https://res.cloudinary.com/jackdbd/image/upload/v1599200654/gdf-head_gqjp24.png</a></p>
<p>Then I concatenated everything:</p>
<pre class="language-python"><code class="language-python">gdf <span class="token operator">=</span> pd<span class="token punctuation">.</span>concat<span class="token punctuation">(</span><span class="token punctuation">[</span>gdf2005<span class="token punctuation">,</span> gdf2006<span class="token punctuation">,</span> gdf2007<span class="token punctuation">,</span>
gdf2008<span class="token punctuation">,</span> gdf2009<span class="token punctuation">,</span> gdf2010<span class="token punctuation">,</span>
gdf2011<span class="token punctuation">,</span> gdf2012<span class="token punctuation">,</span> gdf2013<span class="token punctuation">,</span>
gdf2014<span class="token punctuation">,</span> gdf2015<span class="token punctuation">,</span> gdf2016<span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<p>I experimented a little bit with <a href="https://geoviews.org/">GeoViews</a> in a Jupyter notebook…</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599200647/experiments_wv8vp5.png">https://res.cloudinary.com/jackdbd/image/upload/v1599200647/experiments_wv8vp5.png</a></p>
<h2 id="h-simplify-after-all-less-is-more" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/12-years-of-fires-in-sardinia/#h-simplify-after-all-less-is-more"><span aria-hidden="true">#</span></a> Simplify (after all, <a href="https://en.wikipedia.org/wiki/Ludwig_Mies_van_der_Rohe">Less is More</a>)</h2>
<p>I exported everything with:</p>
<pre class="language-python"><code class="language-python">df<span class="token punctuation">.</span>to_csv<span class="token punctuation">(</span><span class="token string">'sardinia_fires.csv'</span><span class="token punctuation">)</span></code></pre>
<p>While this worked, it resulted in a ~150MB CSV file. This is because I was including in the output the geometries of all the polygons. I uploaded the CSV file to <a href="http://kepler.gl/">Kepler.gl</a> and it actually worked (well, I wasn’t exactly surprised given that Kepler was developed to visualize Uber’s data), but it took some time (I think ~10 minutes to upload).</p>
<p>I explored the data in <a href="https://kepler.gl/demo">Kepler.gl</a> for a couple of minutes. The vast majority of the wild fires in these datasets were quite small, so they looked like points. I decided to get rid of the <code>geometry</code> column in the <code>GeoDataFrame</code> and to export only the coordinates of the centroid of each polygon. This was super easy to do with GeoPandas:</p>
<pre class="language-python"><code class="language-python">gdf<span class="token punctuation">[</span><span class="token string">'CentroidLongitude'</span><span class="token punctuation">]</span> <span class="token operator">=</span> gdf<span class="token punctuation">[</span><span class="token string">'geometry'</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token builtin">apply</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> poly<span class="token punctuation">:</span> poly<span class="token punctuation">.</span>centroid<span class="token punctuation">.</span>bounds<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
gdf<span class="token punctuation">[</span><span class="token string">'CentroidLatitude'</span><span class="token punctuation">]</span> <span class="token operator">=</span> gdf<span class="token punctuation">[</span><span class="token string">'geometry'</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token builtin">apply</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> poly<span class="token punctuation">:</span> poly<span class="token punctuation">.</span>centroid<span class="token punctuation">.</span>bounds<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<p>The resulting CSV file was obviously much smaller.</p>
<p>I decided to use the following <a href="https://www.qlik.com/blog/visual-encoding">visual encoding</a> for the points (well, circles):</p>
<ul>
<li>color correlates with number of fires (yellow = less fires; purple = more fires)</li>
<li>size correlates with area (in hectares).</li>
</ul>
<h2 id="h-success" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/12-years-of-fires-in-sardinia/#h-success"><span aria-hidden="true">#</span></a> Success!</h2>
<p>I took a screenshot:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599200654/sardinia-fires_yakjue.png">https://res.cloudinary.com/jackdbd/image/upload/v1599200654/sardinia-fires_yakjue.png</a></p>
<p>and an animated GIF (I recorded it with <a href="https://github.com/phw/peek">Peek</a>, a really amazing tool):</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599200654/sardinia-fires_adzh9o.gif">https://res.cloudinary.com/jackdbd/image/upload/v1599200654/sardinia-fires_adzh9o.gif</a></p>
<p>I posted it on the <a href="https://www.reddit.com/r/dataisbeautiful/comments/8z1i0p/12_years_of_fires_in_sardinia_20052016_oc/">DataIsBeautiful subreddit</a> and it was quite succesfull.</p>
<p>Someone commented that I should have added a legend, and I agree, but apparently I was too lazy to find out how to add it in <a href="http://kepler.gl/">Kepler.gl</a>.</p>
<p>Other projects where I had to do much more data wrangling had been completely ignored.</p>
<p>Lessons learned:</p>
<ul>
<li>Nobody cares about how much you struggled with data wrangling (but you still have to do it).</li>
<li>Always include a GIF in a README (well, I already knew that…)</li>
<li>Sardinian cities keep their Italian name in English</li>
</ul>
<h2 id="h-code" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/12-years-of-fires-in-sardinia/#h-code"><span aria-hidden="true">#</span></a> Code</h2>
<p>You can find the <a href="https://github.com/jackdbd/sardinia-fires">repository on GitHub</a>.</p>
<h2 id="h-a-note-on-reproducibilty" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/12-years-of-fires-in-sardinia/#h-a-note-on-reproducibilty"><span aria-hidden="true">#</span></a> A Note on Reproducibilty</h2>
<p>I recently tried to reproduce the notebook and I had to exclude the dataset from 2010. I think this is due to some dependency issues with <a href="https://macwright.com/2012/10/31/gis-with-python-shapely-fiona.html">fiona</a>, which is used by GeoPandas.</p>
https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/Export a GeoDataFrame to Spatialite2018-08-21T21:00:03Z2018-08-21T21:00:03Z
<p>I have been doing some geospatial analysis in the last few months, and since I came back from <a href="https://2018.geopython.net/">GeoPython</a> in Basel (really good conference by the way, definitely recommended) I kept playing around with several Python libraries.</p>
<p><a href="https://twitter.com/jackdbd/status/994548711664570369">https://twitter.com/jackdbd/status/994548711664570369</a></p>
<p>One of my favourite talks at the conference was by <a href="https://all-geo.org/volcan01010/">John A Stevenson</a>, a Scottish volcanologist.</p>
<p>Apart from the scans of his amazing field notebooks, beautifully filled with annotations and drawings, I was impressed by his technical expertise and passion for his work.</p>
<p>John mentioned that a common issue for researches that have to showcase their work at conferences is to deal with shapefiles. Instead of having to juggle with many different folders, files, and versions, he suggested to use a portable geospatial database. He mentioned two software solutions: <a href="https://cholmes.wordpress.com/2013/08/20/spatialite-and-geopackage/">Geopackage</a> and <a href="https://en.wikipedia.org/wiki/SpatiaLite">Spatialite</a>.</p>
<p>Given that I had already <a href="https://www.bostongis.com/PrinterFriendly.aspx?content_name=spatialite_tut01">heard about Spatialite</a>, I decided to try it.</p>
<p>In a <a href="https://github.com/jackdbd/aree-protette">toy project of mine</a> I used some shapefiles available on the <a href="https://dati.toscana.it/dataset">Open Data portal of Tuscany</a>. I loaded the shapefiles with <a href="https://geopandas.org/en/stable/">GeoPandas</a>, performed some really simple geospatial analysis, and created a few maps with <a href="https://geoviews.org/">GeoViews</a> and <a href="https://scitools.org.uk/cartopy/docs/latest/index.html">Cartopy</a>. I decided to use Spatialite to export the geometries created by GeoPandas to a SQLite database.</p>
<p>The <code>GeoPandas</code>’s <code>GeoDataFrame</code> class inherits from <code>Pandas</code>’s <code>DataFrame</code>, so it has a <code>to_sql()</code> method. I thought: <em>“I just have to call that method and pass the connection URI to my SQLite database, easy peasy!”</em>.</p>
<p>Well, <a href="https://github.com/geopandas/geopandas/issues/595">turns out</a> it wasn’t actually that easy… so I am writing this post to remember what I did.</p>
<h2 id="h-install-spatialite" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/#h-install-spatialite"><span aria-hidden="true">#</span></a> Install Spatialite</h2>
<p>Spatialite is actually a geospatial <em>extension</em> of SQLite, namely a set of functions that allow to store geospatial data in a SQLite database.</p>
<p>If you are on a Ubuntu-based distro, you can install Spatialite with:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">sudo</span> <span class="token function">apt-get</span> update
<span class="token function">sudo</span> <span class="token function">apt-get</span> <span class="token function">install</span> spatialite-bin</code></pre>
<h2 id="h-check-your-geometries" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/#h-check-your-geometries"><span aria-hidden="true">#</span></a> Check your geometries</h2>
<p>Check the geometry type of each row in the <code>GeoDataFrame</code>. In my case, all geometries were of type <code>Polygon</code>. I think that if you have mixed geometry types (e.g. <code>Polygon</code> and <code>MultiPolygon</code>) you have to use shapely methods (via GeoPandas) to convert them. Or maybe use different columns for different geometries that you want to store in the database.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># gdf is a GeoPandas DataFrame</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>gdf<span class="token punctuation">.</span>geometry<span class="token punctuation">.</span><span class="token builtin">type</span><span class="token punctuation">)</span>
<span class="token operator">>></span><span class="token operator">></span> Polygon</code></pre>
<h2 id="h-create-a-database-table-with-no-geospatial-datatypes" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/#h-create-a-database-table-with-no-geospatial-datatypes"><span aria-hidden="true">#</span></a> Create a database table with no geospatial datatypes</h2>
<p>Even if you cannot use <code>gdf.to_sql()</code> if you have some geospatial data, it’s fine if your <code>GeoDataFrame</code> <strong>doesn’t actually contain geospatial data</strong> (so it’s basically just like any other Pandas <code>DataFrame</code>).</p>
<p>Connect to your SQLite database and create a new table.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> os
<span class="token keyword">import</span> sqlite3
DB_PATH <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>os<span class="token punctuation">.</span>getcwd<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">'your-database.db'</span><span class="token punctuation">)</span>
<span class="token comment"># Drop all geospatial data</span>
df <span class="token operator">=</span> gdf<span class="token punctuation">.</span>drop<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'geometry'</span><span class="token punctuation">,</span> <span class="token string">'AREA'</span><span class="token punctuation">,</span> <span class="token string">'PERIMETER'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span>
<span class="token comment"># Create the table and populate it with non-geospatial datatypes</span>
<span class="token keyword">with</span> sqlite3<span class="token punctuation">.</span>connect<span class="token punctuation">(</span>DB_PATH<span class="token punctuation">)</span> <span class="token keyword">as</span> conn<span class="token punctuation">:</span>
df<span class="token punctuation">.</span>to_sql<span class="token punctuation">(</span><span class="token string">'your_table_name'</span><span class="token punctuation">,</span> conn<span class="token punctuation">,</span> if_exists<span class="token operator">=</span><span class="token string">'replace'</span><span class="token punctuation">,</span> index<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span></code></pre>
<p>Note: This is like any other “standard” export from a pandas DataFrame into a SQLite database table. There is no Spatialite functionality involved here. Not yet.</p>
<h2 id="h-add-a-new-column-to-store-the-geometry" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/#h-add-a-new-column-to-store-the-geometry"><span aria-hidden="true">#</span></a> Add a new column to store the geometry</h2>
<p>Now that you have your table, you can use the Spatialite extension to add a new table column to store your geometry (i.e. your geospatial data.</p>
<p>You might wonder: <em>why do I have to add this column now? Couldn’t I have added the column when I created the table?</em></p>
<p>According to the <a href="https://www.gaia-gis.it/gaia-sins/spatialite-cookbook/html/new-geom.html">Spatialite documentation</a>, you always must <em>first create the table</em>, then add the Geometry-column in a <em>second time and as a separate step</em>.</p>
<p>Another thing you have to is to load the Spatialite extension and to initialize your spatial metadata.</p>
<p>Let’s say that your geometry is a <code>Polygon</code>, you want to use <a href="https://epsg.io/3857">EPSG:3857</a> as a projected coordinate system, and you want to name your column <code>wkb_geometry</code> (see next step why this name). This is what you have to do:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">with</span> sqlite3<span class="token punctuation">.</span>connect<span class="token punctuation">(</span>DB_PATH<span class="token punctuation">)</span> <span class="token keyword">as</span> conn<span class="token punctuation">:</span>
conn<span class="token punctuation">.</span>enable_load_extension<span class="token punctuation">(</span><span class="token boolean">True</span><span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>load_extension<span class="token punctuation">(</span><span class="token string">"mod_spatialite"</span><span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span><span class="token string">"SELECT InitSpatialMetaData(1);"</span><span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>
<span class="token triple-quoted-string string">"""
SELECT AddGeometryColumn('your_table_name', 'wkb_geometry', 3857, 'POLYGON', 2);
"""</span>
<span class="token punctuation">)</span></code></pre>
<p>So, to recap:</p>
<ul>
<li><code>InitSpatialMetaData()</code> and <code>AddGeometryColumn()</code> are functions from Spatialite, so you have to load it as a SQLite extension.</li>
<li><code>InitSpatialMetaData()</code> must be called before attempting to call any other Spatial SQL function.</li>
<li>You just need to call <code>InitSpatialMetaData()</code> once; calling it multiple times is useless but completely harmless.</li>
<li>First create the table, then add the Geometry-column as a separate step.</li>
</ul>
<h2 id="h-convert-each-shapely-geometry-into-a-wkb-representation" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/#h-convert-each-shapely-geometry-into-a-wkb-representation"><span aria-hidden="true">#</span></a> Convert each shapely geometry into a WKB representation</h2>
<p>SQLite 3 supports only a few <a href="https://www.sqlite.org/datatype3.html">storage classes (i.e. datatypes)</a>. You need to store your geospatial data as <a href="https://en.wikipedia.org/wiki/Binary_large_object">BLOB</a>.</p>
<p>GeoPandas stores geospatial data as shapely geometries, so you have to convert them somehow. Thanks to <a href="https://gis.stackexchange.com/a/141854/119309">this answer on GIS Stack Exchange</a> I found that the <code>shapely.wkb</code> module provides <code>dumps()</code> and <code>loads()</code> functions that work almost exactly as their <code>pickle</code> and <code>simplejson</code> module counterparts. See <a href="https://shapely.readthedocs.io/en/stable/manual.html#well-known-formats">here</a> for details.</p>
<p>Each geometry in a GeoPandas <code>GeoDataFrame</code> is a <code>GeoSeries</code>. A <code>GeoSeries</code> is basically a shapely geometry with some additional properties. This means that you can convert a geometry into a binary string with something like this:</p>
<pre class="language-python"><code class="language-python">wkb <span class="token operator">=</span> swkb<span class="token punctuation">.</span>dumps<span class="token punctuation">(</span>gdf<span class="token punctuation">.</span>geometry<span class="token punctuation">.</span>iloc<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span></code></pre>
<p>Since the database column is already there (until now it’s filled with <code>NULL</code>), you have to use a SQL <code>UPDATE SET</code> statement.</p>
<p>I wanted to use <code>executemany</code> to perform a batch update, but in order to do that I also needed a <code>WHERE</code> clause and an identifier to understand which table cell to update.</p>
<p>When you use <code>executemany</code> you have to pass a tuple of tuples as a query parameter, so I prepared my data like this:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> shapely<span class="token punctuation">.</span>wkb <span class="token keyword">as</span> swkb
records <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'some_id'</span><span class="token punctuation">:</span> gdf<span class="token punctuation">.</span>some_id<span class="token punctuation">.</span>iloc<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">'wkb'</span><span class="token punctuation">:</span> swkb<span class="token punctuation">.</span>dumps<span class="token punctuation">(</span>gdf<span class="token punctuation">.</span>geometry<span class="token punctuation">.</span>iloc<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">}</span>
<span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>gdf<span class="token punctuation">.</span>shape<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token punctuation">]</span></code></pre>
<p>As you can see, each shapely geometry has been converted into its own Well Known Binary representation.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span><span class="token builtin">type</span><span class="token punctuation">(</span>gdf<span class="token punctuation">.</span>geometry<span class="token punctuation">.</span>iloc<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token operator">>></span><span class="token operator">></span> <span class="token operator"><</span><span class="token keyword">class</span> <span class="token string">'shapely.geometry.polygon.Polygon'</span><span class="token operator">></span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token builtin">type</span><span class="token punctuation">(</span>records<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'wkb'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token operator">>></span><span class="token operator">></span> <span class="token operator"><</span><span class="token keyword">class</span> <span class="token string">'bytes'</span><span class="token operator">></span></code></pre>
<h2 id="h-populate-the-column-with-binary-data" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/#h-populate-the-column-with-binary-data"><span aria-hidden="true">#</span></a> Populate the column with binary data</h2>
<p>One last step before populating the table is to create the tuple of tuples to use as a query parameter (because I’m doing a batch update with <code>executemany</code>).</p>
<pre class="language-python"><code class="language-python">tuples <span class="token operator">=</span> <span class="token builtin">tuple</span><span class="token punctuation">(</span><span class="token punctuation">(</span>d<span class="token punctuation">[</span><span class="token string">'wkb'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> d<span class="token punctuation">[</span><span class="token string">'some_id'</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token keyword">for</span> d <span class="token keyword">in</span> records<span class="token punctuation">)</span></code></pre>
<p>Finally, the batch update query:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">with</span> sqlite3<span class="token punctuation">.</span>connect<span class="token punctuation">(</span>DB_PATH<span class="token punctuation">)</span> <span class="token keyword">as</span> conn<span class="token punctuation">:</span>
conn<span class="token punctuation">.</span>enable_load_extension<span class="token punctuation">(</span><span class="token boolean">True</span><span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>load_extension<span class="token punctuation">(</span><span class="token string">"mod_spatialite"</span><span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>executemany<span class="token punctuation">(</span>
<span class="token triple-quoted-string string">"""
UPDATE your_table_name
SET wkb_geometry=GeomFromWKB(?, 3857)
WHERE your_table_name.some_id = ?
"""</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>tuples<span class="token punctuation">)</span>
<span class="token punctuation">)</span></code></pre>
<h2 id="h-double-check-that-it-worked" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/#h-double-check-that-it-worked"><span aria-hidden="true">#</span></a> Double check that it worked</h2>
<p>There are several ways to double check that the database contains the right geospatial data.</p>
<p>You can perform a simple query:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">with</span> sqlite3<span class="token punctuation">.</span>connect<span class="token punctuation">(</span>DB_PATH<span class="token punctuation">)</span> <span class="token keyword">as</span> conn<span class="token punctuation">:</span>
conn<span class="token punctuation">.</span>enable_load_extension<span class="token punctuation">(</span><span class="token boolean">True</span><span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>load_extension<span class="token punctuation">(</span><span class="token string">"mod_spatialite"</span><span class="token punctuation">)</span>
cur <span class="token operator">=</span> conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>
<span class="token triple-quoted-string string">"""
SELECT wkb_geometry FROM your_table_name
"""</span>
<span class="token punctuation">)</span>
results <span class="token operator">=</span> cur<span class="token punctuation">.</span>fetchall<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>results<span class="token punctuation">)</span></code></pre>
<p>or you can use a Spatialite viewer like <a href="https://www.gaia-gis.it/fossil/spatialite_gui/index">Spatialite GUI</a> and use its <em>Map Preview</em> feature.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599302022/spatialite-gui_xgzx3z.png">https://res.cloudinary.com/jackdbd/image/upload/v1599302022/spatialite-gui_xgzx3z.png</a></p>
<h2 id="h-reference" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/#h-reference"><span aria-hidden="true">#</span></a> Reference</h2>
<p><a href="https://github.com/jackdbd/aree-protette/blob/master/aree-protette.ipynb">Jupyter notebook</a> where I used Spatialite.</p>
<h2 id="h-extra" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/export-a-geodataframe-to-spatialite/#h-extra"><span aria-hidden="true">#</span></a> Extra</h2>
<p><a href="https://twitter.com/jorisvdbossche">Joris Van den Bossche</a> and <a href="https://twitter.com/levijohnwolf">Levi John Wolf</a> also gave two excellent talks about GeoPandas, PySal, and geospatial analysis. Be sure to check out their tutorials <a href="https://github.com/jorisvandenbossche/geopandas-tutorial">here</a> and <a href="https://github.com/ljwolf/geopython">here</a>. The best thing is that you can run their notebooks with <a href="https://mybinder.org/">binder</a> without having to install anything on your machine!</p>
https://www.giacomodebidda.com/posts/jupyter-notebook-on-hdf5-h5py-pytables-datashader/Jupyter notebook on HDF5, h5py, PyTables, Datashader2018-01-06T21:00:03Z2018-01-06T21:00:03Z
<p>Last month at a <a href="https://www.meetup.com/it-IT/PyData-Munchen/">PyDataMunich</a> meetup I gave a short workshop on HDF5 and the python packages that we can use when dealing with HDF5 files.</p>
<p>I started the workshop with this picture:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303197/h5py-pytables-refactor_kziyfd.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303197/h5py-pytables-refactor_kziyfd.png</a></p>
<p>As you can see, at the moment python developers have to make a choice between h5py and PyTables.</p>
<p>h5py provides a low-level, pythonic interface to the entire C HDF5 library. PyTables takes a different route and provides high-level abstractions that can make your life easier when writing to HDF5 or searching data in a HDF5 file. It also implements compression and chunking for you. You could say PyTables is more “battery included”.</p>
<p>I have more experience with PyTables, so I am a little bit biased towards it.</p>
<p>What I really like about these projects is that they have very different philosophies, and for this reason the mainteners agreed to work together in creating a new python stack for HDF5.</p>
<p>It took me quite a while to prepare this workshop, but it was a great opportunity to play with the <a href="https://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml">New York City taxi dataset</a> and try out <a href="https://github.com/bokeh/datashader">Datashader</a>, an innovative data rasterization pipeline that can be used to create data visualizations from massive datasets.</p>
<p>Here is an image I generated from the yellow taxis data from January 2015. Pickup locations are shown in blue. The pickup locations resulted from a search are shown in yellow.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303197/nyc-yellow-taxis-pickups_nipps2.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303197/nyc-yellow-taxis-pickups_nipps2.png</a></p>
<p>Snippets and notebook are available on <a href="https://github.com/jackdbd/hdf5-pydata-munich">this GitHub repo</a>.</p>
https://www.giacomodebidda.com/posts/interesting-things-other-people-did-in-2017/Interesting things other people did in 20172017-12-31T16:00:03Z2017-12-31T16:00:03Z
<p>I like to compile a list of things I stumbled upon during the year and that I found interesting. It’s an idea I adopted from Jeff Leek’s blog <a href="https://simplystatistics.org/">simply statistics</a>. If you are curious, here is the list from <a href="https://www.giacomodebidda.com/posts/interesting-things-other-people-did-in-2016/">2016</a>.</p>
<p>Here is the list for 2017:</p>
<ul>
<li><a href="http://genius.com/">Genius.com</a> created a cool infographic about <a href="https://genius.com/a/infographic-how-dragon-ball-influenced-a-generation-of-hip-hop-artists">how Dragon Ball influenced a generation of hip hop artists</a>.</li>
<li>Nadieh Bremer created this amazing visualization of <a href="https://dragonballz.visualcinnamon.com/">all fights in Dragon Ball Z</a> for her one year long project <a href="https://www.datasketch.es/">datasketches</a>. The amount of awesome things that both Nadieh and Shirley managed to do this year is just insane. Every time I had a look at their projects I was left wondering: “can I really use D3 to make <em>this</em>?”</li>
<li>James Talmage and Damon Maneice created <a href="https://thetruesize.com/">thetruesize</a>, a web app that lets you select a country and drag and drop it wherever you want on Earth. I really like this app because it makes you realize how distorted the Mercator projection is when comparing the sizes of something really close to the poles (e.g. Greenland) to something on the Equator (e.g. Colombia).</li>
<li>Russell Goldenberg open sourced <a href="https://pudding.cool/process/introducing-scrollama/">Scrollama</a>, a lightweight JavaScript library for scrollytelling. Elliot Bentley at the Wall Street Journal released a similar library called <a href="https://wsj.github.io/two-step/">TwoStep</a>. I have never used any of these scrolling libraries, but I definitely will in one of my next projects.</li>
<li>plotly realeased <a href="https://plot.ly/products/dash/">Dash</a>, a Python library to build reactive and interactive dashboards. I like Dash a lot, and I’ve already written about it <a href="https://www.giacomodebidda.com/posts/visualize-earthquakes-with-plotly-dash/">here</a> and <a href="https://www.giacomodebidda.com/posts/reactive-dashboards-with-plotly-dash/">here</a>. I plan to contribute to this project in 2018.</li>
<li>Ventursky, a Czech meteorological company, created this <a href="https://www.ventusky.com/">astonishing real-time weather map</a>. Man, wind maps are sooo cool, I would watch them for hours…</li>
<li><a href="https://unitethesefuckers.com/">unitethesefuckers</a> is a news aggregator that helps you in getting a balanced perspective of what liberals and conservatives are reading.</li>
<li>Vicki Boykis wrote <a href="https://vickiboykis.com/2017/02/01/what-should-you-think-about-when-using-facebook/">this article on how Facebook is collecting your data</a>. I was already considering quitting Facebook for some time. This piece gave me the final push.</li>
<li>Sasha Trubetskoy made <a href="https://sashamaps.net/docs/maps/roman-roads-index/">this cool subway-style map of Roman roads</a>.</li>
<li>Matt Stiles created the visualization <a href="https://thedailyviz.com/2016/09/17/how-common-is-your-birthday-dailyviz/">how common is your birthday</a>. I didn’t know that September was the busiest month for births. Apparently the 16th of April (my birthday) is not so common.</li>
<li>Matt also published <a href="https://thedailyviz.com/2017/12/22/visualizing-more-than-a-decade-of-north-korean-defections/">Visualizing more than a decade of North Korean defections</a>. I was really surprised to discover that North Korean women are three times more likely to defect than men.</li>
<li>Ben Eater wrote a comprehensive guide on <a href="https://eater.net/?utm_source=hackernewsletter&utm_medium=email&utm_term=fav">how to build an 8-bit computer from scratch</a>.</li>
<li>Stefan Zapf and Christopher Kraushaar created a <a href="https://www.oreilly.com/learning/a-new-visualization-to-beautifully-explore-correlations?imm_mid=0ed213&cmp=em-data-na-na-newsltr_20170208">solar correlation map</a>. I really like this new way of representing correlation, and I started making my own version with D3 and Vue.js (they used matplotlib). I will post it here on the blog when it’s ready.</li>
<li>Eric Roston and Blacki Migliozzi wrote the series of articles <a href="https://www.bloomberg.com/graphics/2017-arctic/">How a Melting Arctic Changes Everything</a></li>
<li>Going from North Pole to South Pole, the New York Times published a <a href="https://www.nytimes.com/interactive/2017/05/18/climate/antarctica-ice-melt-climate-change.html?_r=0">series of articles about Antarctica</a>. It’s a visual masterpiece.</li>
<li>Ideo created this nice <a href="https://fontmap.ideo.com/">Font map</a>.</li>
<li>Russell Goldenberg and Dan Kopf created a cool dataviz about <a href="https://pudding.cool/2017/01/making-it-big/">the unlikely odds of making it big</a>.</li>
<li>The Pudding kept publishing a lot of really good stuff. I particularly liked this piece by Colin Morris: <a href="https://pudding.cool/2017/05/song-repetition/">are Pop Lyrics Getting More Repetitive?</a></li>
<li>One of the coolest things of 2017 were these <a href="https://yourlogicalfallacyis.com/">critical thinking cards that help you avoiding logical fallacies</a>.</li>
<li>Lazaro Gamio created a funny visualization based on a modified version of Chernoff Faces, a technique that maps multiple statistical values to the features of a face. He called his project <a href="https://www.axios.com/an-emoji-built-from-data-for-every-state-2408885674.html">Emoji state of America</a>.</li>
<li>I should follow Andy Kirk from Visualizing Data more consistently. In 2017 he published quite a few interesting articles about data visualizations. <a href="https://www.visualisingdata.com/2017/07/10-significant-visualisation-developments-january-june-2017/">Here is a good one</a>.</li>
<li>Uber released a visualization platform for the web. I already knew about <a href="https://deck.gl/">deck.gl</a>, but it seems that they are releasing a complete framework to allow visual explorations of massive datasets. Xiaoji Chen talks about it <a href="https://eng.uber.com/atg-dataviz/">here</a>.</li>
<li>Adam Pearce at the New York Times wrote a nice article and dataviz when <a href="https://www.nytimes.com/interactive/2017/05/25/sports/basketball/lebron-career-playoff-points-record.html">LeBron James overtook Michael Jordan as the top scorer in the entire NBA playoff history</a>. The level of smoothness of the transitions and animations in the chart is just incredible.</li>
<li><a href="https://www.yegor256.com/best.html">yegor256’s blog</a> has been a great discovery in 2017. His blog posts and his book <a href="https://www.yegor256.com/elegant-objects.html">Elegant Objects</a> literally changed the way I think about object oriented code.</li>
<li>The Center for Spatial Research at Columbia University, and the Masters in Peace Building at the Universidad de Los Andes collaborated in creating <a href="https://centerforspatialresearch.github.io/colombia_site/">Conflict Urbanism: Colombia</a>. The project maps and visualizes crimes committed over a period of 30 years, with the hope of understanding the patterns and ramifications of the Colombian conflict.</li>
<li>Marco Hernandez and Darren Long created a dataviz on <a href="https://multimedia.scmp.com/culture/article/passportIndex/">the power of your passport</a>. South China Morning Post kept publishing really nice visualizations all year around. Here is <a href="https://multimedia.scmp.com/news/china/article/One-Belt-One-Road/europe.html">another one</a>.</li>
<li><a href="https://imgur.com/gallery/CG9w4?utm_source=hackernewsletter&utm_medium=email&utm_term=fav">This guy made a camera that prints a GIF instantly</a>. I know, it’s not related to data visualizations, programming topics or python/js libraries, but it was just too cool not to share it.</li>
</ul>
https://www.giacomodebidda.com/posts/reactive-dashboards-with-plotly-dash/Reactive Dashboards with Plotly Dash2017-12-10T23:00:03Z2017-12-10T23:00:03Z
<p>Last month I gave a talk at <a href="https://www.meetup.com/it-IT/PyMunich/">PyMunich</a> on how to create reactive dashboards with Plotly <a href="https://plot.ly/products/dash/">Dash</a>.</p>
<p>I have already written about Dash <a href="https://www.giacomodebidda.com/posts/visualize-earthquakes-with-plotly-dash/">here</a>, and the weekend before the presentation I managed to deploy <a href="https://github.com/jackdbd/dash-fda">another dashboard</a> on <a href="https://mighty-garden-67470.herokuapp.com/">Heroku</a>.</p>
<p>It seems that the audience liked the topic, and on Twitter <a href="https://github.com/chriddyp">Chris Parmer</a> himself wrote some kind words of appreciation. I wasn’t expecting that. It was pretty cool!</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303202/tweet-chrisp_xunbgu.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303202/tweet-chrisp_xunbgu.png</a></p>
<p>Dash is a pretty new framework in the Python ecosystem, but I think it will manage to get its own space among tools like Jupyter noteboooks and Bokeh interactive plots.</p>
<p><a href="https://github.com/ucg8j">Luke Singham</a> started <a href="https://github.com/Acrotrend/awesome-dash">awesome-dash</a>, a curated list of awesome Dash resources. It contains links to tutorials, code snippets, example dashboards that will definitely be useful if you are just starting with Dash or want to learn a new thing or two. Don’t forget to start it on GitHub!</p>
<p>Here are the slides of my talk:</p>
<iframe src="https://slides.com/jackdbd/deck/embed" width="576" height="420" scrolling="no" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
https://www.giacomodebidda.com/posts/how-to-get-started-with-regl-and-webpack/How to get started with regl and Webpack2017-10-31T19:30:03Z2017-10-31T19:30:03Z
<p>I have been wanting to play around with the <a href="https://github.com/regl-project/regl">regl</a> library since I wathed the <a href="https://www.youtube.com/watch?v=rFjszW5L2aw&t=1196s">talk that Mikola Lysenko gave at PLOTCON 2016</a>, and this week I finally decided to invest some time in setting up a repo with Webpack and some regl examples.</p>
<p>regl is a pretty new library, but it seems quite popular among data visualization practitioners. <a href="https://vallandingham.me/regl_intro.html">Jim Vallandingham</a> and <a href="https://peterbeshai.com/beautifully-animate-points-with-webgl-and-regl.html">Peter Beshai</a> wrote really nice tutorials about regl, and Nadieh Bremer created the stunning visualization <a href="https://bl.ocks.org/nbremer/1acc6c95e3bb374dc78329e94f85a9b0">“A breathing Earth”</a>.</p>
<h2 id="h-regl-and-webgl" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-get-started-with-regl-and-webpack/#h-regl-and-webgl"><span aria-hidden="true">#</span></a> regl and WebGL</h2>
<p>Before starting to learn about regl, you need some basic knowledge about WebGL, the low level API to draw 3D graphics in the browser, and its <a href="https://tsherif.github.io/webgl-presentation/#/">graphics pipeline (aka rendering pipeline)</a>. <a href="https://webglfundamentals.org/webgl/lessons/webgl-how-it-works.html">This article on WebGL Fundamentals</a> does a great job in explaining what WebGL is:</p>
<blockquote>
<p>WebGL is just a rasterization engine. It draws points, lines, and triangles based on code you supply.</p>
</blockquote>
<p>WebGL runs on the GPU on your computer, and you need to provide the code in the form of two functions: a <strong>vertex shader</strong> and a <strong>fragment shader</strong>.</p>
<p>Another important thing to know is that WebGL is a <a href="https://www.packtpub.com/mapt/book/game_development/9781849699792/3/ch03lvl1sec24/understanding-webgl--a-state-machine">state machine</a>: once you modify an attributes, that modification is permanent until you modify that attribute again.</p>
<p>regl is <strong>functional abstration</strong> of WebGL. It simplifies WebGL programming by removing as much shared state as it can get away with.</p>
<p>In regl there are two fundamentals abstractions: <strong>resources</strong> and <strong>commands</strong>.</p>
<ul>
<li>A resource is a handle to something that you load on the GPU, like a texture.</li>
<li>A command is a complete representation of the WebGL state required to perform some draw call. It wraps it up and packages it into a single reusable function.</li>
</ul>
<p>In this article I will only talk about <a href="https://regl.party/api#commands">commands</a>.</p>
<h2 id="h-project-structure-and-boilerplate" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-get-started-with-regl-and-webpack/#h-project-structure-and-boilerplate"><span aria-hidden="true">#</span></a> Project structure and boilerplate</h2>
<p>I found out that if I want to learn a new technology/library/tool I have to play around with it, so I created a repo and I called it <a href="https://github.com/jackdbd/regl-playground">regl-playground</a>.</p>
<p>Let’s start defining the structure for this repo. Here is the root directory:</p>
<pre class="language-shell"><code class="language-shell"><span class="token builtin class-name">.</span>
├── package.json
├── README.md
├── src
└── webpack.config.js</code></pre>
<p>And here is the <code>src</code> directory.</p>
<pre class="language-shell"><code class="language-shell"><span class="token builtin class-name">.</span>
├── css
│ └── main.css
├── js
│ ├── index.js
│ └── one-shot-rendering.js
└── templates
├── index.html
└── one-shot-rendering.html</code></pre>
<p>You will need <code>regl</code> and a few dev dependencies for webpack. If you want to save some time (and keystrokes), you can copy the <code>package.json</code> down below and install all you need with <code>yarn install</code>.</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
<span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"regl-playground"</span><span class="token punctuation">,</span>
<span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"1.0.0"</span><span class="token punctuation">,</span>
<span class="token property">"main"</span><span class="token operator">:</span> <span class="token string">"index.js"</span><span class="token punctuation">,</span>
<span class="token property">"repository"</span><span class="token operator">:</span> <span class="token string">"git@github.com:jackdbd/regl-playground.git"</span><span class="token punctuation">,</span>
<span class="token property">"author"</span><span class="token operator">:</span> <span class="token string">"jackdbd <jackdebidda@gmail.com>"</span><span class="token punctuation">,</span>
<span class="token property">"license"</span><span class="token operator">:</span> <span class="token string">"MIT"</span><span class="token punctuation">,</span>
<span class="token property">"scripts"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"dev"</span><span class="token operator">:</span> <span class="token string">"webpack-dev-server --config webpack.config.js"</span><span class="token punctuation">,</span>
<span class="token property">"lint"</span><span class="token operator">:</span> <span class="token string">"eslint src"</span><span class="token punctuation">,</span>
<span class="token property">"build"</span><span class="token operator">:</span> <span class="token string">"webpack --config webpack.config.js --progress"</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"devDependencies"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"babel-core"</span><span class="token operator">:</span> <span class="token string">"^6.26.0"</span><span class="token punctuation">,</span>
<span class="token property">"babel-loader"</span><span class="token operator">:</span> <span class="token string">"^7.1.2"</span><span class="token punctuation">,</span>
<span class="token property">"babel-preset-es2015"</span><span class="token operator">:</span> <span class="token string">"^6.24.1"</span><span class="token punctuation">,</span>
<span class="token property">"clean-webpack-plugin"</span><span class="token operator">:</span> <span class="token string">"^0.1.17"</span><span class="token punctuation">,</span>
<span class="token property">"eslint"</span><span class="token operator">:</span> <span class="token string">"^4.5.0"</span><span class="token punctuation">,</span>
<span class="token property">"eslint-config-airbnb-base"</span><span class="token operator">:</span> <span class="token string">"^12.1.0"</span><span class="token punctuation">,</span>
<span class="token property">"eslint-plugin-import"</span><span class="token operator">:</span> <span class="token string">"^2.7.0"</span><span class="token punctuation">,</span>
<span class="token property">"extract-text-webpack-plugin"</span><span class="token operator">:</span> <span class="token string">"^3.0.1"</span><span class="token punctuation">,</span>
<span class="token property">"html-webpack-plugin"</span><span class="token operator">:</span> <span class="token string">"^2.30.1"</span><span class="token punctuation">,</span>
<span class="token property">"style-loader"</span><span class="token operator">:</span> <span class="token string">"^0.19.0"</span><span class="token punctuation">,</span>
<span class="token property">"webpack"</span><span class="token operator">:</span> <span class="token string">"^3.8.1"</span><span class="token punctuation">,</span>
<span class="token property">"webpack-bundle-analyzer"</span><span class="token operator">:</span> <span class="token string">"^2.9.0"</span><span class="token punctuation">,</span>
<span class="token property">"webpack-dev-server"</span><span class="token operator">:</span> <span class="token string">"^2.9.3"</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"dependencies"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"regl"</span><span class="token operator">:</span> <span class="token string">"^1.3.0"</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>Next, you will need 2 HTML files, one for the index, one for the actual regl application. I think it’s a good idea to put these files in <code>src/templates</code>.</p>
<pre class="language-html"><code class="language-html"><span class="token comment"><!-- index.html --></span>
<span class="token doctype"><span class="token punctuation"><!</span><span class="token doctype-tag">DOCTYPE</span> <span class="token name">html</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>html</span> <span class="token attr-name">lang</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>en<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>head</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">charset</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>utf-8<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">http-equiv</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>x-ua-compatible<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>ie=edge<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>title</span><span class="token punctuation">></span></span>Home<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>title</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>viewport<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>width=device-width, initial-scale=1.0<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>description<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Home of regl-playground<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token comment"><!-- bundle.css is injected here by html-webpack-plugin --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>head</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>body</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>header</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h1</span><span class="token punctuation">></span></span>List of regl examples<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h1</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>header</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>ul</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>li</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/one-shot-rendering.html<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>one-shot rendering<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>li</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>ul</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>p</span><span class="token punctuation">></span></span>For documentation, see the <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://regl.party/<span class="token punctuation">"</span></span> <span class="token attr-name">target</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>_blank<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>regl API<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span>.<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>p</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>footer</span><span class="token punctuation">></span></span>Examples with regl version <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>code</span><span class="token punctuation">></span></span>1.3.0<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>code</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>footer</span><span class="token punctuation">></span></span>
<span class="token comment"><!-- bundle.js is injected here by html-webpack-plugin --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>body</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>html</span><span class="token punctuation">></span></span></code></pre>
<pre class="language-html"><code class="language-html"><span class="token comment"><!-- one-shot-rendering.html --></span>
<span class="token doctype"><span class="token punctuation"><!</span><span class="token doctype-tag">DOCTYPE</span> <span class="token name">html</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>html</span> <span class="token attr-name">lang</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>en<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>head</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">charset</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>utf-8<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">http-equiv</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>x-ua-compatible<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>ie=edge<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>title</span><span class="token punctuation">></span></span>One shot rendering<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>title</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>viewport<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>width=device-width, initial-scale=1.0<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>description<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>regl example with one shot rendering<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token comment"><!-- bundle.css is injected here by html-webpack-plugin --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>head</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>body</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>ul</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>li</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/index.html<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>Home<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>li</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>ul</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h1</span><span class="token punctuation">></span></span>regl one shot rendering example<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h1</span><span class="token punctuation">></span></span>
<span class="token comment"><!-- bundle.js is injected here by html-webpack-plugin --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>body</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>html</span><span class="token punctuation">></span></span></code></pre>
<p>Then, create a minimal CSS file in <code>src/css</code>.</p>
<pre class="language-css"><code class="language-css"><span class="token comment">/* main.css */</span>
<span class="token selector">h1</span> <span class="token punctuation">{</span>
<span class="token property">color</span><span class="token punctuation">:</span> #0b4192<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>And the Javascript files. We’ll put the code in <code>one-shot-rendering.js</code> later on. For now, just import the CSS, so you can check that webpack is setup correctly.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// index.js</span>
<span class="token keyword">import</span> <span class="token string">'../css/main.css'</span></code></pre>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// one-shot-rendering.js</span>
<span class="token keyword">import</span> <span class="token string">'../css/main.css'</span></code></pre>
<p>Finally, the webpack configuration. I like to include <code>BundleAnalyzerPlugin</code> to check the bundle sizes.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// webpack.config.js</span>
<span class="token keyword">const</span> path <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'path'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> webpack <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'webpack'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> HtmlWebpackPlugin <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'html-webpack-plugin'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> CleanWebpackPlugin <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'clean-webpack-plugin'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> ExtractTextPlugin <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'extract-text-webpack-plugin'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> <span class="token punctuation">{</span> BundleAnalyzerPlugin <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'webpack-bundle-analyzer'</span><span class="token punctuation">)</span>
module<span class="token punctuation">.</span>exports <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token literal-property property">context</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">resolve</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">entry</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">home</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'js'</span><span class="token punctuation">,</span> <span class="token string">'index.js'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token string-property property">'one-shot-rendering'</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>
__dirname<span class="token punctuation">,</span>
<span class="token string">'src'</span><span class="token punctuation">,</span>
<span class="token string">'js'</span><span class="token punctuation">,</span>
<span class="token string">'one-shot-rendering.js'</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">output</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">path</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'dist'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">filename</span><span class="token operator">:</span> <span class="token string">'[name].[chunkhash].bundle.js'</span><span class="token punctuation">,</span>
<span class="token literal-property property">sourceMapFilename</span><span class="token operator">:</span> <span class="token string">'[file].map'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">module</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">rules</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token comment">// rule for .js/.jsx files</span>
<span class="token punctuation">{</span>
<span class="token literal-property property">test</span><span class="token operator">:</span> <span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">\.(js|jsx)$</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">include</span><span class="token operator">:</span> <span class="token punctuation">[</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'js'</span><span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token literal-property property">exclude</span><span class="token operator">:</span> <span class="token punctuation">[</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'node_modules'</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token literal-property property">use</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">loader</span><span class="token operator">:</span> <span class="token string">'babel-loader'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token comment">// rule for css files</span>
<span class="token punctuation">{</span>
<span class="token literal-property property">test</span><span class="token operator">:</span> <span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">\.css$</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">include</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'css'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">use</span><span class="token operator">:</span> ExtractTextPlugin<span class="token punctuation">.</span><span class="token function">extract</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">fallback</span><span class="token operator">:</span> <span class="token string">'style-loader'</span><span class="token punctuation">,</span>
<span class="token literal-property property">use</span><span class="token operator">:</span> <span class="token string">'css-loader'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">target</span><span class="token operator">:</span> <span class="token string">'web'</span><span class="token punctuation">,</span>
<span class="token literal-property property">devtool</span><span class="token operator">:</span> <span class="token string">'source-map'</span><span class="token punctuation">,</span>
<span class="token literal-property property">plugins</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token keyword">new</span> <span class="token class-name">BundleAnalyzerPlugin</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token keyword">new</span> <span class="token class-name">CleanWebpackPlugin</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'dist'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
<span class="token literal-property property">root</span><span class="token operator">:</span> __dirname<span class="token punctuation">,</span>
<span class="token literal-property property">exclude</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">'favicon.ico'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token literal-property property">verbose</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token keyword">new</span> <span class="token class-name">HtmlWebpackPlugin</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">template</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'templates'</span><span class="token punctuation">,</span> <span class="token string">'index.html'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">hash</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
<span class="token literal-property property">filename</span><span class="token operator">:</span> <span class="token string">'index.html'</span><span class="token punctuation">,</span>
<span class="token literal-property property">chunks</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">'commons'</span><span class="token punctuation">,</span> <span class="token string">'home'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token keyword">new</span> <span class="token class-name">HtmlWebpackPlugin</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">template</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>
__dirname<span class="token punctuation">,</span>
<span class="token string">'src'</span><span class="token punctuation">,</span>
<span class="token string">'templates'</span><span class="token punctuation">,</span>
<span class="token string">'one-shot-rendering.html'</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">hash</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
<span class="token literal-property property">filename</span><span class="token operator">:</span> <span class="token string">'one-shot-rendering.html'</span><span class="token punctuation">,</span>
<span class="token literal-property property">chunks</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">'commons'</span><span class="token punctuation">,</span> <span class="token string">'one-shot-rendering'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token keyword">new</span> <span class="token class-name">webpack<span class="token punctuation">.</span>optimize<span class="token punctuation">.</span>CommonsChunkPlugin</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">name</span><span class="token operator">:</span> <span class="token string">'commons'</span><span class="token punctuation">,</span>
<span class="token literal-property property">filename</span><span class="token operator">:</span> <span class="token string">'[name].[chunkhash].bundle.js'</span><span class="token punctuation">,</span>
<span class="token literal-property property">chunks</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">'home'</span><span class="token punctuation">,</span> <span class="token string">'one-shot-rendering'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token keyword">new</span> <span class="token class-name">ExtractTextPlugin</span><span class="token punctuation">(</span><span class="token string">'[name].[chunkhash].bundle.css'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token literal-property property">devServer</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">host</span><span class="token operator">:</span> <span class="token string">'localhost'</span><span class="token punctuation">,</span>
<span class="token literal-property property">port</span><span class="token operator">:</span> <span class="token number">8080</span><span class="token punctuation">,</span>
<span class="token literal-property property">contentBase</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'dist'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">inline</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
<span class="token literal-property property">stats</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">colors</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
<span class="token literal-property property">reasons</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
<span class="token literal-property property">chunks</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span>
<span class="token literal-property property">modules</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">performance</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">hints</span><span class="token operator">:</span> <span class="token string">'warning'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span></code></pre>
<p>Ah, don’t forget <code>.babelrc</code>!</p>
<pre><code>{
"presets": ["es2015"]
}
</code></pre>
<p>Check that everything works by running <code>yarn run dev</code> and going to <code>https://localhost:8080/</code>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599302618/webpack-configured_xzitfv.png">https://res.cloudinary.com/jackdbd/image/upload/v1599302618/webpack-configured_xzitfv.png</a></p>
<h2 id="h-your-first-regl-command" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-get-started-with-regl-and-webpack/#h-your-first-regl-command"><span aria-hidden="true">#</span></a> Your first regl command</h2>
<p>As I said, the regl API provides you with two abstractions: resources and commands. A regl command wraps up all of the WebGL state associated with a draw call (either drawArrays or drawElements) and packages it into a single reusable function.</p>
<p>Here is what you can define in a regl command:</p>
<ul>
<li>vertex shader</li>
<li>fragment shader</li>
<li>uniforms</li>
<li>attributes</li>
<li>primitive</li>
<li>count</li>
</ul>
<p>That’s a lot of terminology! What are these things?</p>
<p>I will try to explain these items in a sentence or two, but if you are looking for some more detailed information you should definitely read the article by <a href="https://www.toptal.com/javascript/3d-graphics-a-webgl-tutorial">Adnan Ademovic on the TopTal blog</a>. It’s a great resource and contains a lot of links to master WebGL.</p>
<p>Here is a summary:</p>
<p><strong>Vertex shader</strong>.<br />
A function written in GLSL that runs on your GPU. Its job is to compute vertex positions and set a variable called <code>gl_Position</code>. It gets the data from Javascript and can pass other data to the fragment shader.</p>
<p><strong>Fragment shader</strong>.<br />
A function written in GLSL that runs on your GPU. Its job is to compute the color of each fragment (you can think of a fragment like a pixel, but the whole story is <a href="https://t-machine.org/index.php/2013/10/05/why-is-a-fragment-shader-named-a-fragment-shader/">more complicated</a> than that).</p>
<p><strong>Uniforms</strong>.<br />
Global variables that you set in Javascript and that are broadcasted to both shaders. They cannot be changed (something similar to <code>const</code> in ES6). They stay the same for all vertices during a single draw call.</p>
<p><strong>Attributes</strong>.<br />
Data that points to Vertex Buffer Objects. Attributes are used in the vertex shader.</p>
<p><strong>Primitive</strong>.<br />
The primitive type for the <a href="https://regl.party/api#elements">element</a> buffer. WebGL is mostly used to draw triangles, but you can also draw individual points. In regl the supported primitives are: <code>'points'</code>, <code>'lines'</code>, <code>'line strip'</code>, <code>'line loop'</code>, <code>'triangles'</code>, <code>'triangle strip'</code> and <code>'triangle fan'</code>.</p>
<p><strong>Count</strong>.<br />
The number of vertices to draw (e.g. if you want to draw 2 triangles you need to set <code>count</code> to 6).</p>
<p>Ok, enough talking. Let’s see some regl action! Open <code>one-shot-rendering.js</code> and write this code:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// one-shot-rendering.js</span>
<span class="token keyword">import</span> <span class="token string">'../css/main.css'</span>
<span class="token comment">// Create a full screen canvas element and a WebGLRenderingContext.</span>
<span class="token keyword">const</span> regl <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'regl'</span><span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> drawTriangle <span class="token operator">=</span> <span class="token function">regl</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token comment">// The vertex shader tells the GPU where to draw the vertices.</span>
<span class="token literal-property property">vert</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
precision mediump float;
uniform float scale;
uniform float pointSize;
attribute vec2 position;
attribute vec3 color;
varying vec3 frag_color; // varying to pass to the fragment shader
float z = 0.0;
float w = 1.0;
void main () {
frag_color = color;
gl_PointSize = pointSize;
gl_Position = vec4(position * scale, z, w);
}
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token comment">// The fragment shader tells the GPU what color to draw.</span>
<span class="token literal-property property">frag</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
precision mediump float;
varying vec3 frag_color; // received from the vertex shader
void main () {
gl_FragColor = vec4(sqrt(frag_color), 1.0);
}
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token comment">// Now that the shaders are defined, we pass the vertices to the GPU</span>
<span class="token literal-property property">attributes</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">position</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token punctuation">[</span><span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">0.75</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">1.0</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token literal-property property">color</span><span class="token operator">:</span> regl<span class="token punctuation">.</span><span class="token function">prop</span><span class="token punctuation">(</span><span class="token string">'rgbColors'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">uniforms</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token comment">// get the prop pointSize and pass it to the shaders</span>
<span class="token function-variable function">pointSize</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter">context<span class="token punctuation">,</span> prop</span><span class="token punctuation">)</span> <span class="token operator">=></span> prop<span class="token punctuation">.</span>pointSize<span class="token punctuation">,</span>
<span class="token comment">// we can also access the props with this shorthand syntax</span>
<span class="token literal-property property">scale</span><span class="token operator">:</span> regl<span class="token punctuation">.</span><span class="token function">prop</span><span class="token punctuation">(</span><span class="token string">'scale'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token comment">// specify the primitive type (the default is 'triangle')</span>
<span class="token literal-property property">primitive</span><span class="token operator">:</span> <span class="token string">'points'</span><span class="token punctuation">,</span>
<span class="token comment">// and we tell the GPU how many vertices to draw</span>
<span class="token literal-property property">count</span><span class="token operator">:</span> <span class="token number">3</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">// In one-shot rendering the command is executed once and immediately.</span>
<span class="token function">drawTriangle</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">pointSize</span><span class="token operator">:</span> <span class="token number">10.0</span><span class="token punctuation">,</span>
<span class="token literal-property property">scale</span><span class="token operator">:</span> <span class="token number">0.5</span><span class="token punctuation">,</span>
<span class="token literal-property property">rgbColors</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token punctuation">[</span><span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>Run <code>yarn run dev</code> and go to <code>https://localhost:8080/</code>. You should see something like this:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599302618/one-shot-rendering-points_bmyos0.png">https://res.cloudinary.com/jackdbd/image/upload/v1599302618/one-shot-rendering-points_bmyos0.png</a><br />
“One-shot rendering: three points at localhost:8080”,</p>
<p>Well… not that exciting you might say. All that code for three points on the screen?</p>
<p>Ok, let’s try changing the line <code>primitive: 'points'</code>. You can either remove it or replace <code>'points'</code> with <code>'triangle'</code> (the default primitive). Now you should see this:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599302618/one-shot-rendering-triangle_y8sbuf.png">https://res.cloudinary.com/jackdbd/image/upload/v1599302618/one-shot-rendering-triangle_y8sbuf.png</a></p>
<p>That’s better! Let’s stop for a second and try to explain what’s going on.</p>
<p>The way we drew this triangle is called <a href="https://regl.party/api#one-shot-rendering">one-shot rendering</a>. The command is executed only once and immediately.</p>
<p>We called the command with some <strong>props</strong>, which will be available in uniforms and attributes.</p>
<p>The line <code>const regl = require('regl')();</code> creates a full screen canvas element and a <code>WebGLRenderingContext</code>. This context will be available in the command <code>drawTriangle</code> and can be accessed in uniforms and attributes.</p>
<p>Context and props can be accessed with <code>pointSize: (context, prop) => prop.pointSize,</code>, or even with a shorthand syntax: <code>regl.context('pixelRatio')</code> and <code>regl.prop('scale')</code>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599302618/context_lgay6x.png">https://res.cloudinary.com/jackdbd/image/upload/v1599302618/context_lgay6x.png</a></p>
<p>As you can see from the image, the <code>drawingBufferWidth</code> is equal to the <code>viewportWidth</code>, and the <code>drawingBufferHeight</code> is equal to the <code>viewportHeight</code>. However, this is true only because we created a <code>WebGLRenderingContext</code> without specifying any initialization argument for the regl contructor and because we didn’t set the <code>viewport</code>.</p>
<h2 id="h-one-canvas-two-sizes" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-get-started-with-regl-and-webpack/#h-one-canvas-two-sizes"><span aria-hidden="true">#</span></a> One canvas, two sizes</h2>
<p>When you think about the canvas for your regl/WebGL application you need to decide how big you want the canvas to be (e.g. full screen, a small portion of the web page, etc), and how many pixels you want to display.</p>
<p>In order to illustrate this concept we need to make a few small <a href="https://webglfundamentals.org/webgl/lessons/webgl-resizing-the-canvas.html">changes to the code</a>.</p>
<p>In your <code>one-shot-rendering.html</code>, add a <code><canvas></code> element.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>body</span><span class="token punctuation">></span></span>
<span class="token comment"><!-- don't change anything above this point --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h1</span><span class="token punctuation">></span></span>regl one shot rendering example<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h1</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>container<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>canvas</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>regl-canvas<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>canvas</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token comment"><!-- don't change anything below this point --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>body</span><span class="token punctuation">></span></span></code></pre>
<p>You can set the size of the canvas in CSS. In your <code>main.css</code> file define a <code>canvas</code> selector. Let’s say that you want the canvas to be as wide as half of the viewport width, and as high as half of the viewport height. You can also add a red border to understand where the canvas is.</p>
<pre class="language-css"><code class="language-css"><span class="token comment">/* Set the size of the canvas element (i.e. how big the canvas is). */</span>
<span class="token selector">canvas</span> <span class="token punctuation">{</span>
<span class="token property">width</span><span class="token punctuation">:</span> 50vw<span class="token punctuation">;</span>
<span class="token property">height</span><span class="token punctuation">:</span> 50vh<span class="token punctuation">;</span>
<span class="token property">display</span><span class="token punctuation">:</span> block<span class="token punctuation">;</span>
<span class="token property">border</span><span class="token punctuation">:</span> 1px solid #f00<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>The size of the canvas has nothing to do with how many pixels you want to have the canvas. For this, you need to modify the Javascript code. In <code>one-shot-rendering.js</code>, replace this line:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> regl <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'regl'</span><span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>with this code:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> canvas <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">getElementById</span><span class="token punctuation">(</span><span class="token string">'regl-canvas'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> regl <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'regl'</span><span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
canvas<span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">// Set the size of the drawingbuffer (i.e. how many pixels are in the canvas)</span>
canvas<span class="token punctuation">.</span>width <span class="token operator">=</span> canvas<span class="token punctuation">.</span>clientWidth
canvas<span class="token punctuation">.</span>height <span class="token operator">=</span> canvas<span class="token punctuation">.</span>clientHeight</code></pre>
<p>and in the <code>drawTriangle</code> command add the <code>viewport</code>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token literal-property property">viewport</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">x</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span>
<span class="token literal-property property">y</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span>
<span class="token literal-property property">width</span><span class="token operator">:</span> canvas<span class="token punctuation">.</span>width<span class="token punctuation">,</span>
<span class="token literal-property property">height</span><span class="token operator">:</span> canvas<span class="token punctuation">.</span>height<span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span></code></pre>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599302618/canvas_shfdsz.png">https://res.cloudinary.com/jackdbd/image/upload/v1599302618/canvas_shfdsz.png</a></p>
<h2 id="h-batch-rendering" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-get-started-with-regl-and-webpack/#h-batch-rendering"><span aria-hidden="true">#</span></a> Batch rendering</h2>
<p>With <a href="https://regl.party/api#batch-rendering">batch rendering</a> you can execute a regl command multiple times. The command is executed once for each element of the array passed as argument. Let’s see it in action.</p>
<p>Create a new HTML file called <code>batch-rendering.html</code>:</p>
<pre class="language-html"><code class="language-html"><span class="token comment"><!-- batch-rendering.html --></span>
<span class="token doctype"><span class="token punctuation"><!</span><span class="token doctype-tag">DOCTYPE</span> <span class="token name">html</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>html</span> <span class="token attr-name">lang</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>en<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>head</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">charset</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>utf-8<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">http-equiv</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>x-ua-compatible<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>ie=edge<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>title</span><span class="token punctuation">></span></span>batch rendering<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>title</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>viewport<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>width=device-width, initial-scale=1.0<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>description<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>regl example with batch rendering<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token comment"><!-- bundle.css is injected here by html-webpack-plugin --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>head</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>body</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>ul</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>li</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/index.html<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>Home<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>li</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>ul</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h1</span><span class="token punctuation">></span></span>regl batch rendering example<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h1</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>container<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>canvas</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>regl-canvas<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>canvas</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token comment"><!-- bundle.js is injected here by html-webpack-plugin --></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>body</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>html</span><span class="token punctuation">></span></span></code></pre>
<p>You will need to create a new <code>entry</code> in webpack, as well as a new configuration for the <code>html-webpack-plugin</code>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// webpack.config.js</span>
<span class="token literal-property property">entry</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token comment">// other entry points</span>
<span class="token string-property property">'batch-rendering'</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'js'</span><span class="token punctuation">,</span> <span class="token string">'batch-rendering.js'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token comment">// other entry points</span>
<span class="token punctuation">}</span>
<span class="token comment">// other configurations of HtmlWebpackPlugin</span>
<span class="token keyword">new</span> <span class="token class-name">HtmlWebpackPlugin</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">template</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'templates'</span><span class="token punctuation">,</span> <span class="token string">'batch-rendering.html'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">hash</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
<span class="token literal-property property">filename</span><span class="token operator">:</span> <span class="token string">'batch-rendering.html'</span><span class="token punctuation">,</span>
<span class="token literal-property property">chunks</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">'commons'</span><span class="token punctuation">,</span> <span class="token string">'batch-rendering'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token comment">// other configurations of HtmlWebpackPlugin</span></code></pre>
<p>Finally, here is the regl application:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// batch-rendering.js</span>
<span class="token keyword">import</span> <span class="token string">'../css/main.css'</span>
<span class="token keyword">const</span> canvas <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">getElementById</span><span class="token punctuation">(</span><span class="token string">'regl-canvas'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> regl <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'regl'</span><span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
canvas<span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
canvas<span class="token punctuation">.</span>width <span class="token operator">=</span> canvas<span class="token punctuation">.</span>clientWidth
canvas<span class="token punctuation">.</span>height <span class="token operator">=</span> canvas<span class="token punctuation">.</span>clientHeight
<span class="token comment">// regl render command to draw a SINGLE triangle</span>
<span class="token keyword">const</span> drawTriangle <span class="token operator">=</span> <span class="token function">regl</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">frag</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
precision mediump float;
uniform vec4 color;
void main() {
gl_FragColor = color;
}
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">vert</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
precision mediump float;
uniform float angle;
uniform vec2 offset;
attribute vec2 position;
float x, y, z, w;
void main() {
x = cos(angle) * position.x + sin(angle) * position.y + offset.x;
y = -sin(angle) * position.x + cos(angle) * position.y + offset.y;
z = 0.0;
w = 1.0;
gl_Position = vec4(x, y, z, w);
}
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">viewport</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">x</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span>
<span class="token literal-property property">y</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span>
<span class="token literal-property property">width</span><span class="token operator">:</span> canvas<span class="token punctuation">.</span>width<span class="token punctuation">,</span>
<span class="token literal-property property">height</span><span class="token operator">:</span> canvas<span class="token punctuation">.</span>height<span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">attributes</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token comment">// [x,y] positions of the 3 vertices (without the offset)</span>
<span class="token literal-property property">position</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">0.25</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">0.1</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">0.5</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">uniforms</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token comment">// Destructure context and pass only tick. Pass props just because we need</span>
<span class="token comment">// to pass a third argument: batchId, which gives the index of the regl</span>
<span class="token comment">// 'drawTriangle' render command.</span>
<span class="token function-variable function">color</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> tick <span class="token punctuation">}</span><span class="token punctuation">,</span> props<span class="token punctuation">,</span> batchId</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> r <span class="token operator">=</span> Math<span class="token punctuation">.</span><span class="token function">sin</span><span class="token punctuation">(</span>
<span class="token number">0.02</span> <span class="token operator">*</span> <span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token number">0.1</span> <span class="token operator">+</span> Math<span class="token punctuation">.</span><span class="token function">sin</span><span class="token punctuation">(</span>batchId<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token punctuation">(</span>tick <span class="token operator">+</span> <span class="token number">3.0</span> <span class="token operator">*</span> batchId<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">)</span>
<span class="token keyword">const</span> g <span class="token operator">=</span> Math<span class="token punctuation">.</span><span class="token function">cos</span><span class="token punctuation">(</span><span class="token number">0.02</span> <span class="token operator">*</span> <span class="token punctuation">(</span><span class="token number">0.02</span> <span class="token operator">*</span> tick <span class="token operator">+</span> <span class="token number">0.1</span> <span class="token operator">*</span> batchId<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> b <span class="token operator">=</span> Math<span class="token punctuation">.</span><span class="token function">sin</span><span class="token punctuation">(</span>
<span class="token number">0.02</span> <span class="token operator">*</span> <span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token number">0.3</span> <span class="token operator">+</span> Math<span class="token punctuation">.</span><span class="token function">cos</span><span class="token punctuation">(</span><span class="token number">2.0</span> <span class="token operator">*</span> batchId<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">*</span> tick <span class="token operator">+</span> <span class="token number">0.8</span> <span class="token operator">*</span> batchId<span class="token punctuation">)</span>
<span class="token punctuation">)</span>
<span class="token keyword">const</span> alpha <span class="token operator">=</span> <span class="token number">1.0</span>
<span class="token keyword">return</span> <span class="token punctuation">[</span>r<span class="token punctuation">,</span> g<span class="token punctuation">,</span> b<span class="token punctuation">,</span> alpha<span class="token punctuation">]</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token function-variable function">angle</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> tick <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token number">0.01</span> <span class="token operator">*</span> tick<span class="token punctuation">,</span>
<span class="token literal-property property">offset</span><span class="token operator">:</span> regl<span class="token punctuation">.</span><span class="token function">prop</span><span class="token punctuation">(</span><span class="token string">'offset'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token comment">// disable the depth buffer</span>
<span class="token comment">// https://learningwebgl.com/blog/?p=859</span>
<span class="token literal-property property">depth</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">enable</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">count</span><span class="token operator">:</span> <span class="token number">3</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">// Here we register a per-frame callback to draw the whole scene</span>
regl<span class="token punctuation">.</span><span class="token function">frame</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token comment">// clear the color buffer</span>
regl<span class="token punctuation">.</span><span class="token function">clear</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">color</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token comment">// r, g, b, a</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">/* In batch rendering a regl rendering command can be executed multiple times
by passing a non-negative integer or an array as the first argument.
The batchId is initially 0 and incremented each time the render command is
executed.
Note: this command draws a SINGLE triangle, but since we are passing an
array of 5 elements it is executed 5 times. */</span>
<span class="token function">drawTriangle</span><span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token comment">// props0</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">0.15</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">0.15</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token comment">// props1...</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">0.15</span><span class="token punctuation">,</span> <span class="token number">0.15</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token number">0.5</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">0.5</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599302618/batch-rendering_ge93a1.png">https://res.cloudinary.com/jackdbd/image/upload/v1599302618/batch-rendering_ge93a1.png</a></p>
<h2 id="h-glsl%3F-there-is-a-loader-for-that" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-get-started-with-regl-and-webpack/#h-glsl%3F-there-is-a-loader-for-that"><span aria-hidden="true">#</span></a> GLSL? There is a loader for that!</h2>
<p>I don’t know about you, but defining shaders with back ticks looks awful to me. It would be much better to write <code>.glsl</code> files and then load them in a regl application. We could use linters and autocompletion, but even more importantly, we would avoid copy pasting the shaders in JS every single time we need them.</p>
<p>Luckily with Webpack it’s pretty easy to fix this issue. There is a loader for that!</p>
<p>Ok, so you need to do these things:</p>
<ol>
<li>add <code>webpack-glsl-loader</code> to <code>devDependencies</code></li>
<li>create <code>.glsl</code> files for the vertex shader and for the fragment shader</li>
<li>configure webpack to load <code>.glsl</code> files with <code>webpack-glsl-loader</code></li>
<li><code>require</code> the shaders in the regl application</li>
</ol>
<p>As an example, we’ll remove the shader code between the back ticks in <code>batch-rendering.js</code> and we’ll use the GLSL loader instead.</p>
<p>Install the GLSL loader with <code>yarn add --dev webpack-glsl-loader</code>.</p>
<p>Create GLSL files for your shaders. Create <code>src/glsl/vertex/batch.glsl</code> for the vertex shader and <code>src/glsl/fragment/batch.glsl</code> for the fragment shader. You just have to copy and paste the code between the back ticks in <code>batch-rendering.js</code>.</p>
<p>Configure webpack to use <code>webpack-glsl-loader</code> to load <code>.glsl</code> files.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// webpack.config.js</span>
<span class="token comment">// rule for .glsl files (shaders)</span>
<span class="token punctuation">{</span>
<span class="token literal-property property">test</span><span class="token operator">:</span> <span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">\.glsl$</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">use</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token literal-property property">loader</span><span class="token operator">:</span> <span class="token string">'webpack-glsl-loader'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span></code></pre>
<p>Finally, <code>require</code> your shaders in <code>batch-rendering.js</code>.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// batch-rendering.js</span>
<span class="token keyword">import</span> <span class="token string">'../css/main.css'</span>
<span class="token keyword">const</span> vertexShader <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'../glsl/vertex/batch.glsl'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> fragmentShader <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'../glsl/fragment/batch.glsl'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> canvas <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">getElementById</span><span class="token punctuation">(</span><span class="token string">'regl-canvas'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> regl <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'regl'</span><span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
canvas<span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
canvas<span class="token punctuation">.</span>width <span class="token operator">=</span> canvas<span class="token punctuation">.</span>clientWidth
canvas<span class="token punctuation">.</span>height <span class="token operator">=</span> canvas<span class="token punctuation">.</span>clientHeight
<span class="token comment">// regl render command to draw a SINGLE triangle</span>
<span class="token keyword">const</span> drawTriangle <span class="token operator">=</span> <span class="token function">regl</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">vert</span><span class="token operator">:</span> vertexShader<span class="token punctuation">,</span>
<span class="token literal-property property">frag</span><span class="token operator">:</span> fragmentShader<span class="token punctuation">,</span>
<span class="token literal-property property">viewport</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">x</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span>
<span class="token literal-property property">y</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span>
<span class="token literal-property property">width</span><span class="token operator">:</span> canvas<span class="token punctuation">.</span>width<span class="token punctuation">,</span>
<span class="token literal-property property">height</span><span class="token operator">:</span> canvas<span class="token punctuation">.</span>height<span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">attributes</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token comment">// [x,y] positions of the 3 vertices (without the offset)</span>
<span class="token literal-property property">position</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">0.25</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">0.1</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">0.5</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">uniforms</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token comment">// Destructure context and pass only tick. Pass props just because we need</span>
<span class="token comment">// to pass a third argument: batchId, which gives the index of the regl</span>
<span class="token comment">// 'drawTriangle' render command.</span>
<span class="token function-variable function">color</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> tick <span class="token punctuation">}</span><span class="token punctuation">,</span> props<span class="token punctuation">,</span> batchId</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> r <span class="token operator">=</span> Math<span class="token punctuation">.</span><span class="token function">sin</span><span class="token punctuation">(</span>
<span class="token number">0.02</span> <span class="token operator">*</span> <span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token number">0.1</span> <span class="token operator">+</span> Math<span class="token punctuation">.</span><span class="token function">sin</span><span class="token punctuation">(</span>batchId<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token punctuation">(</span>tick <span class="token operator">+</span> <span class="token number">3.0</span> <span class="token operator">*</span> batchId<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">)</span>
<span class="token keyword">const</span> g <span class="token operator">=</span> Math<span class="token punctuation">.</span><span class="token function">cos</span><span class="token punctuation">(</span><span class="token number">0.02</span> <span class="token operator">*</span> <span class="token punctuation">(</span><span class="token number">0.02</span> <span class="token operator">*</span> tick <span class="token operator">+</span> <span class="token number">0.1</span> <span class="token operator">*</span> batchId<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> b <span class="token operator">=</span> Math<span class="token punctuation">.</span><span class="token function">sin</span><span class="token punctuation">(</span>
<span class="token number">0.02</span> <span class="token operator">*</span> <span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token number">0.3</span> <span class="token operator">+</span> Math<span class="token punctuation">.</span><span class="token function">cos</span><span class="token punctuation">(</span><span class="token number">2.0</span> <span class="token operator">*</span> batchId<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">*</span> tick <span class="token operator">+</span> <span class="token number">0.8</span> <span class="token operator">*</span> batchId<span class="token punctuation">)</span>
<span class="token punctuation">)</span>
<span class="token keyword">const</span> alpha <span class="token operator">=</span> <span class="token number">1.0</span>
<span class="token keyword">return</span> <span class="token punctuation">[</span>r<span class="token punctuation">,</span> g<span class="token punctuation">,</span> b<span class="token punctuation">,</span> alpha<span class="token punctuation">]</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token function-variable function">angle</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">{</span> tick <span class="token punctuation">}</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token number">0.01</span> <span class="token operator">*</span> tick<span class="token punctuation">,</span>
<span class="token literal-property property">offset</span><span class="token operator">:</span> regl<span class="token punctuation">.</span><span class="token function">prop</span><span class="token punctuation">(</span><span class="token string">'offset'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token comment">// disable the depth buffer</span>
<span class="token comment">// https://learningwebgl.com/blog/?p=859</span>
<span class="token literal-property property">depth</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">enable</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">count</span><span class="token operator">:</span> <span class="token number">3</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">// Here we register a per-frame callback to draw the whole scene</span>
regl<span class="token punctuation">.</span><span class="token function">frame</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
regl<span class="token punctuation">.</span><span class="token function">clear</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">color</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token comment">// r, g, b, a</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token comment">/* In batch rendering a regl rendering command can be executed multiple times
by passing a non-negative integer or an array as the first argument.
The batchId is initially 0 and incremented each time the render command is
executed.
Note: this command draws a SINGLE triangle, but since we are passing an
array of 5 elements it is executed 5 times. */</span>
<span class="token function">drawTriangle</span><span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token comment">// props0</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">0.15</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">0.15</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token comment">// props1...</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">0.15</span><span class="token punctuation">,</span> <span class="token number">0.15</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token number">0.5</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span> <span class="token literal-property property">offset</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">0.5</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-hungarian-notation%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-get-started-with-regl-and-webpack/#h-hungarian-notation%3F"><span aria-hidden="true">#</span></a> Hungarian notation?</h2>
<p>On <a href="https://webglfundamentals.org/webgl/lessons/webgl-shaders-and-glsl.html">WebGL Fundamentals</a> they say that it’s common practice to place a letter in front of the variables to indicate their type: <code>u</code> for <code>uniforms</code>, <code>a</code> for <code>attributes</code> and the <code>v</code> for <code>varyings</code>. I’m not a huge fan of this <a href="https://en.wikipedia.org/wiki/Hungarian_notation">Hungarian notation</a> though. I don’t think it really improves the readability of the code and I don’t think I will use it.</p>
<h2 id="h-conclusion" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-get-started-with-regl-and-webpack/#h-conclusion"><span aria-hidden="true">#</span></a> Conclusion</h2>
<p>I plan to write several articles about regl in the near future. In this one we learned how to configure Webpack for regl applications. In the next one I will create a few examples with d3 and regl.</p>
<p>Stay tuned!</p>
https://www.giacomodebidda.com/posts/a-5-minute-intro-to-hypothesis/A 5 minute Intro to Hypothesis2017-10-14T08:30:03Z2017-10-14T08:30:03Z
<p>Let’s say you have a Python module called <code>example.py</code> where you have a function called <code>add_numbers</code> that you want to test.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># example.py</span>
<span class="token keyword">def</span> <span class="token function">add_numbers</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> a <span class="token operator">+</span> b</code></pre>
<p>This function adds two numbers, so among other things you might want to check that the commutative property holds for all the inputs that the function receives. You create a file called <code>test_example.py</code> and start writing a simple unit test to prove it.</p>
<h2 id="h-a-simple-unit-test" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-5-minute-intro-to-hypothesis/#h-a-simple-unit-test"><span aria-hidden="true">#</span></a> A simple unit test</h2>
<pre class="language-python"><code class="language-python"><span class="token comment"># test_example.py</span>
<span class="token keyword">import</span> unittest
<span class="token keyword">from</span> your_python_module<span class="token punctuation">.</span>example <span class="token keyword">import</span> add_numbers
<span class="token keyword">class</span> <span class="token class-name">TestAddNumbers</span><span class="token punctuation">(</span>unittest<span class="token punctuation">.</span>TestCase<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">test_add_numers_is_commutative</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>assertEqual<span class="token punctuation">(</span>add_numbers<span class="token punctuation">(</span><span class="token number">1.23</span><span class="token punctuation">,</span> <span class="token number">4.56</span><span class="token punctuation">)</span><span class="token punctuation">,</span> add_numbers<span class="token punctuation">(</span><span class="token number">4.56</span><span class="token punctuation">,</span> <span class="token number">1.23</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
unittest<span class="token punctuation">.</span>main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>That’s cool, but you actually didn’t prove that the commutative property holds in general. You have just proved that <em>for this specific case</em> such property holds.</p>
<p>You realize that the combination of <code>1.23</code> and <code>4.56</code> is a very tiny subset of the entire input space of numbers that your function can receive, so you write more tests.</p>
<h2 id="h-a-common-solution-write-more-test-cases" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-5-minute-intro-to-hypothesis/#h-a-common-solution-write-more-test-cases"><span aria-hidden="true">#</span></a> A common solution: write more test cases</h2>
<pre class="language-python"><code class="language-python"><span class="token comment"># test_example.py</span>
<span class="token comment"># more tests here...</span>
<span class="token keyword">def</span> <span class="token function">test_add_numers_is_commutative_another_case</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>assertEqual<span class="token punctuation">(</span>add_numbers<span class="token punctuation">(</span><span class="token number">0.789</span><span class="token punctuation">,</span> <span class="token number">321</span><span class="token punctuation">)</span><span class="token punctuation">,</span> add_numbers<span class="token punctuation">(</span><span class="token number">321</span><span class="token punctuation">,</span> <span class="token number">0.789</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># more tests here...</span></code></pre>
<p>Not a huge gain. You have just proved that <em>for these other specific cases</em> that you wrote the commutative property holds. And obviously you don’t want to write a million test cases by hand.</p>
<p>Maybe you have heard about <a href="https://en.wikipedia.org/wiki/Fuzzing">fuzzing</a>, and you want to use it to create random test cases every time you run the test.</p>
<h2 id="h-a-better-solution-fuzzing-and-ddt" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-5-minute-intro-to-hypothesis/#h-a-better-solution-fuzzing-and-ddt"><span aria-hidden="true">#</span></a> A better solution: fuzzing and ddt</h2>
<p>You can create a pair of random floats, so every time you run the test you have a new test case. That’s a bit dangerous though, because if you have a failure you cannot reproduce it easily.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># test_example.py</span>
<span class="token keyword">import</span> random
<span class="token keyword">import</span> unittest
<span class="token keyword">class</span> <span class="token class-name">TestAddNumbers</span><span class="token punctuation">(</span>unittest<span class="token punctuation">.</span>TestCase<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">test_add_numers_is_commutative</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
a <span class="token operator">=</span> random<span class="token punctuation">.</span>random<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">10.0</span>
b <span class="token operator">=</span> random<span class="token punctuation">.</span>random<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">10.0</span>
self<span class="token punctuation">.</span>assertEqual<span class="token punctuation">(</span>add_numbers<span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">,</span> add_numbers<span class="token punctuation">(</span>b<span class="token punctuation">,</span> a<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
unittest<span class="token punctuation">.</span>main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>You can also use a library called <a href="https://www.giacomodebidda.com/posts/multiply-your-python-unit-test-cases-with-ddt/">ddt</a> to generate many random test cases.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># test_example.py</span>
<span class="token keyword">import</span> random
<span class="token keyword">import</span> unittest
<span class="token keyword">from</span> ddt <span class="token keyword">import</span> ddt<span class="token punctuation">,</span> idata<span class="token punctuation">,</span> unpack
<span class="token keyword">def</span> <span class="token function">float_pairs_generator</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
num_test_cases <span class="token operator">=</span> <span class="token number">100</span>
<span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>num_test_cases<span class="token punctuation">)</span><span class="token punctuation">:</span>
a <span class="token operator">=</span> random<span class="token punctuation">.</span>random<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">10.0</span>
b <span class="token operator">=</span> random<span class="token punctuation">.</span>random<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">10.0</span>
<span class="token keyword">yield</span> <span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@ddt</span>
<span class="token keyword">class</span> <span class="token class-name">TestAddNumbers</span><span class="token punctuation">(</span>unittest<span class="token punctuation">.</span>TestCase<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@idata</span><span class="token punctuation">(</span>float_pairs_generator<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@unpack</span>
<span class="token keyword">def</span> <span class="token function">test_add_floats_ddt</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>assertEqual<span class="token punctuation">(</span>add_numbers<span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">,</span> add_numbers<span class="token punctuation">(</span>b<span class="token punctuation">,</span> a<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
unittest<span class="token punctuation">.</span>main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>Here the <code>float_pairs_generator</code> generator function creates 100 random pairs of floats. With this trick you can multiply the number of test cases while keeping your tests easy to maintain.</p>
<p>That’s definitely a step in the right direction, but if you think about it we are still testing some random combinations of numbers between 0.0 and 10.0 here. Not a very extensive portion of the input domain of the function <code>add_numbers</code>.</p>
<p>You have two options:</p>
<ol>
<li>find a way to generate <em>domain objects</em> that your function can accept. In this case the domain objects are the floats that <code>add_numbers</code> can receive.</li>
<li>use hypothesis</li>
</ol>
<p>I don’t know about you, but I’m going for the second one.</p>
<h2 id="h-the-best-solution-hypothesis" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-5-minute-intro-to-hypothesis/#h-the-best-solution-hypothesis"><span aria-hidden="true">#</span></a> The best solution: Hypothesis</h2>
<p>Here is how you write a test that checks that the commutative property holds for a pair of floats.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># test_example.py</span>
<span class="token keyword">from</span> hypothesis <span class="token keyword">import</span> given
<span class="token keyword">from</span> hypothesis<span class="token punctuation">.</span>strategies <span class="token keyword">import</span> floats
<span class="token keyword">from</span> your_python_module<span class="token punctuation">.</span>example <span class="token keyword">import</span> add_numbers
<span class="token decorator annotation punctuation">@given</span><span class="token punctuation">(</span>a<span class="token operator">=</span>floats<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> b<span class="token operator">=</span>floats<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">test_add_numbers</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">assert</span> add_numbers<span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span> <span class="token operator">==</span> add_numbers<span class="token punctuation">(</span>b<span class="token punctuation">,</span> a<span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
test_add_numbers<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>When I ran this test I was shocked. It failed!</p>
<p>WTF! How it that possible that this test fails, after I have tried 100 test cases with ddt?</p>
<p>Luckily with hypothesis you can increase the verbosity level of your test by using the <code>@settings</code> decorator.</p>
<p>Let’s say you also want to test a specific test case: <code>a == 1.23</code> and <code>b == 4.56</code>. For this you can use the <code>@example</code> decorator. This is nice because now your test provides some <em>documentation</em> to anyone who wants to use the <code>add_numbers</code> function, and at the same time you are testing a specific case that you know about or that might be particularly hard to hit.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># test_example.py</span>
<span class="token keyword">from</span> hypothesis <span class="token keyword">import</span> given<span class="token punctuation">,</span> example<span class="token punctuation">,</span> settings<span class="token punctuation">,</span> Verbosity
<span class="token keyword">from</span> hypothesis<span class="token punctuation">.</span>strategies <span class="token keyword">import</span> floats
<span class="token keyword">from</span> your_python_module<span class="token punctuation">.</span>example <span class="token keyword">import</span> add_numbers
<span class="token decorator annotation punctuation">@settings</span><span class="token punctuation">(</span>verbosity<span class="token operator">=</span>Verbosity<span class="token punctuation">.</span>verbose<span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@example</span><span class="token punctuation">(</span>a<span class="token operator">=</span><span class="token number">1.23</span><span class="token punctuation">,</span> b<span class="token operator">=</span><span class="token number">4.56</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@given</span><span class="token punctuation">(</span>a<span class="token operator">=</span>floats<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> b<span class="token operator">=</span>floats<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">test_add_numbers</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">assert</span> add_numbers<span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span> <span class="token operator">==</span> add_numbers<span class="token punctuation">(</span>b<span class="token punctuation">,</span> a<span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
test_add_numbers<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>Obviously the test fails again, but this time you get more insights about it.</p>
<pre class="language-shell"><code class="language-shell">Trying example: test_add_numbers<span class="token punctuation">(</span>a<span class="token operator">=</span><span class="token number">1.23</span>, <span class="token assign-left variable">b</span><span class="token operator">=</span><span class="token number">4.56</span><span class="token punctuation">)</span>
Trying example: test_add_numbers<span class="token punctuation">(</span>a<span class="token operator">=</span><span class="token number">0.0</span>, <span class="token assign-left variable">b</span><span class="token operator">=</span>nan<span class="token punctuation">)</span>
Traceback <span class="token punctuation">(</span>most recent call last<span class="token punctuation">)</span>:
<span class="token comment"># Traceback here...</span>
AssertionError
Trying example: test_add_numbers<span class="token punctuation">(</span>a<span class="token operator">=</span><span class="token number">0.0</span>, <span class="token assign-left variable">b</span><span class="token operator">=</span>nan<span class="token punctuation">)</span>
Traceback <span class="token punctuation">(</span>most recent call last<span class="token punctuation">)</span>:
<span class="token comment"># Traceback here...</span>
AssertionError
Trying example: test_add_numbers<span class="token punctuation">(</span>a<span class="token operator">=</span><span class="token number">0.0</span>, <span class="token assign-left variable">b</span><span class="token operator">=</span><span class="token number">1.0</span><span class="token punctuation">)</span>
Trying example: test_add_numbers<span class="token punctuation">(</span>a<span class="token operator">=</span><span class="token number">0.0</span>, <span class="token assign-left variable">b</span><span class="token operator">=</span><span class="token number">4293918720.0</span><span class="token punctuation">)</span>
Trying example: test_add_numbers<span class="token punctuation">(</span>a<span class="token operator">=</span><span class="token number">0.0</span>, <span class="token assign-left variable">b</span><span class="token operator">=</span><span class="token number">281406257233920.0</span><span class="token punctuation">)</span>
Trying example: test_add_numbers<span class="token punctuation">(</span>a<span class="token operator">=</span><span class="token number">0.0</span>, <span class="token assign-left variable">b</span><span class="token operator">=</span><span class="token number">7</span>.204000185188352e+16<span class="token punctuation">)</span>
Trying example: test_add_numbers<span class="token punctuation">(</span>a<span class="token operator">=</span><span class="token number">0.0</span>, <span class="token assign-left variable">b</span><span class="token operator">=</span>inf<span class="token punctuation">)</span>
<span class="token comment"># more test cases...</span>
You can <span class="token function">add</span> @seed<span class="token punctuation">(</span><span class="token number">247616548810050264291730850370106354271</span><span class="token punctuation">)</span> to this <span class="token builtin class-name">test</span> to reproduce this failure.</code></pre>
<p>That’s really helpful. You can see all the test case that were successful and the ones that caused a failure. You get also a <code>seed</code> that you can use to reproduce this very specific failure at a later time or on a different computer. This is so awesome.</p>
<p>Anyway, why does this test fail? It fails because in Python <code>nan</code> and <code>inf</code> are valid floats, so the function <code>floats()</code> might create some test cases that have <code>a == nan</code> and/or <code>b == inf</code>.</p>
<p>Are <code>nan</code> and <code>inf</code> valid inputs for your application? Maybe. It depends on your application.</p>
<p>If you are absolutely sure that <code>add_numbers</code> will never receive a <code>nan</code> or a <code>inf</code> as inputs, you can write a test that never generates either <code>nan</code> or <code>inf</code>. You just have to set <code>allow_nan</code> and <code>allow_infinity</code> to <code>False</code> in the <code>@given</code> decorator. Easy peasy.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># test_example.py</span>
<span class="token keyword">from</span> hypothesis <span class="token keyword">import</span> given<span class="token punctuation">,</span> example<span class="token punctuation">,</span> settings<span class="token punctuation">,</span> Verbosity
<span class="token keyword">from</span> hypothesis<span class="token punctuation">.</span>strategies <span class="token keyword">import</span> floats
<span class="token keyword">from</span> your_python_module<span class="token punctuation">.</span>example <span class="token keyword">import</span> add_numbers
<span class="token decorator annotation punctuation">@given</span><span class="token punctuation">(</span>
a<span class="token operator">=</span>floats<span class="token punctuation">(</span>allow_nan<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">,</span> allow_infinity<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
b<span class="token operator">=</span>floats<span class="token punctuation">(</span>allow_nan<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">,</span> allow_infinity<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">test_add_numbers</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">assert</span> add_numbers<span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span> <span class="token operator">==</span> add_numbers<span class="token punctuation">(</span>b<span class="token punctuation">,</span> a<span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
test_add_numbers<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>But what if <code>add_numbers</code> could in fact receive <code>nan</code> or <code>inf</code> as inputs (a much more realistic assumption). In this case the test should be able to generate <code>nan</code> or <code>inf</code>, your function should raise specific exceptions that you will have to handle somewhere else in your application, and the test should not consider <em>such specific exceptions</em> as failures.</p>
<p>Here is how <code>example.py</code> might look:</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># example.py</span>
<span class="token keyword">import</span> math
<span class="token keyword">class</span> <span class="token class-name">NaNIsNotAllowed</span><span class="token punctuation">(</span>ValueError<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token keyword">class</span> <span class="token class-name">InfIsNotAllowed</span><span class="token punctuation">(</span>ValueError<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token keyword">def</span> <span class="token function">add_numbers</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> math<span class="token punctuation">.</span>isnan<span class="token punctuation">(</span>a<span class="token punctuation">)</span> <span class="token keyword">or</span> math<span class="token punctuation">.</span>isnan<span class="token punctuation">(</span>b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> NaNIsNotAllowed<span class="token punctuation">(</span><span class="token string">'nan is not a valid input'</span><span class="token punctuation">)</span>
<span class="token keyword">elif</span> math<span class="token punctuation">.</span>isinf<span class="token punctuation">(</span>a<span class="token punctuation">)</span> <span class="token keyword">or</span> math<span class="token punctuation">.</span>isinf<span class="token punctuation">(</span>b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> InfIsNotAllowed<span class="token punctuation">(</span><span class="token string">'inf is not a valid input'</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> a <span class="token operator">+</span> b</code></pre>
<p>And here is how you write the test with hypothesis:</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># test_example.py</span>
<span class="token keyword">from</span> hypothesis <span class="token keyword">import</span> given
<span class="token keyword">from</span> hypothesis<span class="token punctuation">.</span>strategies <span class="token keyword">import</span> floats
<span class="token keyword">from</span> your_python_module<span class="token punctuation">.</span>example <span class="token keyword">import</span> add_numbers<span class="token punctuation">,</span> NaNIsNotAllowed<span class="token punctuation">,</span> InfIsNotAllowed
<span class="token decorator annotation punctuation">@given</span><span class="token punctuation">(</span>a<span class="token operator">=</span>floats<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> b<span class="token operator">=</span>floats<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">test_add_numbers</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
<span class="token keyword">assert</span> add_numbers<span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span> <span class="token operator">==</span> add_numbers<span class="token punctuation">(</span>b<span class="token punctuation">,</span> a<span class="token punctuation">)</span>
<span class="token keyword">except</span> <span class="token punctuation">(</span>NaNIsNotAllowed<span class="token punctuation">,</span> InfIsNotAllowed<span class="token punctuation">)</span><span class="token punctuation">:</span>
reject<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
test_add_numbers<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>This test will generate some <code>nan</code> and <code>inf</code> inputs, <code>add_numbers</code> will raise either <code>NaNIsNotAllowed</code> or <code>InfIsNotAllowed</code> and the test will catch these exceptions and reject them as test failures (i.e. the test case will be considered a success when either <code>NaNIsNotAllowed</code> or <code>InfIsNotAllowed</code> occurs).</p>
<p>Can you really afford to reject <code>nan</code> as an input value for <code>add_numbers</code>? Maybe not. Let’s say your code needs to sum two samples in a time series, and one sample of the time series is missing: <code>nan</code> would be a perfectly valid input for <code>add_numbers</code> in such case.</p>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-5-minute-intro-to-hypothesis/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<ul>
<li><a href="https://hypothesis.works/articles/what-is-property-based-testing/">What is Property Based Testing?</a></li>
<li><a href="https://hypothesis.works/articles/getting-started-with-hypothesis/">Getting started with Hypothesis</a></li>
<li><a href="https://hypothesis.works/articles/anatomy-of-a-test/">Anatomy of a Hypothesis Based Test</a></li>
<li><a href="https://hypothesis.works/articles/incremental-property-based-testing/">Evolving toward property-based testing with Hypothesis</a></li>
</ul>
https://www.giacomodebidda.com/posts/how-to-import-d3-plugins-with-webpack/How to import d3 plugins with Webpack2017-10-13T19:30:03Z2017-10-13T19:30:03Z
<p>Last week I started working on a new visualization and I wanted to include a couple of d3 plugins: <a href="https://d3-legend.susielu.com/">d3-legend</a> by Susie Lu and <a href="https://pbeshai.github.io/d3-line-chunked/">d3-line-chunked</a> by Peter Beshai.</p>
<p>I spent some time figuring out how to include these plugins with webpack, so I’m writing this small reminder here.</p>
<p>As far as I know, there are at least 3 ways to import d3 and some plugins.</p>
<h2 id="h-the-wrong-way-to-do-it" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-import-d3-plugins-with-webpack/#h-the-wrong-way-to-do-it"><span aria-hidden="true">#</span></a> The wrong way to do it</h2>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> d3 <span class="token keyword">from</span> <span class="token string">'d3'</span>
<span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> d3Legend <span class="token keyword">from</span> <span class="token string">'d3-svg-legend'</span>
<span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> d3LineChunked <span class="token keyword">from</span> <span class="token string">'d3-line-chunked'</span></code></pre>
<p>This is bad for 2 reasons:</p>
<ol>
<li>we are importing the entire d3 library</li>
<li>the plugins are not attached to <code>d3</code>, so in order to use them we have to do something like this: <code>const colorLegend = d3Legend.legendColor()</code>.</li>
</ol>
<h2 id="h-the-lazy-way-to-do-it" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-import-d3-plugins-with-webpack/#h-the-lazy-way-to-do-it"><span aria-hidden="true">#</span></a> The lazy way to do it</h2>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> d3Base <span class="token keyword">from</span> <span class="token string">'d3'</span>
<span class="token keyword">import</span> <span class="token punctuation">{</span> legendColor <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'d3-svg-legend'</span>
<span class="token keyword">import</span> <span class="token punctuation">{</span> lineChunked <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'d3-line-chunked'</span>
<span class="token comment">// attach all d3 plugins to the d3 library</span>
<span class="token keyword">const</span> d3 <span class="token operator">=</span> Object<span class="token punctuation">.</span><span class="token function">assign</span><span class="token punctuation">(</span>d3Base<span class="token punctuation">,</span> <span class="token punctuation">{</span> legendColor<span class="token punctuation">,</span> lineChunked <span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>Here we are still importing the entire d3 library, but now the plugins are attached to the <code>d3</code> object. This means that we can use them like this: <code>const colorLegend = d3.legendColor()</code>. I have to say that I start writing a visualization with this setup. It’s vey compact and pretty convenient, because we don’t have to worry about which d3 functions/submodules we need.</p>
<h2 id="h-the-efficient-way-to-do-it" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-import-d3-plugins-with-webpack/#h-the-efficient-way-to-do-it"><span aria-hidden="true">#</span></a> The efficient way to do it</h2>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> select<span class="token punctuation">,</span> selectAll <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'d3-selection'</span>
<span class="token keyword">import</span> <span class="token punctuation">{</span> min<span class="token punctuation">,</span> extent<span class="token punctuation">,</span> range<span class="token punctuation">,</span> descending <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'d3-array'</span>
<span class="token keyword">import</span> <span class="token punctuation">{</span> format <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'d3-format'</span>
<span class="token keyword">import</span> <span class="token punctuation">{</span> scaleLinear <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'d3-scale'</span>
<span class="token keyword">import</span> <span class="token operator">*</span> <span class="token keyword">as</span> request <span class="token keyword">from</span> <span class="token string">'d3-request'</span> <span class="token comment">// d3 submodule (contains d3.csv, d3.json, etc)</span>
<span class="token keyword">import</span> <span class="token punctuation">{</span> legendColor <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'d3-svg-legend'</span> <span class="token comment">// d3 plugin</span>
<span class="token keyword">import</span> <span class="token punctuation">{</span> lineChunked <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'d3-line-chunked'</span> <span class="token comment">// d3 plugin</span>
<span class="token comment">// create a Object with only the subset of functions/submodules/plugins that we need</span>
<span class="token keyword">const</span> d3 <span class="token operator">=</span> Object<span class="token punctuation">.</span><span class="token function">assign</span><span class="token punctuation">(</span>
<span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span>
select<span class="token punctuation">,</span>
selectAll<span class="token punctuation">,</span>
min<span class="token punctuation">,</span>
extent<span class="token punctuation">,</span>
range<span class="token punctuation">,</span>
descending<span class="token punctuation">,</span>
format<span class="token punctuation">,</span>
scaleLinear<span class="token punctuation">,</span>
legendColor<span class="token punctuation">,</span>
lineChunked<span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
request
<span class="token punctuation">)</span></code></pre>
<p>D3 version 4 is not a monolithic library like D3 version 3, but a collection of small modules. This is perfect for a module bundler like Webpack, because it means that we can include in your bundles only the functions that we actually need. As you can see, this is a bit tedious though, that’s why I start with the “lazy way to do it” and change to the “efficient way to do it” only when my visualization is basically finished.</p>
https://www.giacomodebidda.com/posts/a-simple-git-hook-for-your-python-projects/A simple git hook for your Python projects2017-09-10T21:30:03Z2017-09-10T21:30:03Z
<p>A <a href="https://githooks.com/">git hook</a> is a script that git executes before or after a relevant git event or action is triggered. The hooks are stored in the <code>.git/hooks</code> directory of your repository, which is created automatically when you run <code>git init</code>.</p>
<p>Git hooks can be really useful to enforce a certain policy on your commits, push your changes to a continuous integration server, or automatically deploy your code.</p>
<p>I wanted to enforce a very simple policy for my commits: <em>no broken code should be deployed on the master branch</em>. So I wrote this small <code>pre-commit</code> hook:</p>
<pre class="language-shell"><code class="language-shell"><span class="token shebang important">#!/bin/sh</span>
<span class="token assign-left variable">current_branch</span><span class="token operator">=</span><span class="token variable"><span class="token variable">`</span><span class="token function">git</span> branch <span class="token operator">|</span> <span class="token function">grep</span> <span class="token string">'*'</span> <span class="token operator">|</span> <span class="token function">sed</span> <span class="token string">'s/* //'</span><span class="token variable">`</span></span>
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token string">"<span class="token variable">$current_branch</span>"</span> <span class="token operator">=</span> <span class="token string">"master"</span> <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">"You are about to commit on master. I will run your tests first..."</span>
python <span class="token parameter variable">-m</span> unittest discover tests
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token variable">$?</span> <span class="token parameter variable">-eq</span> <span class="token number">0</span> <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token comment"># tests passed, proceed to prepare commit message</span>
<span class="token builtin class-name">exit</span> <span class="token number">0</span>
<span class="token keyword">else</span>
<span class="token comment"># some tests failed, prevent from committing broken code on master</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Some tests failed. You are not allowed to commit broken code on master! Aborting the commit."</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Note: you can still commit broken code on feature branches"</span>
<span class="token builtin class-name">exit</span> <span class="token number">1</span>
<span class="token keyword">fi</span>
<span class="token keyword">fi</span></code></pre>
<p>It’s a simple <em>client side hook</em> that runs all of my Python tests before committing on <code>master</code>. I can still create a feature branch and commit broken code on that, but as soon as I try to merge the feature branch into master, all test run. If any of the tests fails I can’t commit. Simple as that.</p>
<p>Git hooks are language agnostic. I wrote this small hook as a shell script, but you can use other languages liek Perl, Ruby or Python. <a href="https://github.com/bahattincinic/python-git-hook/blob/master/pre-commit">Here</a> is an example of a <code>pre-commit</code> hook in written in Python.</p>
https://www.giacomodebidda.com/posts/visualize-earthquakes-with-plotly-dash/Visualize Earthquakes with Plotly Dash2017-08-31T22:12:03Z2017-08-31T22:12:03Z
<p>Three years ago I followed a few data science courses offered by the Johns Hopkins University on Coursera. Today these courses should be available among the ones in the <a href="https://www.coursera.org/specializations/jhu-data-science#courses">Data Science specialization</a>. All programming assignments were – and still are – in R. At the end of one course we had to create a small web application with <a href="https://shiny.rstudio.com/">Shiny</a> and deploy it on <a href="https://www.shinyapps.io/">shinyapps</a>. At the time I wasn’t that comfortable in writing Javascript and CSS, so having to worry only about R code was quite a relief. I still have the <a href="https://github.com/jackdbd/ShinyEarthquakes">web app that I wrote</a>.</p>
<p>Some time ago I had the idea of rewriting the entire thing in Python, so I started looking for a Python equivalent of Shiny. I stumbled upon <a href="https://github.com/adamhajari/spyre">Spyre</a>, <a href="https://github.com/stitchfix/pyxley">Pyxley</a> and <a href="https://github.com/apache/incubator-superset">Superset</a>). I immediately discarded Superset. It looked amazing, but I wanted something for a very small application, not an enterprise-ready business intelligence tool. Spyre didn’t convince me, and I tried but struggled with Pyxley.</p>
<p>I toyed with the idea of writing the application with a combination of Flask for the logic and routing, Vue.js for the front-end, Webpack for asset bundling and maybe a SASS framework (or toolkit, like <a href="https://www.oddbird.net/susy/">Susy</a>) for styling. I knew I would have to invest a considerable amount of time to put everything together, so I left the project on the side for a while.</p>
<p>A few months passed and I discovered a few more packages: <a href="https://github.com/jwkvam/bowtie">Bowtie</a>, <a href="https://demo.bokehplots.com/">Bokeh</a>, <a href="https://plot.ly/dash/">Dash</a>. I found out that you can also create an <a href="https://plot.ly/python/create-online-dashboard/">online dashboard with plotly</a>.</p>
<p>According to the <a href="https://plot.ly/dash/introduction">documentation</a>, “Dash is simple enough that you can bind a user interface around your Python code in an afternoon”. In fact, for a simple dashboard with a dropdown menu as the input, and a time series as the output, you need <a href="https://medium.com/@plotlygraphs/introducing-dash-5ecf7191b503">less than 50 lines of code</a>.</p>
<p>Dash allows you to create <em>reactive</em> web applications. This means that changes to <em>input</em> UI component/s trigger changes to an <em>output</em> UI component.</p>
<p>The UI components are created with D3.js and WebGL, so they look amazing. And you get all of this without having to write any HTML/JS/CSS. Under the hood Dash converts React components (written in JavaScript) into Python classes that are compatible with the Dash ecosystem.</p>
<p>The <a href="https://plot.ly/dash/getting-started">getting started</a> is top-notch, so I suggest you to start from there if you want to try Dash out. Here I will briefly describe what I did for my app.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303705/demo_dudpt8.gif">https://res.cloudinary.com/jackdbd/image/upload/v1599303705/demo_dudpt8.gif</a></p>
<h2 id="h-imports" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/visualize-earthquakes-with-plotly-dash/#h-imports"><span aria-hidden="true">#</span></a> Imports</h2>
<p>Here are my import statements. <code>dash_html_components</code> are pure HTML components, and <code>dash_core_components</code> are the reactive components. You need to use <em>one or more</em> <code>Input</code> to trigger changes to a <em>single</em> <code>Output</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> os
<span class="token keyword">import</span> arrow
<span class="token keyword">import</span> requests
<span class="token keyword">import</span> functools
<span class="token keyword">import</span> pandas <span class="token keyword">as</span> pd
<span class="token keyword">import</span> dash_core_components <span class="token keyword">as</span> dcc
<span class="token keyword">import</span> dash_html_components <span class="token keyword">as</span> html
<span class="token keyword">import</span> plotly<span class="token punctuation">.</span>graph_objs <span class="token keyword">as</span> go
<span class="token keyword">import</span> plotly<span class="token punctuation">.</span>plotly <span class="token keyword">as</span> py
<span class="token keyword">from</span> flask <span class="token keyword">import</span> Flask<span class="token punctuation">,</span> json
<span class="token keyword">from</span> dash <span class="token keyword">import</span> Dash
<span class="token keyword">from</span> dash<span class="token punctuation">.</span>dependencies <span class="token keyword">import</span> Input<span class="token punctuation">,</span> Output
<span class="token keyword">from</span> dotenv <span class="token keyword">import</span> load_dotenv</code></pre>
<p>When the app is running on my computer I enable <code>debug</code> and load the environment variables from a <code>.env</code> file (not checked in).<br />
When the app is running on <a href="https://belle-croissant-54211.herokuapp.com/">Heroku</a> I disable <code>debug</code> and use an external Javascript snippet to include Google Analytics. I can’t remeber where I found the <code>try/except</code> to understand whether the app is on Heroku or not, but I find it very pythonic.</p>
<blockquote>
<p><a href="https://docs.python.org/3/glossary.html#term-eafp">EAFP</a>: easier to ask for forgiveness than permission.</p>
</blockquote>
<pre class="language-python"><code class="language-python"><span class="token keyword">try</span><span class="token punctuation">:</span>
<span class="token comment"># the app is on Heroku</span>
os<span class="token punctuation">.</span>environ<span class="token punctuation">[</span><span class="token string">'DYNO'</span><span class="token punctuation">]</span>
debug <span class="token operator">=</span> <span class="token boolean">False</span>
<span class="token comment"># google analytics with my tracking ID</span>
external_js<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token string">'https://codepen.io/jackdbd/pen/NgmpzR.js'</span><span class="token punctuation">)</span>
<span class="token keyword">except</span> KeyError<span class="token punctuation">:</span>
debug <span class="token operator">=</span> <span class="token boolean">True</span>
dotenv_path <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>dirname<span class="token punctuation">(</span>__file__<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">'.env'</span><span class="token punctuation">)</span>
load_dotenv<span class="token punctuation">(</span>dotenv_path<span class="token punctuation">)</span></code></pre>
<p>The world map I am displaying requires a <a href="https://plot.ly/settings/api">plotly API key</a> and a <a href="https://www.mapbox.com/studio/account/tokens/">Mapbox API access token</a>.</p>
<pre class="language-python"><code class="language-python">py<span class="token punctuation">.</span>sign_in<span class="token punctuation">(</span>os<span class="token punctuation">.</span>environ<span class="token punctuation">[</span><span class="token string">'PLOTLY_USERNAME'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> os<span class="token punctuation">.</span>environ<span class="token punctuation">[</span><span class="token string">'PLOTLY_API_KEY'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
mapbox_access_token <span class="token operator">=</span> os<span class="token punctuation">.</span>environ<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">'MAPBOX_ACCESS_TOKEN'</span><span class="token punctuation">,</span> <span class="token string">'mapbox-token'</span><span class="token punctuation">)</span></code></pre>
<p>Here is how I initialize my Dash app. I create a Flask app first because I want to use a secret key. I don’t think you can set a secret key directly when you instantiate the <code>Dash</code> class.</p>
<pre class="language-python"><code class="language-python">app_name <span class="token operator">=</span> <span class="token string">'Dash Earthquakes'</span>
server <span class="token operator">=</span> Flask<span class="token punctuation">(</span>app_name<span class="token punctuation">)</span>
server<span class="token punctuation">.</span>secret_key <span class="token operator">=</span> os<span class="token punctuation">.</span>environ<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">'SECRET_KEY'</span><span class="token punctuation">,</span> <span class="token string">'default-secret-key'</span><span class="token punctuation">)</span>
app <span class="token operator">=</span> Dash<span class="token punctuation">(</span>name<span class="token operator">=</span>app_name<span class="token punctuation">,</span> server<span class="token operator">=</span>server<span class="token punctuation">)</span></code></pre>
<h2 id="h-data" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/visualize-earthquakes-with-plotly-dash/#h-data"><span aria-hidden="true">#</span></a> Data</h2>
<p>I get the latest 4.5+ magnitude earthquakes from the <a href="https://www.usgs.gov/">USGS</a> website with a basic, synchronous <code>GET</code> request.<br />
Next time I will try to make an asynchronous request with <a href="https://www.terriblecode.com/blog/asynchronous-http-requests-in-python/">asyncio</a> or one of the following libraries: <a href="https://github.com/kennethreitz/grequests">grequests</a>, <a href="https://github.com/theelous3/asks">asks</a>, <a href="https://github.com/scribu/curio-http">curio-http</a>, <a href="https://github.com/ross/requests-futures">requests-futures</a>.</p>
<pre class="language-python"><code class="language-python">usgs <span class="token operator">=</span> <span class="token string">'https://earthquake.usgs.gov/earthquakes/'</span>
geoJsonFeed <span class="token operator">=</span> <span class="token string">'feed/v1.0/summary/4.5_month.geojson'</span>
url <span class="token operator">=</span> <span class="token string">'{}{}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>usgs<span class="token punctuation">,</span> geoJsonFeed<span class="token punctuation">)</span>
req <span class="token operator">=</span> requests<span class="token punctuation">.</span>get<span class="token punctuation">(</span>url<span class="token punctuation">)</span>
data <span class="token operator">=</span> json<span class="token punctuation">.</span>loads<span class="token punctuation">(</span>req<span class="token punctuation">.</span>text<span class="token punctuation">)</span></code></pre>
<p>Choosing the right colors for a visualization is <a href="https://jakevdp.github.io/blog/2014/10/16/how-bad-is-your-colormap/">surprisingly hard</a>, so I use <a href="https://colorbrewer2.org/">ColorBrewer</a>.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># https://colorbrewer2.org/#type=sequential&scheme=YlOrRd&n=5</span>
colorscale_magnitude <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token string">'#ffffb2'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token number">0.25</span><span class="token punctuation">,</span> <span class="token string">'#fecc5c'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'#fd8d3c'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token number">0.75</span><span class="token punctuation">,</span> <span class="token string">'#f03b20'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token string">'#bd0026'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
<span class="token comment"># https://colorbrewer2.org/#type=sequential&scheme=Greys&n=3</span>
colorscale_depth <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token string">'#f0f0f0'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'#bdbdbd'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token number">0.1</span><span class="token punctuation">,</span> <span class="token string">'#636363'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span></code></pre>
<p>Finally, some Dash code. Every Dash app requires a <code>layout</code>. The python code you write here will be converted in HTML components. I use a few functions to create portions of the dashboard. This way the layout is a bit cleaner and easier to modify.</p>
<pre class="language-python"><code class="language-python">app<span class="token punctuation">.</span>layout <span class="token operator">=</span> html<span class="token punctuation">.</span>Div<span class="token punctuation">(</span>
children<span class="token operator">=</span><span class="token punctuation">[</span>
create_header<span class="token punctuation">(</span>app_name<span class="token punctuation">)</span><span class="token punctuation">,</span>
html<span class="token punctuation">.</span>Div<span class="token punctuation">(</span>
children<span class="token operator">=</span><span class="token punctuation">[</span>
html<span class="token punctuation">.</span>Div<span class="token punctuation">(</span>create_dropdowns<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> className<span class="token operator">=</span><span class="token string">'row'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
html<span class="token punctuation">.</span>Div<span class="token punctuation">(</span>create_content<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> className<span class="token operator">=</span><span class="token string">'row'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
html<span class="token punctuation">.</span>Div<span class="token punctuation">(</span>create_description<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> className<span class="token operator">=</span><span class="token string">'row'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
html<span class="token punctuation">.</span>Div<span class="token punctuation">(</span>create_table<span class="token punctuation">(</span>dataframe<span class="token punctuation">)</span><span class="token punctuation">,</span> className<span class="token operator">=</span><span class="token string">'row'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token comment"># html.Hr(),</span>
create_footer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
className<span class="token operator">=</span><span class="token string">'container'</span><span class="token punctuation">,</span>
style<span class="token operator">=</span><span class="token punctuation">{</span><span class="token string">'font-family'</span><span class="token punctuation">:</span> theme<span class="token punctuation">[</span><span class="token string">'font-family'</span><span class="token punctuation">]</span><span class="token punctuation">}</span>
<span class="token punctuation">)</span></code></pre>
<p>Here are a couple of functions that are responsible for a portion of the UI. If you want you can check the complete code on <a href="https://github.com/jackdbd/dash-earthquakes">GitHub</a>.</p>
<p><code>create_dropdown</code> creates two <strong>dash core components</strong>. They have to be dash core components, and not simple HTML elements, because each dropdown is an <code>Input</code> for the <code>Graph</code> object (also a dash core component).</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">create_dropdowns</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
drop1 <span class="token operator">=</span> dcc<span class="token punctuation">.</span>Dropdown<span class="token punctuation">(</span>
options<span class="token operator">=</span><span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'Light'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'light'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'Dark'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'dark'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'Satellite'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'satellite'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span>
<span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'Custom'</span><span class="token punctuation">,</span>
<span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'mapbox://styles/jackdbd/cj6nva4oi14542rqr3djx1liz'</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
value<span class="token operator">=</span><span class="token string">'dark'</span><span class="token punctuation">,</span>
<span class="token builtin">id</span><span class="token operator">=</span><span class="token string">'dropdown-map-style'</span><span class="token punctuation">,</span>
className<span class="token operator">=</span><span class="token string">'three columns offset-by-one'</span>
<span class="token punctuation">)</span>
drop2 <span class="token operator">=</span> dcc<span class="token punctuation">.</span>Dropdown<span class="token punctuation">(</span>
options<span class="token operator">=</span><span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'World'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'world'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'Europe'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'europe'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'North America'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'north_america'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'South America'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'south_america'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'Africa'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'africa'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'Asia'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'asia'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'label'</span><span class="token punctuation">:</span> <span class="token string">'Oceania'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">:</span> <span class="token string">'oceania'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
value<span class="token operator">=</span><span class="token string">'world'</span><span class="token punctuation">,</span>
<span class="token builtin">id</span><span class="token operator">=</span><span class="token string">'dropdown-region'</span><span class="token punctuation">,</span>
className<span class="token operator">=</span><span class="token string">'three columns offset-by-four'</span>
<span class="token punctuation">)</span>
<span class="token keyword">return</span> <span class="token punctuation">[</span>drop1<span class="token punctuation">,</span> drop2<span class="token punctuation">]</span></code></pre>
<p><code>create_content</code> creates a <code>DIV</code> with an empty figure inside and return it. The figure will be updated when <code>_update_graph</code> is triggered (see below).</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">create_content</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
graph <span class="token operator">=</span> dcc<span class="token punctuation">.</span>Graph<span class="token punctuation">(</span><span class="token builtin">id</span><span class="token operator">=</span><span class="token string">'graph-geo'</span><span class="token punctuation">)</span>
content <span class="token operator">=</span> html<span class="token punctuation">.</span>Div<span class="token punctuation">(</span>graph<span class="token punctuation">,</span> <span class="token builtin">id</span><span class="token operator">=</span><span class="token string">'content'</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> content</code></pre>
<p>Now that you have inputs – the two dropdowns – and an output – the Graph – you can define the reactive callback <code>_update_graph</code>.</p>
<p>The way an <code>Input</code> object and an <code>Output</code> object are created is with the dash core component <code>id</code> attribute. I really like the way the relationship between inputs and output must be declared. It’s very explicit: the <code>value</code> attribute of a <code>Dropdown</code> component triggers a change in the <code>figure</code> attribute of the <code>Graph</code> component.</p>
<p><code>_update_graph</code> is rather long because every <code>Figure</code> needs a <code>layout</code> and some <code>data</code>. I have to define a bunch of parameters for the <code>layout</code> and two overlaid <code>Scattermapbox</code> for the <code>data</code>.</p>
<p>I use the underscore in front of this function to suggest that it should not be called. In fact, only changes to the dropdown values should trigger its execution.</p>
<pre class="language-python"><code class="language-python"><span class="token decorator annotation punctuation">@app<span class="token punctuation">.</span>callback</span><span class="token punctuation">(</span>
output<span class="token operator">=</span>Output<span class="token punctuation">(</span><span class="token string">'graph-geo'</span><span class="token punctuation">,</span> <span class="token string">'figure'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
inputs<span class="token operator">=</span><span class="token punctuation">[</span>Input<span class="token punctuation">(</span><span class="token string">'dropdown-map-style'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
Input<span class="token punctuation">(</span><span class="token string">'dropdown-region'</span><span class="token punctuation">,</span> <span class="token string">'value'</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">_update_graph</span><span class="token punctuation">(</span>map_style<span class="token punctuation">,</span> region<span class="token punctuation">)</span><span class="token punctuation">:</span>
dff <span class="token operator">=</span> dataframe
radius_multiplier <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">'inner'</span><span class="token punctuation">:</span> <span class="token number">1.5</span><span class="token punctuation">,</span> <span class="token string">'outer'</span><span class="token punctuation">:</span> <span class="token number">3</span><span class="token punctuation">}</span>
layout <span class="token operator">=</span> go<span class="token punctuation">.</span>Layout<span class="token punctuation">(</span>
title<span class="token operator">=</span>metadata<span class="token punctuation">[</span><span class="token string">'title'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
autosize<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span>
hovermode<span class="token operator">=</span><span class="token string">'closest'</span><span class="token punctuation">,</span>
height<span class="token operator">=</span><span class="token number">750</span><span class="token punctuation">,</span>
font<span class="token operator">=</span><span class="token builtin">dict</span><span class="token punctuation">(</span>family<span class="token operator">=</span>theme<span class="token punctuation">[</span><span class="token string">'font-family'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
margin<span class="token operator">=</span>go<span class="token punctuation">.</span>Margin<span class="token punctuation">(</span>l<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span> r<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span> t<span class="token operator">=</span><span class="token number">45</span><span class="token punctuation">,</span> b<span class="token operator">=</span><span class="token number">10</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
mapbox<span class="token operator">=</span><span class="token builtin">dict</span><span class="token punctuation">(</span>
accesstoken<span class="token operator">=</span>mapbox_access_token<span class="token punctuation">,</span>
bearing<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span>
center<span class="token operator">=</span><span class="token builtin">dict</span><span class="token punctuation">(</span>
lat<span class="token operator">=</span>regions<span class="token punctuation">[</span>region<span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'lat'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
lon<span class="token operator">=</span>regions<span class="token punctuation">[</span>region<span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'lon'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
pitch<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span>
zoom<span class="token operator">=</span>regions<span class="token punctuation">[</span>region<span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'zoom'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
style<span class="token operator">=</span>map_style<span class="token punctuation">,</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span>
data <span class="token operator">=</span> go<span class="token punctuation">.</span>Data<span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token comment"># outer circles represent magnitude</span>
go<span class="token punctuation">.</span>Scattermapbox<span class="token punctuation">(</span>
lat<span class="token operator">=</span>dff<span class="token punctuation">[</span><span class="token string">'Latitude'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
lon<span class="token operator">=</span>dff<span class="token punctuation">[</span><span class="token string">'Longitude'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
mode<span class="token operator">=</span><span class="token string">'markers'</span><span class="token punctuation">,</span>
marker<span class="token operator">=</span>go<span class="token punctuation">.</span>Marker<span class="token punctuation">(</span>
size<span class="token operator">=</span>dff<span class="token punctuation">[</span><span class="token string">'Magnitude'</span><span class="token punctuation">]</span> <span class="token operator">*</span> radius_multiplier<span class="token punctuation">[</span><span class="token string">'outer'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
colorscale<span class="token operator">=</span>colorscale_magnitude<span class="token punctuation">,</span>
color<span class="token operator">=</span>dff<span class="token punctuation">[</span><span class="token string">'Magnitude'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
opacity<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
text<span class="token operator">=</span>dff<span class="token punctuation">[</span><span class="token string">'Text'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token comment"># hoverinfo='text',</span>
showlegend<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token comment"># inner circles represent depth</span>
go<span class="token punctuation">.</span>Scattermapbox<span class="token punctuation">(</span>
lat<span class="token operator">=</span>dff<span class="token punctuation">[</span><span class="token string">'Latitude'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
lon<span class="token operator">=</span>dff<span class="token punctuation">[</span><span class="token string">'Longitude'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
mode<span class="token operator">=</span><span class="token string">'markers'</span><span class="token punctuation">,</span>
marker<span class="token operator">=</span>go<span class="token punctuation">.</span>Marker<span class="token punctuation">(</span>
size<span class="token operator">=</span>dff<span class="token punctuation">[</span><span class="token string">'Magnitude'</span><span class="token punctuation">]</span> <span class="token operator">*</span> radius_multiplier<span class="token punctuation">[</span><span class="token string">'inner'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
colorscale<span class="token operator">=</span>colorscale_depth<span class="token punctuation">,</span>
color<span class="token operator">=</span>dff<span class="token punctuation">[</span><span class="token string">'Depth'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
opacity<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token comment"># hovering behavior is already handled by outer circles</span>
hoverinfo<span class="token operator">=</span><span class="token string">'skip'</span><span class="token punctuation">,</span>
showlegend<span class="token operator">=</span><span class="token boolean">False</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span>
figure <span class="token operator">=</span> go<span class="token punctuation">.</span>Figure<span class="token punctuation">(</span>data<span class="token operator">=</span>data<span class="token punctuation">,</span> layout<span class="token operator">=</span>layout<span class="token punctuation">)</span>
<span class="token keyword">return</span> figure</code></pre>
<p>As I said at the beginning, you can create Dash apps without having to write any Javascript or CSS. The problem is that even for a very small app like this one, you will probably want to change the styling, add a small script, or maybe just include Google Analytics.</p>
<p>For example, in this app I have to display roughly 300-500 earthquakes in a table, and I use a jQuery plugin to have a nice-looking table with pagination and search functionality. I also added Font Awesome, some styling from the Dash Team and a Google font.</p>
<pre class="language-python"><code class="language-python">external_js <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token comment"># jQuery, DataTables, script to initialize DataTables</span>
<span class="token string">'https://code.jquery.com/jquery-3.2.1.slim.min.js'</span><span class="token punctuation">,</span>
<span class="token string">'//cdn.datatables.net/1.10.15/js/jquery.dataTables.min.js'</span><span class="token punctuation">,</span>
<span class="token comment"># small hack for DataTables</span>
<span class="token string">'https://codepen.io/jackdbd/pen/bROVgV.js'</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
external_css <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token comment"># dash stylesheet</span>
<span class="token string">'https://codepen.io/chriddyp/pen/bWLwgP.css'</span><span class="token punctuation">,</span>
<span class="token string">'https://fonts.googleapis.com/css?family=Raleway'</span><span class="token punctuation">,</span>
<span class="token string">'//maxcdn.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css'</span><span class="token punctuation">,</span>
<span class="token string">'//cdn.datatables.net/1.10.15/css/jquery.dataTables.min.css'</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
<span class="token keyword">for</span> js <span class="token keyword">in</span> external_js<span class="token punctuation">:</span>
app<span class="token punctuation">.</span>scripts<span class="token punctuation">.</span>append_script<span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token string">'external_url'</span><span class="token punctuation">:</span> js<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> css <span class="token keyword">in</span> external_css<span class="token punctuation">:</span>
app<span class="token punctuation">.</span>css<span class="token punctuation">.</span>append_css<span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token string">'external_url'</span><span class="token punctuation">:</span> css<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-conclusion" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/visualize-earthquakes-with-plotly-dash/#h-conclusion"><span aria-hidden="true">#</span></a> Conclusion</h2>
<p>I had a lot of fun in creating this app, and I’m sure there are many use-cases where a quick (reactive) web app is useful.<br />
I will keep using Dash for future projects. I also want to <a href="https://plot.ly/dash/plugins">write my own component</a> to practice React.js a bit.</p>
<p>I’m still a bit skeptic about the idea of creating complex layouts in Python though. Even for a small app like this, the layout seems a bit too cumbersome. Applications with a lot of styling might not be ideal as well.</p>
<p>That being said, if you want to build something relatively simple in a day or two, I think Dash is great!</p>
<p>You can find the code for the entire application <a href="https://github.com/jackdbd/dash-earthquakes">on GitHub</a></p>
https://www.giacomodebidda.com/posts/a-few-timeless-lessons-from-peopleware/A few timeless lessons from Peopleware2017-08-29T21:12:03Z2017-08-29T21:12:03Z
<p>One of the best books I’ve ever read is Peopleware, by Tom DeMarco and Timothy Lister. It’s a well known book in the software development industry, and even if it focusues more on managers than “creative” roles (designers, developers, etc), I think that anyone working in this field should read it.</p>
<p>According to Wikipedia, <a href="https://en.wikipedia.org/wiki/Peopleware">peopleware</a> can refer to anything that has to do with the role of people in the development or use of computer software and hardware systems. This includes topics as developer productivity, teamwork, group dynamics, project management, organizational factors, human interface design, and human-machine-interaction.</p>
<p>The book was published in 1987 but it’s still a must read today for a very simple reason. I’m quoting a passage from the book to highlight it:</p>
<blockquote>
<p>The major problems of our work are not so much technological as sociological in nature.</p>
</blockquote>
<h2 id="h-a-simple-formula" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-few-timeless-lessons-from-peopleware/#h-a-simple-formula"><span aria-hidden="true">#</span></a> A simple formula</h2>
<p>The entire book aims to replace the <em>manager-as-strategist</em> view with the folling simple – yet surprisingly hard to achieve – formula:</p>
<ul>
<li>get the right people</li>
<li>make them happy so they don’t want to leave</li>
<li>turn them loose</li>
</ul>
<p>People don’t change that much, so if they are not right for a job from the start, they will likely never be. DeMarco and Lister stress this fact because when it comes to hiring, rules of common sense are often suspended. They illustrate this by depicting a scenario where a hiring manager needs to hire a juggler:</p>
<blockquote>
<p>Candidate: Umm… don’t you want to see me juggle?<br /><br />
Manager: Gee, I never thought of that.</p>
</blockquote>
<p>The book suggests a few hiring strategies, like asking potential candidates a portfolio. However, I would say that it focuses more on how to keep people rather than find them.</p>
<p>Retaining employees should be a matter of utmost importance not only because finding new people requires time and resources, but because the <em>organization itself learns</em> from its people.</p>
<p>For a company, a great way to achieve a low turnover* (i.e. keeping people) is to invest heavily in personal growth and to provide a great sense of direction to its employees. This results in a good job satisfaction and a strong binding effect.</p>
<p>In the short run it might be cheaper to fire the person who needs retraining and hire someone else who already has the required skills. Most organizations do just that. The best organizations do not. They realize that retraining helps to build the mentality of permanence that results in low turnover and a strong sense of community. People tend to stay at such companies because there is a widespread sense that you are expected to stay.</p>
<blockquote>
<p>Learning is limited by an organization’s ability to keep its people. When turnover is high, learning is unlikely to stick or can’t take place at all.</p>
</blockquote>
<p>*A low turnover rate is not necessarily a good thing though. See <a href="https://www.ere.net/a-low-turnover-rate-could-mean-that-you-have-ugly-employees/">this article</a> for a different, interesting perspective.</p>
<h2 id="h-teams" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-few-timeless-lessons-from-peopleware/#h-teams"><span aria-hidden="true">#</span></a> Teams</h2>
<p>Many, many books today talk about effective teams, often citing examples from the military. They say that best teams are rather small in size (e.g. 6 - 13 people), they don’t really need a boss and are essentially self-driven. I think that Peopleware nailed it down 30 years ago:</p>
<blockquote>
<p>the structure of a team is a network, not a hierarchy</p>
</blockquote>
<p>On the best teams, different individuals provide occasional leadership, taking charge in areas where they have particular strengths. No one is the permanent leader, because that person would then cease to be a peer and the team interaction would begin to break down.</p>
<p>Today the typical team of knowledge-workers has a mix of skills, and most of the time there isn’t a single coach that mastered all of the technologies the team has to work with. Team members themselves provide most of the coaching to other team members. The authors use the expression “a basic hygienic act of peer-coaching”. In great teams this is going on all the time: team members sit down in pairs to transfer knowledge. There is always one learner and one teacher, and their roles tend to switch back and forth over time.</p>
<p>The authors found out that office spaces of top performer teams are quieter, more private, better protected from interruption. This seems to me a great point against open office spaces that are so popular today. However, DeMarco and Lister also admit that a meaningful measurement of productivity is a complex and elusive thing. A soft science they say, and it has to be performed differently in each different work sector.</p>
<p>They also describe the ideal conditions that allow a team to reach <a href="https://en.wikipedia.org/wiki/Flow_(psychology)">flow</a>, a concept introduced by the Hungarian psychologist <a href="https://www.ted.com/talks/mihaly_csikszentmihalyi_on_flow">Mihaly Csikszentmihalyi</a>. In this state of mind a person achieves a deep, nearly meditative involvement, he experiences a gentle sense of euphoria and is largely unaware of the passage of time.</p>
<p>For anyone involved in engineering, design, development, or writing, <strong>flow is a must</strong>, and only when he enters flow state his productivity is maximized.</p>
<p>If you have been in this state before, you know what I am talking about. If not, have a look at <a href="https://www.youtube.com/watch?v=LrQnnhAXLt0">this piano performance by Peter Bence</a>. Can you see how completely immersed in the music he is? He looks like he is in ecstasy.</p>
<h2 id="h-managers" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-few-timeless-lessons-from-peopleware/#h-managers"><span aria-hidden="true">#</span></a> Managers</h2>
<p>Peopleware is definitely against micromanagement and “carrot and stick” approaches.</p>
<p>Most of creative people (e.g. designers, engineers) love their work, so a manager doesn’t really have to adopt strict measures to keep these people working. In fact, he may even have to take steps to make them work less, and thus get more <em>meaningful work</em> done.</p>
<p>Leaving people “loose” doesn’t mean restraining from measuring their performance. Even if you are not sure, you should start nonetheless, because according to the <a href="https://vanderburg.org/blog/2003/02/03/gilbs-trap.html">Gilb’s Law</a> anything you need to quantify can be measured in some way that is superior to not measuring it at all.</p>
<p>The authors advocate the rise of an “invisible manager”, someone who <a href="https://www.oscarberg.net/blog/2012/05/invisible-manager/">stands somewhere behind the scenes, observing that things are alright, and acts on things which aren’t</a>. Such manager can manage a team without the team members knowing they’ve been “managed”.</p>
<blockquote>
<p>The manager’s function is not to make people work, but to make it possible for people to work.</p>
</blockquote>
<p>For a manager is essential to read and internalize the <a href="https://mysticmundane.blogspot.com/2008/03/seven-false-hopes-of-software.html">seven false hopes of software management</a>. I particularly like the number 5: Because of the backlog, you need to double productivity immediately. I struggle with this rule every time I want to start a small personal project. I have a Trello board with more than 100 items in my “Backlog” list. I know I would have fun in doing them, but I also know that I don’t have the time for doing even half of them. And when something ends up in a Backlog, it’s hard to prioritize it and/or think of it as a potential loss of my leisure time.</p>
<h2 id="h-quality" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-few-timeless-lessons-from-peopleware/#h-quality"><span aria-hidden="true">#</span></a> Quality</h2>
<p>A manager might jeopardize product quality by setting unreachable deadlines. He might think of it as a challenge for his workers to strive for excellence, while experienced workers might realize that they will need to sacrifice the quality of the product and their own job satisfaction. In fact, employees are often as eager as managers to get the job done, provided that they don’t have to compromise their standard of quality.</p>
<blockquote>
<p>People under time pressure don’t work better, they just work faster.</p>
</blockquote>
<h2 id="h-methodologies" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-few-timeless-lessons-from-peopleware/#h-methodologies"><span aria-hidden="true">#</span></a> Methodologies</h2>
<p>The authors are pretty critical of methodologies. They say that methodologies aim to achieve a <em>convergence</em> of methods, which by itself would be a good thing. There are better ways to achieve convergence of methods though:</p>
<ul>
<li>training: people do what they know how to do. If you give them all a common core of methods, they will tend to use those methods.</li>
<li>tools: if you agree on a set of workflows, designs, software libraries, etc, people will tend to stick to them.</li>
<li>peer review: active peer review mechanisms facilitate training and standardization of tools, so the promote convergence.</li>
</ul>
<h2 id="h-meetings" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-few-timeless-lessons-from-peopleware/#h-meetings"><span aria-hidden="true">#</span></a> Meetings</h2>
<p>Every other day we read articles against unnecessary meetings. Take for example this recent piece about Jeff Bezos’s <a href="https://www.businessinsider.com/amazon-jeff-bezos-two-pizza-rule-productive-meetings-2017-7?IR=T">two pizza rule for meetings</a>. I think most of these articles do not add much value to what the authors of Peopleware wrote 30 years ago: regular meetings are most likely <em>unnecessary</em>.</p>
<p>There should be a meeting only when it’s actually <em>needed</em>. There must be a real reason for <em>all the people</em> invited to think through some matter together. The purpose of the meeting is to <em>reach consensus</em>. Such a meeting is, almost by definition, an <em>ad hoc affair</em>.</p>
<blockquote>
<p>Any regular get-together is […] somewhat suspect as likely to have a ceremonial purpose rather than a focused goal of consensus.</p>
</blockquote>
<p>When people’s time is wasted in unnecessary meetings or by early overstaffing, they’ll know it. They’ll be frustrated and they’ll<br />
know why.</p>
<h2 id="h-workaholism" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-few-timeless-lessons-from-peopleware/#h-workaholism"><span aria-hidden="true">#</span></a> Workaholism</h2>
<p>I really like the fact that the DeMarco and Lister say that managers should really worry about workaholism. I think it’s very wise. A workaholic employee is a problem not only for himself, his family and collegues, but for his manager too. Sooner or later he will realize that he has got his value system wrong, and the consequences of this realization will be devastating for him.</p>
<p>I think this quote on burnout is a masterpiece:</p>
<blockquote>
<p>The realization that one has sacrificed a more important value (family, love, home, youth) for a less important value (work)<br />
is devastating. It makes the person who has unwittingly sacrificed seek revenge. He doesn’t go to the boss and explain calmly and<br />
thoughtfully that things have to change in the future. He just quits, another case of burnout. One way or the other, he’s gone.</p>
</blockquote>
<h2 id="h-conclusion" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/a-few-timeless-lessons-from-peopleware/#h-conclusion"><span aria-hidden="true">#</span></a> Conclusion</h2>
<p>Give up your tech news for a couple of days and read this book instead! It’s well written and gets to the point much better than most of the blog posts that you can find out there. It does not delve deep into group dynamics, project management or technical topics, but it can teach you timeless lessons for your career and life in general.</p>
https://www.giacomodebidda.com/posts/sklearn-pandas/sklearn-pandas2017-08-20T09:02:03Z2017-08-20T09:02:03Z
<p><a href="https://github.com/pandas-dev/sklearn-pandas">sklearn-pandas</a> is a small library that provides a bridge between <a href="https://github.com/scikit-learn/scikit-learn">scikit-learn</a>’s machine learning methods and pandas Data Frames.</p>
<p>In this blog post I will show you a simple example on how to use sklearn-pandas in a classification problem. I will use the Titanic dataset from Kaggle. You can find training set e test set <a href="https://www.kaggle.com/c/titanic/data">here</a>.</p>
<h2 id="h-imports" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/sklearn-pandas/#h-imports"><span aria-hidden="true">#</span></a> Imports</h2>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> os
<span class="token keyword">import</span> pandas <span class="token keyword">as</span> pd
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>preprocessing <span class="token keyword">import</span> LabelBinarizer<span class="token punctuation">,</span> Imputer<span class="token punctuation">,</span> LabelEncoder<span class="token punctuation">,</span> \
FunctionTransformer<span class="token punctuation">,</span> Binarizer<span class="token punctuation">,</span> StandardScaler<span class="token punctuation">,</span> MultiLabelBinarizer
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>pipeline <span class="token keyword">import</span> Pipeline
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>ensemble <span class="token keyword">import</span> RandomForestClassifier
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>model_selection <span class="token keyword">import</span> StratifiedKFold<span class="token punctuation">,</span> cross_val_score
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>metrics <span class="token keyword">import</span> accuracy_score
<span class="token keyword">from</span> sklearn_pandas <span class="token keyword">import</span> DataFrameMapper<span class="token punctuation">,</span> CategoricalImputer
here <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>abspath<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>dirname<span class="token punctuation">(</span>__file__<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-data" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/sklearn-pandas/#h-data"><span aria-hidden="true">#</span></a> Data</h2>
<p>Kaggle provides separate files for training set e test set: <code>trains.csv</code> contains class labels (0 = dead; 1 = survived), while <code>test.csv</code> does not.</p>
<p>If you want you can perform some basic EDA (Exploratory Data Analysis). Be careful of <a href="https://www.quora.com/Whats-data-leakage-in-data-science">data leakege</a> though! Don’t use test data to carry on EDA, otherwise you will be tempted to select some features or perform some operations based on what you see on the test data. Here I will concatenate training set and test set just to see the total number of samples and the missing values of the entire dataset. I will “touch” the test set only at the end, for prediction.</p>
<pre class="language-python"><code class="language-python">data_directory <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>abspath<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>here<span class="token punctuation">,</span> <span class="token string">'data'</span><span class="token punctuation">,</span> <span class="token string">'titanic'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
train_csv <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>data_directory<span class="token punctuation">,</span> <span class="token string">'train.csv'</span><span class="token punctuation">)</span>
test_csv <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>data_directory<span class="token punctuation">,</span> <span class="token string">'test.csv'</span><span class="token punctuation">)</span>
df_train <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_csv<span class="token punctuation">(</span>train_csv<span class="token punctuation">,</span> header<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span> index_col<span class="token operator">=</span><span class="token string">'PassengerId'</span><span class="token punctuation">)</span>
df_test <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_csv<span class="token punctuation">(</span>test_csv<span class="token punctuation">,</span> header<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span> index_col<span class="token operator">=</span><span class="token string">'PassengerId'</span><span class="token punctuation">)</span>
df <span class="token operator">=</span> pd<span class="token punctuation">.</span>concat<span class="token punctuation">(</span><span class="token punctuation">[</span>df_train<span class="token punctuation">,</span> df_test<span class="token punctuation">]</span><span class="token punctuation">,</span> keys<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">'train'</span><span class="token punctuation">,</span> <span class="token string">'test'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- Info ---'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>df<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- Describe ---'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>df<span class="token punctuation">.</span>describe<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- Features ---'</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> feature <span class="token keyword">in</span> <span class="token builtin">set</span><span class="token punctuation">(</span>df_train<span class="token punctuation">.</span>columns<span class="token punctuation">.</span>values<span class="token punctuation">)</span><span class="token punctuation">.</span>difference<span class="token punctuation">(</span><span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Name'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>feature<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>df<span class="token punctuation">[</span>feature<span class="token punctuation">]</span><span class="token punctuation">.</span>value_counts<span class="token punctuation">(</span>dropna<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'-'</span> <span class="token operator">*</span> <span class="token number">40</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-features" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/sklearn-pandas/#h-features"><span aria-hidden="true">#</span></a> Features</h2>
<p>When working on a machine learning problem, feature engineering is manually <a href="https://www.quora.com/What-is-feature-engineering/answer/Tomasz-Malisiewicz?srid=tdBp">designing what the input x’s should be</a>.</p>
<p>With sklearn-pandas you can use the <code>DataFrameMapper</code> class to declare transformations and variable imputations.</p>
<p><code>default=False</code> means that only the variables specified in the <code>DataFrameMapper</code> will be kept. All other variables will be discarded.</p>
<p><code>None</code> means that no transformation will be applied to that variable.</p>
<p><code>LabelBinarizer</code> converts a categorical variable into a <em>dummy variable</em> (aka <em>binary variable</em>). A dummy variable is either 1 or 0, whether a condition is met or not (in pandas categorical variables can be converted into dummy variables with the method <a href="https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html">get_dummies</a>).</p>
<p><code>Imputer</code> is a scikit-learn class that can perform NA imputation for quantitative variables, while <code>CategoricalImputer</code> is a sklearn-pandas class that works on categorical variables too. <a href="https://en.wikipedia.org/wiki/Imputation_(statistics)">Missing value imputation</a> is a broad topic, and in other languages there are entire packages dedicated to it. For example, in R you can find <a href="https://www.r-bloggers.com/imputing-missing-data-with-r-mice-package/">MICE</a> and <a href="https://github.com/IQSS/Amelia">Amelia</a>.</p>
<p>In a <code>DataFrameMapper</code> you can also provide a custom name for the transformed features – to be used instead of the automatically generated one – by specifying it as the third argument of the feature definition.</p>
<p>The difference between specifying the column selector as <code>'column'</code> (as a simple string) and <code>['column']</code> (as a list with one element is the shape of the array that is passed to the transformer. In the first case, a one dimensional array will be passed, while in the second case it will be a 2-dimensional array with one column, i.e. a column vector.</p>
<p><em>Example:</em> with a simple string <code>Imputer()</code> will discard <code>NaN</code> values for the column <code>Age</code>, and the fitting process will fail because of a mismatch of the size of this array and the other arrays in the <code>DataFrame</code>.</p>
<pre class="language-python"><code class="language-python">mapper <span class="token operator">=</span> DataFrameMapper<span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token punctuation">(</span><span class="token string">'Pclass'</span><span class="token punctuation">,</span> <span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'Sex'</span><span class="token punctuation">,</span> LabelBinarizer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Age'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>Imputer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'SibSp'</span><span class="token punctuation">,</span> <span class="token boolean">None</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token string">'alias'</span><span class="token punctuation">:</span> <span class="token string">'Some variable'</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Ticket'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>LabelBinarizer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Fare'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> Imputer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'Embarked'</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>CategoricalImputer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> MultiLabelBinarizer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span> default<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span></code></pre>
<p>Once again, here is how to use <code>DataMapper</code>:</p>
<pre class="language-python"><code class="language-python">mapper <span class="token operator">=</span> DataFrameMapper<span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Age'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>Imputer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token comment"># OK</span>
<span class="token punctuation">(</span><span class="token string">'Age'</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>Imputer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token comment"># NO!</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-pipeline" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/sklearn-pandas/#h-pipeline"><span aria-hidden="true">#</span></a> Pipeline</h2>
<p>Now that the you defined the features you want to use, you can build a scikit-learn <code>Pipeline</code>. The first step of the Pipeline is the <code>mapper</code> you have just defined. The last step is a scikit-learn <code>Estimator</code> that will run the classification. In this case I chose a <code>RandomForestClassifier</code> with a basic configuration. Between these two steps you can define additional ones. For example, you might want to <a href="https://en.wikipedia.org/wiki/Standard_score">z-normalize</a> your features with a <code>StandardScaler</code>.</p>
<pre class="language-python"><code class="language-python">pipeline <span class="token operator">=</span> Pipeline<span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token punctuation">(</span><span class="token string">'feature_mapper'</span><span class="token punctuation">,</span> mapper<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'scaler'</span><span class="token punctuation">,</span> StandardScaler<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'classifier'</span><span class="token punctuation">,</span> RandomForestClassifier<span class="token punctuation">(</span>n_estimators<span class="token operator">=</span><span class="token number">50</span><span class="token punctuation">,</span> random_state<span class="token operator">=</span>seed<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-cross-validation" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/sklearn-pandas/#h-cross-validation"><span aria-hidden="true">#</span></a> Cross validation</h2>
<p>The pipeline is ready, so you can train your model. In order to provide several estimates of the model’s accuracy you can use cross validation. scikit-learn provides the convenient function <code>cross_val_score</code> to do that, but you can also do it manually. Keep in mind that we are not touching the test set here: <code>xx_train</code> and <code>xx_test</code> are both part of the entire training set. We just split the entire training set to train it on <code>xx_train</code> and predict on <code>xx_test</code>.</p>
<pre class="language-python"><code class="language-python">x_train <span class="token operator">=</span> df_train<span class="token punctuation">[</span>df_train<span class="token punctuation">.</span>columns<span class="token punctuation">.</span>drop<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Survived'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">]</span>
y_train <span class="token operator">=</span> df_train<span class="token punctuation">[</span><span class="token string">'Survived'</span><span class="token punctuation">]</span>
<span class="token comment"># one way of computing cross-validated accuracy estimates</span>
skf <span class="token operator">=</span> StratifiedKFold<span class="token punctuation">(</span>n_splits<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">,</span> shuffle<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span> random_state<span class="token operator">=</span>seed<span class="token punctuation">)</span>
scores <span class="token operator">=</span> cross_val_score<span class="token punctuation">(</span>pipeline<span class="token punctuation">,</span> x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">,</span> cv<span class="token operator">=</span>skf<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Accuracy estimates: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>scores<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># another way of computing cross-validated accuracy estimates</span>
<span class="token keyword">for</span> i_split<span class="token punctuation">,</span> <span class="token punctuation">(</span>ii_train<span class="token punctuation">,</span> ii_test<span class="token punctuation">)</span> <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>skf<span class="token punctuation">.</span>split<span class="token punctuation">(</span>X<span class="token operator">=</span>x_train<span class="token punctuation">,</span> y<span class="token operator">=</span>y_train<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token comment"># x_train (independent variables, aka features) is a pandas DataFrame.</span>
<span class="token comment"># xx_train and xx_test are pandas dataframes</span>
xx_train <span class="token operator">=</span> x_train<span class="token punctuation">.</span>iloc<span class="token punctuation">[</span>ii_train<span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span>
xx_test <span class="token operator">=</span> x_train<span class="token punctuation">.</span>iloc<span class="token punctuation">[</span>ii_test<span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span>
<span class="token comment"># y_train (target variable) is a pandas Series.</span>
<span class="token comment"># yy_train and yy_test are numpy arrays</span>
yy_train <span class="token operator">=</span> y_train<span class="token punctuation">.</span>values<span class="token punctuation">[</span>ii_train<span class="token punctuation">]</span>
yy_test <span class="token operator">=</span> y_train<span class="token punctuation">.</span>values<span class="token punctuation">[</span>ii_test<span class="token punctuation">]</span>
model <span class="token operator">=</span> pipeline<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>X<span class="token operator">=</span>xx_train<span class="token punctuation">,</span> y<span class="token operator">=</span>yy_train<span class="token punctuation">)</span>
predictions <span class="token operator">=</span> model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>xx_test<span class="token punctuation">)</span>
score <span class="token operator">=</span> accuracy_score<span class="token punctuation">(</span>y_true<span class="token operator">=</span>yy_test<span class="token punctuation">,</span> y_pred<span class="token operator">=</span>predictions<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Accuracy of split num {}: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>i_split<span class="token punctuation">,</span> score<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># final model (retrain it on the entire train set)</span>
model <span class="token operator">=</span> pipeline<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>X<span class="token operator">=</span>x_train<span class="token punctuation">,</span> y<span class="token operator">=</span>y_train<span class="token punctuation">)</span></code></pre>
<h2 id="h-predict" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/sklearn-pandas/#h-predict"><span aria-hidden="true">#</span></a> Predict</h2>
<p>Now that the model is trained we can finally predict data that we have never seen before (i.e. the test set).</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># In this problem df_test doesn't contain the target variable 'Survived'</span>
x_test <span class="token operator">=</span> df_test
predictions <span class="token operator">=</span> model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Predictions (0 = dead, 1 = survived)'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>predictions<span class="token punctuation">)</span></code></pre>
<h2 id="h-the-entire-script" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/sklearn-pandas/#h-the-entire-script"><span aria-hidden="true">#</span></a> The entire script</h2>
<p>Here is the entire script:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> os
<span class="token keyword">import</span> pandas <span class="token keyword">as</span> pd
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>preprocessing <span class="token keyword">import</span> LabelBinarizer<span class="token punctuation">,</span> Imputer<span class="token punctuation">,</span> LabelEncoder<span class="token punctuation">,</span> \
FunctionTransformer<span class="token punctuation">,</span> Binarizer<span class="token punctuation">,</span> StandardScaler<span class="token punctuation">,</span> MultiLabelBinarizer
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>pipeline <span class="token keyword">import</span> Pipeline
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>ensemble <span class="token keyword">import</span> RandomForestClassifier
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>model_selection <span class="token keyword">import</span> StratifiedKFold<span class="token punctuation">,</span> cross_val_score
<span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>metrics <span class="token keyword">import</span> accuracy_score
<span class="token keyword">from</span> sklearn_pandas <span class="token keyword">import</span> DataFrameMapper<span class="token punctuation">,</span> CategoricalImputer
here <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>abspath<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>dirname<span class="token punctuation">(</span>__file__<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span>seed<span class="token operator">=</span><span class="token number">42</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
data_directory <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>abspath<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>here<span class="token punctuation">,</span> <span class="token string">'data'</span><span class="token punctuation">,</span> <span class="token string">'titanic'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
train_csv <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>data_directory<span class="token punctuation">,</span> <span class="token string">'train.csv'</span><span class="token punctuation">)</span>
test_csv <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>data_directory<span class="token punctuation">,</span> <span class="token string">'test.csv'</span><span class="token punctuation">)</span>
df_train <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_csv<span class="token punctuation">(</span>train_csv<span class="token punctuation">,</span> header<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span> index_col<span class="token operator">=</span><span class="token string">'PassengerId'</span><span class="token punctuation">)</span>
df_test <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_csv<span class="token punctuation">(</span>test_csv<span class="token punctuation">,</span> header<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span> index_col<span class="token operator">=</span><span class="token string">'PassengerId'</span><span class="token punctuation">)</span>
df <span class="token operator">=</span> pd<span class="token punctuation">.</span>concat<span class="token punctuation">(</span><span class="token punctuation">[</span>df_train<span class="token punctuation">,</span> df_test<span class="token punctuation">]</span><span class="token punctuation">,</span> keys<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">'train'</span><span class="token punctuation">,</span> <span class="token string">'test'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- Info ---'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>df<span class="token punctuation">.</span>info<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- Describe ---'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>df<span class="token punctuation">.</span>describe<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- Features ---'</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> feature <span class="token keyword">in</span> <span class="token builtin">set</span><span class="token punctuation">(</span>df_train<span class="token punctuation">.</span>columns<span class="token punctuation">.</span>values<span class="token punctuation">)</span><span class="token punctuation">.</span>difference<span class="token punctuation">(</span><span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Name'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>feature<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>df<span class="token punctuation">[</span>feature<span class="token punctuation">]</span><span class="token punctuation">.</span>value_counts<span class="token punctuation">(</span>dropna<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'-'</span> <span class="token operator">*</span> <span class="token number">40</span><span class="token punctuation">)</span>
mapper <span class="token operator">=</span> DataFrameMapper<span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token punctuation">(</span><span class="token string">'Pclass'</span><span class="token punctuation">,</span> <span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'Sex'</span><span class="token punctuation">,</span> LabelBinarizer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Age'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>Imputer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'SibSp'</span><span class="token punctuation">,</span> <span class="token boolean">None</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token string">'alias'</span><span class="token punctuation">:</span> <span class="token string">'Some variable'</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Ticket'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>LabelBinarizer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Fare'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> Imputer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'Embarked'</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>CategoricalImputer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> MultiLabelBinarizer<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span> default<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span>
pipeline <span class="token operator">=</span> Pipeline<span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token punctuation">(</span><span class="token string">'feature_mapper'</span><span class="token punctuation">,</span> mapper<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'scaler'</span><span class="token punctuation">,</span> StandardScaler<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">(</span><span class="token string">'classifier'</span><span class="token punctuation">,</span> RandomForestClassifier<span class="token punctuation">(</span>n_estimators<span class="token operator">=</span><span class="token number">50</span><span class="token punctuation">,</span> random_state<span class="token operator">=</span>seed<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span>
x_train <span class="token operator">=</span> df_train<span class="token punctuation">[</span>df_train<span class="token punctuation">.</span>columns<span class="token punctuation">.</span>drop<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'Survived'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">]</span>
y_train <span class="token operator">=</span> df_train<span class="token punctuation">[</span><span class="token string">'Survived'</span><span class="token punctuation">]</span>
<span class="token comment"># one way of computing cross-validated accuracy estimates</span>
skf <span class="token operator">=</span> StratifiedKFold<span class="token punctuation">(</span>n_splits<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">,</span> shuffle<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span> random_state<span class="token operator">=</span>seed<span class="token punctuation">)</span>
scores <span class="token operator">=</span> cross_val_score<span class="token punctuation">(</span>pipeline<span class="token punctuation">,</span> x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">,</span> cv<span class="token operator">=</span>skf<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Accuracy estimates: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>scores<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># another way of computing cross-validated accuracy estimates</span>
<span class="token keyword">for</span> i_split<span class="token punctuation">,</span> <span class="token punctuation">(</span>ii_train<span class="token punctuation">,</span> ii_test<span class="token punctuation">)</span> <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>skf<span class="token punctuation">.</span>split<span class="token punctuation">(</span>X<span class="token operator">=</span>x_train<span class="token punctuation">,</span> y<span class="token operator">=</span>y_train<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token comment"># x_train (independent variables, aka features) is a pandas DataFrame.</span>
<span class="token comment"># xx_train and xx_test are pandas dataframes</span>
xx_train <span class="token operator">=</span> x_train<span class="token punctuation">.</span>iloc<span class="token punctuation">[</span>ii_train<span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span>
xx_test <span class="token operator">=</span> x_train<span class="token punctuation">.</span>iloc<span class="token punctuation">[</span>ii_test<span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span>
<span class="token comment"># y_train (target variable) is a pandas Series.</span>
<span class="token comment"># yy_train and yy_test are numpy arrays</span>
yy_train <span class="token operator">=</span> y_train<span class="token punctuation">.</span>values<span class="token punctuation">[</span>ii_train<span class="token punctuation">]</span>
yy_test <span class="token operator">=</span> y_train<span class="token punctuation">.</span>values<span class="token punctuation">[</span>ii_test<span class="token punctuation">]</span>
model <span class="token operator">=</span> pipeline<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>X<span class="token operator">=</span>xx_train<span class="token punctuation">,</span> y<span class="token operator">=</span>yy_train<span class="token punctuation">)</span>
predictions <span class="token operator">=</span> model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>xx_test<span class="token punctuation">)</span>
score <span class="token operator">=</span> accuracy_score<span class="token punctuation">(</span>y_true<span class="token operator">=</span>yy_test<span class="token punctuation">,</span> y_pred<span class="token operator">=</span>predictions<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Accuracy of split num {}: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>i_split<span class="token punctuation">,</span> score<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># final model (retrain it on the entire train set)</span>
model <span class="token operator">=</span> pipeline<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>X<span class="token operator">=</span>x_train<span class="token punctuation">,</span> y<span class="token operator">=</span>y_train<span class="token punctuation">)</span>
<span class="token comment"># In this problem df_test doesn't contain the target variable 'Survived'</span>
x_test <span class="token operator">=</span> df_test
predictions <span class="token operator">=</span> model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Predictions (0 = dead, 1 = survived)'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>predictions<span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>If you run it, you should get these results with <code>seed=42</code>:</p>
<pre class="language-shell"><code class="language-shell">Accuracy estimates: <span class="token punctuation">[</span> <span class="token number">0.7979798</span> <span class="token number">0.83501684</span> <span class="token number">0.81144781</span><span class="token punctuation">]</span>
Accuracy of <span class="token function">split</span> num <span class="token number">0</span>: <span class="token number">0.797979797979798</span>
Accuracy of <span class="token function">split</span> num <span class="token number">1</span>: <span class="token number">0.835016835016835</span>
Accuracy of <span class="token function">split</span> num <span class="token number">2</span>: <span class="token number">0.8114478114478114</span>
Predictions <span class="token punctuation">(</span><span class="token number">0</span> <span class="token operator">=</span> dead, <span class="token number">1</span> <span class="token operator">=</span> survived<span class="token punctuation">)</span>
<span class="token punctuation">[</span><span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span>
<span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span>
<span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span>
<span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span>
<span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span>
<span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span>
<span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span>
<span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span>
<span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span>
<span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span>
<span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span>
<span class="token number">0</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">1</span> <span class="token number">0</span> <span class="token number">0</span> <span class="token number">0</span><span class="token punctuation">]</span></code></pre>
https://www.giacomodebidda.com/posts/quickstart-python-projects-with-invoke/Quickstart Python projects with Invoke2017-08-07T20:12:03Z2017-08-07T20:12:03Z
<p>Every time I want to try out a new Python library or develop a small weekend project I need to create a git repository and configure a virtual environment. Most of the time I forget one passage or another and I end up losing a few minutes searching on Google how to perform some trivial task. Not fun.</p>
<p>Luckily, there are several tools to automate these boring, repetivive tasks. I know a little bit of <a href="https://www.gnu.org/software/make/">make</a> and <a href="https://www.fabfile.org/">Fabric</a>, but I came across <a href="https://docs.pyinvoke.org/en/latest/index.html">Invoke</a> and I wanted to try it.</p>
<p>I will show you how to automate the creation and the initial configuration of a basic Python project with Invoke.</p>
<p>According to the documentation, Invoke is a task execution tool & library that provides a clean, high level API for unning shell commands and defining/organizing task functions from a <code>tasks.py</code> file. It’s in a very early-stage development (version <code>0.20.3</code> at the time of this writing) but the documentation is pretty good.</p>
<p>I wanted to create a series of tasks – well, one basically – that could automate the process of:</p>
<ol>
<li>create a directory and initialize a git repository inside</li>
<li>setup a virtual environment a few dependencies (pylint, flake8)</li>
<li>configure the virtual environment for Visual Studio Code</li>
<li>configure the virtual environment for PyCharm</li>
</ol>
<p>I didn’t manage to automate step 4, but the first three were fairly easy to deal with.</p>
<p><em>Note: if you want to follow along, install invoke with <code>pip install invoke</code> and create a <code>tasks.py</code> file.</em></p>
<p>Let’s start by importing invoke and defining a very simple task.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">from</span> invoke <span class="token keyword">import</span> task
<span class="token comment"># location where I keep all of my repositories</span>
REPOS <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>abspath<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>__file__<span class="token punctuation">,</span> <span class="token string">'..'</span><span class="token punctuation">,</span> <span class="token string">'..'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@task</span>
<span class="token keyword">def</span> <span class="token function">greet</span><span class="token punctuation">(</span>ctx<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Hi! I will create a new Project in {}...'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>REPOS<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>You can see that the function <code>greet</code> is decorated with the decorator <code>@task</code>, which makes it an <code>invoke</code> task. Every invoke task has a <a href="https://docs.pyinvoke.org/en/latest/api/context.html">Context</a> and can be invoked from the terminal with:</p>
<pre class="language-shell"><code class="language-shell">invoke <span class="token operator"><</span>task-name<span class="token operator">></span></code></pre>
<p>Let’s create anothere simple task, and use the Context this time.</p>
<pre class="language-python"><code class="language-python"><span class="token decorator annotation punctuation">@task</span>
<span class="token keyword">def</span> <span class="token function">message</span><span class="token punctuation">(</span>ctx<span class="token punctuation">)</span><span class="token punctuation">:</span>
msg <span class="token operator">=</span> <span class="token string">'In PyCharm, setup the virtualenv for your project in:\n'</span> \
<span class="token string">'Settings > Project > Project Interpreter > gear icon > Add Local'</span> \
<span class="token string">'\nThe virtualenv should be located at:\n'</span> \
<span class="token string">'~/.virtualenvs/<virtualenv created by pipenv>'</span>
ctx<span class="token punctuation">.</span>run<span class="token punctuation">(</span><span class="token string">'echo {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>msg<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>The Context is the primary API endpoint, and encapsulates information about the state. As you can see, with Context.run you can run shell commands.</p>
<p>You can declare tasks to be executed before and/or after a task. You can also define a help for any particular task. If present, you can read the help message by typing <code>invoke <task-name> --help</code>.</p>
<p>Here is the task that I’m currently using when I want to start a new Python project.</p>
<pre class="language-python"><code class="language-python"><span class="token decorator annotation punctuation">@task</span><span class="token punctuation">(</span>
pre<span class="token operator">=</span><span class="token punctuation">[</span>greet<span class="token punctuation">]</span><span class="token punctuation">,</span>
post<span class="token operator">=</span><span class="token punctuation">[</span>message<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token builtin">help</span><span class="token operator">=</span><span class="token punctuation">{</span>
<span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'Name of the directory to create. A git repository will be '</span>
<span class="token string">'initialized (default: my-repo)'</span><span class="token punctuation">,</span>
<span class="token string">'virtualenv'</span><span class="token punctuation">:</span> <span class="token string">'If True, use pipenv to create a Python 3.6 virtualenv '</span>
<span class="token string">'and lock the dependencies (default: False)'</span>
<span class="token punctuation">}</span>
<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">mkrepo</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> name<span class="token operator">=</span><span class="token string">'my-repo'</span><span class="token punctuation">,</span> virtualenv<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Create git repo and make Initial Commit.
Parameters
----------
ctx : Context
name : str
virtualenv : bool
Examples
--------
invoke mkrepo -n my-repo -v
"""</span>
repo_dir <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>REPOS<span class="token punctuation">,</span> name<span class="token punctuation">)</span>
os<span class="token punctuation">.</span>mkdir<span class="token punctuation">(</span>repo_dir<span class="token punctuation">)</span>
_setup_git_repo<span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> repo_dir<span class="token punctuation">)</span>
<span class="token keyword">if</span> virtualenv<span class="token punctuation">:</span>
_setup_pipenv<span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> repo_dir<span class="token punctuation">)</span></code></pre>
<p>All those underscores in front of the function’s name are just there to remember the user that he should not call those functions directly.</p>
<p>The list of available tasks can be shown with these commands:</p>
<pre class="language-shell"><code class="language-shell">invoke <span class="token parameter variable">--list</span> <span class="token comment"># or...</span>
invoke <span class="token parameter variable">-l</span></code></pre>
<pre class="language-shell"><code class="language-shell">jack@ThinkJack:~/Repos/invoke-tasks<span class="token punctuation">(</span>master<span class="token punctuation">)</span>$ invoke <span class="token parameter variable">-l</span>
Available tasks:
greet
message
mkrepo Create <span class="token function">git</span> repo and <span class="token function">make</span> Initial Commit.</code></pre>
<p>The task <code>mkrepo</code> calls a few functions. As you can see down here, these are just functions, not tasks, because I don’t want to expose them to the invoke command-line interface. They still need a Context to run shell commands, so I need to pass the Context to them.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">_setup_git_repo</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> repo_dir<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Initialize git repository and make Initial Commit.
Parameters
----------
ctx : Context
repo_dir : str
"""</span>
cmd <span class="token operator">=</span> <span class="token triple-quoted-string string">"""
cd {} ;
git init ;
echo ".idea/" > .gitignore ;
touch README.md ;
git add . ;
git commit -m "Initial Commit"
"""</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>repo_dir<span class="token punctuation">)</span>
ctx<span class="token punctuation">.</span>run<span class="token punctuation">(</span>cmd<span class="token punctuation">)</span>
os<span class="token punctuation">.</span>mkdir<span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>repo_dir<span class="token punctuation">,</span> <span class="token string">'.vscode'</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>I recently started to use <a href="https://www.giacomodebidda.com/posts/pipenv/">Pipenv</a>. Here is a function to configure a Python virtual environment for a new project.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">_setup_pipenv</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> repo_dir<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Create a python 3.6 virtual environment with pipenv, lock and commit.
Parameters
----------
ctx : Context
repo_dir : str
"""</span>
cmd <span class="token operator">=</span> <span class="token string">'cd {} ;'</span> \
<span class="token string">'pipenv --python python3.6 ;'</span> \
<span class="token string">'pipenv install --dev pylint ;'</span> \
<span class="token string">'pipenv install --dev flake8 ;'</span> \
<span class="token string">'pipenv lock'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>repo_dir<span class="token punctuation">)</span>
ctx<span class="token punctuation">.</span>run<span class="token punctuation">(</span>cmd<span class="token punctuation">)</span>
cmd <span class="token operator">=</span> <span class="token string">'cd {} ;'</span> \
<span class="token string">'git add Pipfile Pipfile.lock ;'</span> \
<span class="token string">'git commit -m "Lock dependencies"'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>repo_dir<span class="token punctuation">)</span>
ctx<span class="token punctuation">.</span>run<span class="token punctuation">(</span>cmd<span class="token punctuation">)</span>
_create_python_module<span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> repo_dir<span class="token punctuation">)</span>
_create_vscode_settings<span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> repo_dir<span class="token punctuation">)</span></code></pre>
<p>I also wanted to create a simple <code>example.py</code> file, just to save a few characters when I start writing code. That thing beginning with <code><<EOF</code> and ending with <code>EOF</code> is a <a href="https://tldp.org/LDP/abs/html/here-docs.html">Here Document</a>.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">_create_python_module</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> repo_dir<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Create a small python module.
Parameters
----------
ctx : Context
repo_dir : str
"""</span>
pymodule_path <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>repo_dir<span class="token punctuation">,</span> <span class="token string">'example.py'</span><span class="token punctuation">)</span>
<span class="token comment"># create example.py with a Here Document</span>
cmd <span class="token operator">=</span> <span class="token triple-quoted-string string">"""
cat > {pymodule_path} <<EOF
def main():
print('before breakpoint')
print('place breakpoint here')
if __name__ == '__main__':
main()
EOF
"""</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>pymodule_path<span class="token operator">=</span>pymodule_path<span class="token punctuation">)</span>
<span class="token comment"># in alternative, create a Here Document on a single line and use .format</span>
<span class="token comment"># to replace newlines and indentations</span>
<span class="token comment"># cat > {pymodule_path} <<EOF{newline}def main():{newline}{indent}print('example'){newline}EOF""".format(newline='\n', indent=' ', pymodule_path=pymodule_path)</span>
ctx<span class="token punctuation">.</span>run<span class="token punctuation">(</span>cmd<span class="token punctuation">)</span></code></pre>
<p>I use Visual Studio Code a lot, so I want the path to the Python interpreter to be configured as soon as I start a new project. This can be done by create a <code>settings.json</code> for the <strong>Workspace settings</strong>.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">_create_vscode_settings</span><span class="token punctuation">(</span>ctx<span class="token punctuation">,</span> repo_dir<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Create Workspace settings to use in Visual Studio Code.
Parameters
----------
ctx : Context
repo_dir : str
"""</span>
cmd <span class="token operator">=</span> <span class="token string">'cd {} ;'</span> \
<span class="token string">'pipenv --venv'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>repo_dir<span class="token punctuation">)</span>
result <span class="token operator">=</span> ctx<span class="token punctuation">.</span>run<span class="token punctuation">(</span>cmd<span class="token punctuation">)</span>
<span class="token comment"># last character of stdout is a newline, so we strip it out</span>
venv_path <span class="token operator">=</span> result<span class="token punctuation">.</span>stdout<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">]</span>
json_path <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>repo_dir<span class="token punctuation">,</span> <span class="token string">".vscode"</span><span class="token punctuation">,</span> <span class="token string">"settings.json"</span><span class="token punctuation">)</span>
<span class="token comment"># create settings.json with a Here Document</span>
cmd <span class="token operator">=</span> <span class="token triple-quoted-string string">"""
cat > {json_path} <<EOF
{VS_CODE_SETTINGS_HERE}
EOF
"""</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>json_path<span class="token operator">=</span>json_path<span class="token punctuation">,</span> venv_path<span class="token operator">=</span>venv_path<span class="token punctuation">)</span>
ctx<span class="token punctuation">.</span>run<span class="token punctuation">(</span>cmd<span class="token punctuation">)</span></code></pre>
<p>And replace <code>VS_CODE_SETTINGS_HERE</code> with something like this:</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
<span class="token property">"editor.rulers"</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token number">80</span><span class="token punctuation">,</span> <span class="token number">100</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token property">"python.pythonPath"</span><span class="token operator">:</span> <span class="token string">"{venv_path}"</span>
<span class="token punctuation">}</span></code></pre>
<p><em>Note: Oviously you don’t need to set the <code>editor.rules</code>, but since I use it in all of my projects I decided to include it.</em></p>
<h2 id="h-conclusion" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/quickstart-python-projects-with-invoke/#h-conclusion"><span aria-hidden="true">#</span></a> Conclusion</h2>
<p>With Invoke you can also execute shell commands with <code>sudo</code>, create <a href="https://docs.pyinvoke.org/en/latest/concepts/namespaces.html">namespaces</a> and use a <a href="https://docs.pyinvoke.org/en/latest/concepts/testing.html#use-mockcontext">MockContext</a>. I didn’t need these features this time, but I think I will try them for more complex tasks. I really liked the clean API and the easy of use of Invoke.</p>
https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/How to write a killer README2017-07-29T19:12:03Z2017-07-29T19:12:03Z
<p>A README file is the first thing that someone encounters when he finds your project. It’s a public-facing measure of your work, so it’s definitely a good idea to spend some time reading articles and guidelines about how to write it well. Other developers and your future self will thank you for a well-crafted README.</p>
<blockquote>
<p>For every hour of code you write, spend an hour writing your README.</p>
<p>— <a href="https://blog.cwrichardkim.com/how-to-get-hundreds-of-stars-on-your-github-project-345b065e20a2">Richard Kim</a></p>
</blockquote>
<p>There are many great resources about how to write a good README. Here I will summarize what I learned and add my perspective.</p>
<p>I think that most articles miss the point that for different project <em>types</em> you need to structure the README in a different way, and to include different type of information.</p>
<p>I can think about the following project types:</p>
<ul>
<li>library or tool</li>
<li>project starter or project clone</li>
<li>cookbook repository</li>
<li>list of articles or resources on a topic</li>
</ul>
<p>I will try to describe, including some examples for each category, how a good README looks like.</p>
<h2 id="h-library-or-tool" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/#h-library-or-tool"><span aria-hidden="true">#</span></a> Library or tool</h2>
<p>Stephen Whitmore, author of <a href="https://github.com/noffle/art-of-readme">The Art of README</a>, and Ken Williams, a PERL developer, say that a README is an <em>abstraction</em> for your project:</p>
<blockquote>
<p>Your documentation is complete when someone can use your module without ever having to look at its code. This is very important. This makes it possible for you to separate your module’s documented interface from its internal implementation (guts). This is good because it means that you are free to change the module’s internals as long as the interface remains the same.</p>
<p>Remember: the documentation, not the code, defines what a module does.</p>
<p>— Ken Williams</p>
</blockquote>
<p>Tom Preston Warner even suggests to <a href="https://tom.preston-werner.com/2010/08/23/readme-driven-development.html">writing the README first</a>. I have never tried, but Tom’s reasons are very well sound: writing a README first gives you clarity about what you actually want to build, and lets you avoid some pitfalls when you start writing code.</p>
<p>A developer who reads your README should be able to answer these questions:</p>
<ol>
<li>Does it solve my problem?</li>
<li>Can I use this code?</li>
<li>Can I trust this code?</li>
</ol>
<p>Let’s see how you can answer each one of these.</p>
<h4 id="h-does-it-solve-my-problem%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/#h-does-it-solve-my-problem%3F"><span aria-hidden="true">#</span></a> Does it solve my problem?</h4>
<p>Ideally, your project name should be self-explanatory, and your project description should be a one-liner. You have to showing what your project is all about. Answer this question first, so if a developer discovers that your project is not what he needs, he can quickly move on. Write an Introduction and a Usage section to answer this question.</p>
<blockquote>
<p>[The Introduction] should be a few sentences that explain exactly what the purpose of the code is, why someone would want to use it, and how it works. Answer the what, why, and the how.</p>
<p>— <a href="https://m.dotdev.co/how-to-write-a-readme-that-rocks-bc29f279611a">Eric L. Barnes</a></p>
</blockquote>
<p>Have a look at the README of <a href="https://github.com/kennethreitz/pipenv">Pipenv</a>. In half a page you can see a short description, an animated GIF, a list of features and user testimonials. In a few seconds you can understand if this is something that you may want to use or not.</p>
<p>Everyone loves animated GIFs, and I think they are far superior to screenshots when it comes to show your project in action. If it’s applicable to your project, include an animated GIF. There are several tools for creating a GIF from a screen capture. I use <a href="https://www.maketecheasier.com/record-screen-as-animated-gif-ubuntu/"><code>byzanz</code></a>.</p>
<p>Some big projects heavily rely on nice screenshots and GIFs. For example, <a href="https://github.com/apache/incubator-superset">Superset</a> includes a ton of screenshots to make you want to try it out, and add a link to the <a href="https://superset.incubator.apache.org/">extensive documentation</a> for installation and configuration.</p>
<p>I don’t think that a list of user testimonials is that important, but you should definitely include a list of features to give other developers a clear picture of what your project does.</p>
<h4 id="h-can-i-use-this-code%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/#h-can-i-use-this-code%3F"><span aria-hidden="true">#</span></a> Can I use this code?</h4>
<p>You should include a section for the installation/configuration of your project and a section for some common usage. Be clear about your project requirements and dependencies. Does your project work only on Ubuntu? Have you tried on Windows or on a Mac?</p>
<p>You should write which license your project is using. You can also omit this information in the README and include a LICENSE file instead, but adding the license type (not the entire text!) to the README doesn’t hurt. If you have a non-permissive license, stick it at the very top.</p>
<h4 id="h-can-i-trust-this-code%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/#h-can-i-trust-this-code%3F"><span aria-hidden="true">#</span></a> Can I trust this code?</h4>
<p>The first thing I look when I find a new project is the date of the last commit. This gives me an idea whether the project is still maintained or not.</p>
<p>Build badges and Continuous Integration can help you if you want to gain more trust from other developers. If your project becomes popular, you can also add a badge that shows the number of downloads per month. This could give you some social validation and persuade other developers to adopt your project. Badges look cool and you might overdo here. For each badge, consider: “what real value is this badge providing to the typical viewer of this README?”</p>
<p>A developer might have additional questions about your project. Don’t forget to include some contact details (email, Twitter).</p>
<h4 id="h-additional-sections" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/#h-additional-sections"><span aria-hidden="true">#</span></a> Additional sections</h4>
<p>If you are aware of similar projects you can list them and maybe add a brief comparison with your project. For example, have a look at <a href="https://github.com/audreyr/cookiecutter">Cookiecutter</a>. I don’t think it makes sense to add a very detailed comparison with other projects. Maybe you can write an article on your blog and link it in the README.</p>
<p>If your project grows, you could add a FAQ section and a How to Contribute section.</p>
<p>What about the changelog? I think that the information in the changelog is too detailed and not of much use for someone who sees your project for the first time. If we accept a README as an abstraction, it doesn’t really make sense to include a changelog in it. I think it’s better to have the changelog in a different file. Also, keep in mind that if your project grows the changelog will take too much space in the README.</p>
<h2 id="h-project-starter-or-project-clone" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/#h-project-starter-or-project-clone"><span aria-hidden="true">#</span></a> Project starter or project clone</h2>
<p>The README for a project starter is very similar to the one of a library.</p>
<p>When I find a project starter I look for these things:</p>
<ul>
<li>Screenshots / GIF / Live demo: I want to see the project in action. Don’t underestimate even a simple screenshot! It’s a great driver for motivation. I know that if I take the time to install this project I will get a fully functional project.</li>
<li>Installation / Quickstart: the documentation here must be concise. I don’t like to spend hours just to decide if I want to try it out or not.</li>
<li>Features: does it have everything I want? Is there anything else that I can replace or remove? How can I do it?</li>
<li>References: I want to know why the project starter is structured this way. It would probably be too long to write that in the README, but you can add a list of articles that I can read (e.g. maybe you decided to follow <a href="https://12factor.net/">the twelve factors</a>).</li>
<li>FAQs: if the project is big enough, I think it makes sense to add a section for the most common questions a developer might have about the project.</li>
</ul>
<p>A good example of a project starter is <a href="https://github.com/pydanny/cookiecutter-django">Cookiecutter Django</a>, even if it lacks a screenshot of the web app. I also like <a href="https://github.com/sloria/cookiecutter-flask">Cookiecutter Flask</a>, which is a lot less verbose and shows a nice screenshot of the project.</p>
<p>In a project clone, a screenshot / GIF is even more important. Have a look at a Hacker News clone website built with Vue.js. See the difference between the <a href="https://github.com/vuejs/vue-hackernews">boring, older version</a>, and the <a href="https://github.com/vuejs/vue-hackernews-2.0">shiny, more recent version</a>. The older version has a brief set of instructions how to install it, run it, and includes a link to the Live demo. The newer version adds a nice screenshot, a list of features and an architecture overview.</p>
<h2 id="h-cookbook-repository" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/#h-cookbook-repository"><span aria-hidden="true">#</span></a> Cookbook repository</h2>
<p>With “Cookbook repository” I mean a repository where you have a set of small, standalone modules. You can call them recipes. Each recipe is independent of the other ones and performs a specific task, solves a particular problem or shows how to use a particular library across different scenarios.</p>
<p>A good example of cookbook repository is <a href="https://github.com/stanfordjournalism/search-script-scrape">Search-Script-Scrape: 101 webscraping and research tasks for the data journalist</a>. Each script focuses on a single task and can be used independently: you run it and it prints the answer you are looking for.</p>
<p>Another good example is <a href="https://github.com/eriklindernoren/ML-From-Scratch">Machine Learning From Scratch</a>, which contains bare bones Python implementations of some of the fundamental Machine Learning models and algorithms.</p>
<p>The most important things in a cookbook repository are:</p>
<ul>
<li>Table of Contents: if the list of recipes is rather long and can be subdivided into smaller topics, create sections (e.g. Supervised Learning, Unsupervised Learning).</li>
<li>Reproducibility: I should be able to run the scripts without any issue. The dependencies are clearly stated and there is a section to explain how to run them. If a script relies on external data, there is a link to the original dataset.</li>
<li>Recipes are task-specific.</li>
<li>Recipes are not too big.</li>
<li>References: there are links to articles/resources to delve deeper into a specific topic (e.g. a script about how to use <a href="https://lxml.de/">lxml</a> to perform web scraping on a particular website could have a short README with a list of articles about lxml and a screenshot of the target website).</li>
</ul>
<p>Keep in mind that on GitHub you can define a README in each directory, so you can include the necessary information where it is more relevant.</p>
<pre class="language-shell"><code class="language-shell">├── README.md
├── sectionA
│ ├── README.md
│ ├── recipeA1.py
│ ├── recipeA2.py
│ └── recipeA3.py
├── sectionB
│ ├── README.md
│ ├── recipeB1.py
│ └── recipeB2.py
└── requirements.txt</code></pre>
<h2 id="h-list-of-articles-or-resources-on-a-topic" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/#h-list-of-articles-or-resources-on-a-topic"><span aria-hidden="true">#</span></a> List of articles or resources on a topic</h2>
<p>Sometimes a repository is just a curated list of resources: a list of links and short descriptions, and basically no code. With this category, I’m referring to all the repositories created with <a href="https://github.com/sindresorhus/awesome">Awesome</a>. Using a repository for a curated list of resources is a great idea, because this way if you care about a topic, you can easily share your knowledge with other developers. Everyone can contribute by adding links to the articles that he finds interesting.</p>
<p>In this kind of repository, the README <em>is</em> the repository, and most of what is valid for the other types of repositories is just not applicable. However, here is what I would like to find:</p>
<ul>
<li>Table of Content: especially if a topic is broad enough, there should be several well-defined sections. This is a must for this kind of project. Don’t write a single, unordered list of hundreds of articles. Use sections.</li>
<li>How to contribute: if you want to gather all the good resources about a subject, make it easy for people interested in the subject to add the links to the articles that they found interesting.</li>
<li>A basic code of conduct: if the project is very popular, probably it’s a good idea to have one.</li>
</ul>
<p>A great example of a curated list of resources is <a href="https://github.com/sindresorhus/awesome-nodejs">Awesome Node.js</a>.<br />
Other examples are <a href="https://github.com/eyeseast/awesome-journalism">awesome-journalism</a>, <a href="https://github.com/vinta/awesome-python">awesome-python</a>, <a href="https://github.com/braydie/HowToBeAProgrammer">HowToBeAProgrammer</a>, <a href="https://github.com/wearehive/project-guidelines">project-guidelines</a>, and of course <a href="https://github.com/matiassingers/awesome-readme">Awesome README</a>.</p>
<h2 id="h-resources" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/how-to-write-a-killer-readme/#h-resources"><span aria-hidden="true">#</span></a> Resources</h2>
<p>I wrote this article after reading several articles on how to write a great README. Here are the ones that you <em>have to</em> read:</p>
<ul>
<li><a href="https://github.com/noffle/art-of-readme">The Art of README</a></li>
<li><a href="https://tom.preston-werner.com/2010/08/23/readme-driven-development.html">README driven development</a></li>
<li><a href="https://m.dotdev.co/how-to-write-a-readme-that-rocks-bc29f279611a">How to write a README that rocks</a></li>
<li><a href="https://blog.cwrichardkim.com/how-to-get-hundreds-of-stars-on-your-github-project-345b065e20a2">How To Get Thousands of Stars on Your Github Project</a></li>
<li><a href="https://changelog.com/posts/top-ten-reasons-why-i-wont-use-your-open-source-project">Top ten reasons why I won’t use your open source project</a></li>
</ul>
https://www.giacomodebidda.com/posts/pipenv/Pipenv2017-07-28T23:12:03Z2017-07-28T23:12:03Z
<p>When it comes to manage environments and dependencies in Python projects, a very popular combination includes <a href="https://www.giacomodebidda.com/virtual-environments-with-virtualenvwrapper/">Virtualenvwrapper</a> to manage environments, and pip to install/remove Python packages. Of course, every time you start working on a project you need to remember to activate the virtual environment of that particular project. For example, with Virtualenvwrapper you would use <code>workon <your-python-project></code>.</p>
<p>A few weeks ago I found out about <a href="https://docs.pipenv.org/en/latest/">Pipenv</a>, an experimental project by Kenneth Reitz. While I was reading the documentation I couldn’t help but notice how much Pipenv resembles the Javascript package manager <a href="https://yarnpkg.com/lang/en/">yarn</a>. With yarn — or <a href="https://en.wikipedia.org/wiki/Npm_(software)">npm</a> — you have a single tool that you can use to install/remove packages <strong>and</strong> create/manage environments.</p>
<p>In this post I will show you how to use Pipenv in two scenarios: when starting a new project, and when installing the dependencies of a pre-existing project.</p>
<p>If you want to follow along, start with installing Pipenv <em>globally</em>:</p>
<pre class="language-shell"><code class="language-shell">pip <span class="token function">install</span> pipenv</code></pre>
<h2 id="h-starting-a-new-python-project" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/pipenv/#h-starting-a-new-python-project"><span aria-hidden="true">#</span></a> Starting a new Python project</h2>
<p>Create a dummy repository and cd into it:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">mkdir</span> my-dummy-repo
<span class="token builtin class-name">cd</span> my-dummy-repo</code></pre>
<p>Create a Python 2 virtual environment:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token parameter variable">--two</span></code></pre>
<p>You should see this output in your terminal:</p>
<pre class="language-shell"><code class="language-shell">Creating a Pipfile <span class="token keyword">for</span> this project<span class="token punctuation">..</span>.
Creating a virtualenv <span class="token keyword">for</span> this project<span class="token punctuation">..</span>.
Running virtualenv with interpreter /usr/bin/python2
New python executable <span class="token keyword">in</span> /home/jack/.virtualenvs/my-dummy-repo-zmrIuv74/bin/python2
Also creating executable <span class="token keyword">in</span> /home/jack/.virtualenvs/my-dummy-repo-zmrIuv74/bin/python
Installing setuptools, pip, wheel<span class="token punctuation">..</span>.done.
Virtualenv location: /home/jack/.virtualenvs/my-dummy-repo-zmrIuv74</code></pre>
<p>The command you have just entered creates a new virtual environment and a <code>Pipfile</code>, a replacement of <code>requirements.txt</code>. In a <code>Pipfile</code>, all Python dependencies are declared with a <a href="https://github.com/toml-lang/toml">TOML</a> syntax. To be honest, I don’t particularly like the TOML syntax, and I find the <code>package.json</code> file (generated by npm or yarn) much more readable. Anyway, here is how the <code>Pipfile</code> looks at this point:</p>
<pre class="language-toml"><code class="language-toml"><span class="token punctuation">[</span><span class="token punctuation">[</span><span class="token table class-name">source</span><span class="token punctuation">]</span><span class="token punctuation">]</span>
<span class="token key property">url</span> <span class="token punctuation">=</span> <span class="token string">"https://pypi.python.org/simple"</span>
<span class="token key property">verify_ssl</span> <span class="token punctuation">=</span> <span class="token boolean">true</span></code></pre>
<p>Let’s say that we didn’t really want to create a Python 2 environment, but a Python 3.6 environment. No big deal, with Pipenv we can remove the virtual environment with:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token parameter variable">--rm</span></code></pre>
<p>and create a Python 3.6 environment with:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token parameter variable">--python</span> python3.6</code></pre>
<p>If you take a look at the <code>Pipfile</code>, you will see that nothing has changed. This is perfectly ok. We have just removed a virtual environment and created a new one. We didn’t change any dependency.</p>
<pre class="language-toml"><code class="language-toml"><span class="token punctuation">[</span><span class="token punctuation">[</span><span class="token table class-name">source</span><span class="token punctuation">]</span><span class="token punctuation">]</span>
<span class="token key property">url</span> <span class="token punctuation">=</span> <span class="token string">"https://pypi.python.org/simple"</span>
<span class="token key property">verify_ssl</span> <span class="token punctuation">=</span> <span class="token boolean">true</span></code></pre>
<p>Bu where is this virtual environment?</p>
<p>Pipenv creates all virtual environments at the location specified by the environment variable <code>WORKON_HOME</code>, just as virtualenvwrapper does. You can check where the virtual environment is by typing:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token parameter variable">--venv</span>
<span class="token comment"># or...</span>
<span class="token builtin class-name">echo</span> <span class="token variable">$VIRTUAL_ENV</span>
<span class="token comment"># or...</span>
pip <span class="token parameter variable">-V</span></code></pre>
<p>In my case <code>WORKON_HOME</code> is <code>/home/jack/.virtualenvs/</code>, that’s why the output of <code>pipenv --venv</code> is <code>/home/jack/.virtualenvs/my-dummy-repo-zmrIuv74</code>.</p>
<p><em>Note:</em> you can also activate, deactivate, remove this virtual environment with the same commands you would use for Virtualenvwrapper: <code>workon <your-virtual-enviroment></code>, <code>deactivate</code>, <code>rmvirtualenv <your-virtual-enviroment></code>.</p>
<p>Let’s install some packages.</p>
<p>Let’s say that for your project you need pandas. You can install the latest version with:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token function">install</span> pandas</code></pre>
<p>Of course you can also choose a specific version when installing a dependency:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token function">install</span> <span class="token assign-left variable">requests</span><span class="token operator">==</span><span class="token number">2.13</span>.0</code></pre>
<p>Finally, let’s install a development dependency:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token function">install</span> ddt <span class="token parameter variable">--dev</span></code></pre>
<p>If you now type <code>pip list</code> in the terminal, you will see:</p>
<pre class="language-shell"><code class="language-shell">ddt <span class="token punctuation">(</span><span class="token number">1.1</span>.1<span class="token punctuation">)</span>
numpy <span class="token punctuation">(</span><span class="token number">1.13</span>.1<span class="token punctuation">)</span>
pandas <span class="token punctuation">(</span><span class="token number">0.20</span>.3<span class="token punctuation">)</span>
pip <span class="token punctuation">(</span><span class="token number">9.0</span>.1<span class="token punctuation">)</span>
python-dateutil <span class="token punctuation">(</span><span class="token number">2.6</span>.1<span class="token punctuation">)</span>
pytz <span class="token punctuation">(</span><span class="token number">2017.2</span><span class="token punctuation">)</span>
requests <span class="token punctuation">(</span><span class="token number">2.13</span>.0<span class="token punctuation">)</span>
setuptools <span class="token punctuation">(</span><span class="token number">36.2</span>.4<span class="token punctuation">)</span>
six <span class="token punctuation">(</span><span class="token number">1.10</span>.0<span class="token punctuation">)</span>
wheel <span class="token punctuation">(</span><span class="token number">0.30</span>.0a0<span class="token punctuation">)</span></code></pre>
<p>Your <code>Pipfile</code> will look like this:</p>
<pre class="language-toml"><code class="language-toml"><span class="token punctuation">[</span><span class="token punctuation">[</span><span class="token table class-name">source</span><span class="token punctuation">]</span><span class="token punctuation">]</span>
<span class="token key property">url</span> <span class="token punctuation">=</span> <span class="token string">"https://pypi.python.org/simple"</span>
<span class="token key property">verify_ssl</span> <span class="token punctuation">=</span> <span class="token boolean">true</span>
<span class="token punctuation">[</span><span class="token table class-name">dev-packages</span><span class="token punctuation">]</span>
<span class="token key property">ddt</span> <span class="token punctuation">=</span> <span class="token string">"*"</span>
<span class="token punctuation">[</span><span class="token table class-name">packages</span><span class="token punctuation">]</span>
<span class="token key property">pandas</span> <span class="token punctuation">=</span> <span class="token string">"*"</span>
<span class="token key property">requests</span> <span class="token punctuation">=</span> <span class="token string">"==2.13.0"</span></code></pre>
<p>Now your dependencies are set on your system, but you still need to “freeze” them for everyone else. With the usual pip workflow, you could use <code>pip freeze > requirements.txt</code>. With Pipenv you have to type a lot less:</p>
<pre class="language-shell"><code class="language-shell">pipenv lock</code></pre>
<p>which gives you this output:</p>
<pre class="language-shell"><code class="language-shell">Locking <span class="token punctuation">[</span>dev-packages<span class="token punctuation">]</span> dependencies<span class="token punctuation">..</span>.
Locking <span class="token punctuation">[</span>packages<span class="token punctuation">]</span> dependencies<span class="token punctuation">..</span>.
Updated Pipfile.lock<span class="token operator">!</span></code></pre>
<p>The first time I used pipenv, it felt a bit weird. I am familiar with yarn, and with yarn I don’t have to run two commands to install/lock the dependencies. I would just need to run <code>yarn add d3</code>, or <code>yarn add webpack --dev</code>. In fact, someone opened an <a href="https://github.com/kennethreitz/pipenv/issues/404">issue on GitHub</a> and asked Kenneth why the need of manually invoking <code>pipenv lock</code> after <code>pipenv install</code> (or <code>pipenv uninstall</code>). Pipenv is still in an early stage, and Kenneth commented that he decided to have two commands because <code>pipenv lock</code> is quite slow (on my computer, locking the requirements for this dummy project took roughly 10 seconds).</p>
<p>It should be noted that you can easily combine install and lock in a single command:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token function">install</span> pandas <span class="token parameter variable">--lock</span></code></pre>
<p>Now that alll dependencies are locked, let’s have a look at the <code>Pipfile.lock</code> file (basically a json file):</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
<span class="token property">"_meta"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"hash"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"sha256"</span><span class="token operator">:</span>
<span class="token string">"a770116c27db7e68723c431e7a00bc50e32d6ae6c719998e01a332da646757b6"</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"requires"</span><span class="token operator">:</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"sources"</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token property">"url"</span><span class="token operator">:</span> <span class="token string">"https://pypi.python.org/simple"</span><span class="token punctuation">,</span>
<span class="token property">"verify_ssl"</span><span class="token operator">:</span> <span class="token boolean">true</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"default"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"numpy"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"==1.13.1"</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"pandas"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"==0.20.3"</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"python-dateutil"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"==2.6.1"</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"pytz"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"==2017.2"</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"requests"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"==2.13.0"</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"six"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"==1.10.0"</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"develop"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"ddt"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"==1.1.1"</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>Another handy command is <code>pipenv update</code>, which updates pip to the latest version, uninstalls all packages, and finally re-installs all packages declared in the [packages] section of the <code>Pipfile</code> (for each package, it installs the latest compatible version).</p>
<pre class="language-shell"><code class="language-shell">Updating all dependencies from Pipfile<span class="token punctuation">..</span>.
Found <span class="token number">6</span> installed package<span class="token punctuation">(</span>s<span class="token punctuation">)</span>, purging<span class="token punctuation">..</span>.
Uninstalling ddt-1.1.1:
Successfully uninstalled ddt-1.1.1
Uninstalling numpy-1.13.1:
Successfully uninstalled numpy-1.13.1
Uninstalling pandas-0.20.3:
Successfully uninstalled pandas-0.20.3
Uninstalling python-dateutil-2.6.1:
Successfully uninstalled python-dateutil-2.6.1
Uninstalling pytz-2017.2:
Successfully uninstalled pytz-2017.2
Uninstalling requests-2.13.0:
Successfully uninstalled requests-2.13.0
Environment now purged and fresh<span class="token operator">!</span>
Pipfile found at /home/jack/Repos/my-dummy-repo/Pipfile. Considering this to be the project home.
Installing dependencies from Pipfile.lock<span class="token punctuation">..</span>.
Requirement already satisfied: <span class="token assign-left variable">six</span><span class="token operator">==</span><span class="token number">1.10</span>.0 <span class="token keyword">in</span> /home/jack/.virtualenvs/my-dummy-repo-zmrIuv74/lib/python3.6/site-packages
Collecting <span class="token assign-left variable">pytz</span><span class="token operator">==</span><span class="token number">2017.2</span>
Using cached pytz-2017.2-py2.py3-none-any.whl
Collecting python-dateutil<span class="token operator">==</span><span class="token number">2.6</span>.1
Using cached python_dateutil-2.6.1-py2.py3-none-any.whl
Collecting <span class="token assign-left variable">requests</span><span class="token operator">==</span><span class="token number">2.13</span>.0
Using cached requests-2.13.0-py2.py3-none-any.whl
Collecting <span class="token assign-left variable">numpy</span><span class="token operator">==</span><span class="token number">1.13</span>.1
Using cached numpy-1.13.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting <span class="token assign-left variable">pandas</span><span class="token operator">==</span><span class="token number">0.20</span>.3
Using cached pandas-0.20.3-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: pytz, python-dateutil, requests, numpy, pandas
Successfully installed numpy-1.13.1 pandas-0.20.3 python-dateutil-2.6.1 pytz-2017.2 requests-2.13.0
⠸
All dependencies are now up-to-date<span class="token operator">!</span></code></pre>
<p><em>Note:</em> you <strong>should</strong> put <code>Pipfile</code> and <code>Pipfile.lock</code> under version control.</p>
<h2 id="h-installing-dependencies-of-a-pre-existing-project" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/pipenv/#h-installing-dependencies-of-a-pre-existing-project"><span aria-hidden="true">#</span></a> Installing dependencies of a pre-existing project</h2>
<p>Some weeks ago I discovered <a href="https://github.com/plotly/dash">Plotly Dash</a>, a cool project based on React.js and the Plotly API. Dash allows you to develop reactive web apps and interactive dashboards in Python. I felt super-exited about it, because I have been looking for the Python equivalent of the <a href="https://shiny.rstudio.com/">R Shiny package</a> for a long time. I also wanted to try Pipenv in a real project, so I decided to develop a small project with Dash and to use Pipenv to manage the virtual environment and the project dependencies.</p>
<p>You can clone my Dash project via SSH:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> clone git@github.com:jackdbd/dash-earthquakes.git</code></pre>
<p>Then go to the project directory:</p>
<pre class="language-shell"><code class="language-shell"><span class="token builtin class-name">cd</span> dash-earthquakes</code></pre>
<p>Now you have to create the virtual environment. I developed this project in a Python 3.6 enviroment, so it makes sense for you to use the same setup:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token parameter variable">--python</span> python3.6</code></pre>
<p>Install all dependencies. You could use the <code>--dev</code> flag to include all development dependencies, but in this small project I don’t have them:</p>
<pre class="language-shell"><code class="language-shell">pipenv <span class="token function">install</span></code></pre>
<p>You can also update and lock the dependencies once again:</p>
<pre class="language-shell"><code class="language-shell">pipenv update
pipenv lock</code></pre>
<p>Finally, you can have a look at the packages available in this virtual environment by typing:</p>
<pre class="language-shell"><code class="language-shell">pip list</code></pre>
<h2 id="h-pipenv-and-travis-ci" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/pipenv/#h-pipenv-and-travis-ci"><span aria-hidden="true">#</span></a> Pipenv and Travis CI</h2>
<p>I use Pipenv for <a href="https://github.com/jackdbd/dash-earthquakes">dash-earthquakes</a>, a very simple dashboard that displays the most recent eathquakes in the world (if you are interested in how I created this application, I wrote an <a href="https://www.giacomodebidda.com/posts/visualize-earthquakes-with-plotly-dash/">article</a> about it).</p>
<p>I wanted to setup <a href="https://travis-ci.org/">Travis</a> for this project, but it took me a while to figure out how to use pipenv in this environment. A possible solution comes from the <a href="https://docs.pipenv.org/en/latest/advanced.html">pipenv advanced documentation</a>: use a <code>Makefile</code> to install pipenv and run the tests.</p>
<p>Here is how the <code>Makefile</code> looks like.</p>
<pre><code>help:
@echo ' init'
@echo ' install pipenv and all project dependencies'
@echo ' test'
@echo ' run all tests'
init:
@echo 'Install python dependencies'
pip install pipenv
pipenv install
test:
@echo 'Run all tests'
pipenv run py.test tests
</code></pre>
<p>And here is a basic <code>.travis.yml</code> file.</p>
<pre class="language-yaml"><code class="language-yaml"><span class="token key atrule">language</span><span class="token punctuation">:</span> python
<span class="token key atrule">python</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token number">3.5</span>
<span class="token punctuation">-</span> <span class="token number">3.6</span>
<span class="token key atrule">install</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> make init
<span class="token key atrule">script</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> make test</code></pre>
<h2 id="h-pipen-and-pyup" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/pipenv/#h-pipen-and-pyup"><span aria-hidden="true">#</span></a> Pipen and PyUp</h2>
<p>I usually keep track of my Python dependencies with the <a href="https://pyup.io/">PyUp</a> bot. Unfortunately, it seems that PyUp does not support pipenv at the moment.</p>
<p>Of course you can activate the virtual environment and create a <code>requirements.txt</code> with <code>pip freeze > requirements.txt</code>, but you would still need to check that file in, to make it available for PyUp. That is confusing, because pipenv don’t use requirement files.</p>
https://www.giacomodebidda.com/posts/my-personal-git-memo/My personal Git Memo2017-07-20T16:30:03Z2017-07-20T16:30:03Z
<p>Here is an unordered list of git commands, configurations, tricks, articles, gotchas that I don’t want to forget.</p>
<ul>
<li>Find out all commit hashes of the git submodules your branch is pointing at</li>
<li>Find commit by message string</li>
<li>Show all commits from an author, in a specified date range</li>
<li>Fix <code>.git/index.lock</code> error</li>
<li>Permanently remove a file from a repo</li>
<li>Show branch name in Linux terminal</li>
<li>Compare a file across 2 branches</li>
<li>List all commits for a specific file</li>
<li>Configure git aliases</li>
<li>Commit only part of a file</li>
<li>Git hooks</li>
<li>Discard changes to git submodules</li>
</ul>
<hr />
<h2 id="h-find-out-all-commit-hashes-of-the-git-submodules-your-branch-is-pointing-at" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-find-out-all-commit-hashes-of-the-git-submodules-your-branch-is-pointing-at"><span aria-hidden="true">#</span></a> Find out all commit hashes of the git submodules your branch is pointing at</h2>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> ls-tree <span class="token operator"><</span>branch you are currently on<span class="token operator">></span>:<span class="token operator"><</span>path to directory containing the submodules<span class="token operator">></span></code></pre>
<p><em>Example</em></p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> ls-tree master:external</code></pre>
<p>Reference <a href="https://stackoverflow.com/a/5033973">here</a></p>
<h2 id="h-find-commit-by-message-string" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-find-commit-by-message-string"><span aria-hidden="true">#</span></a> Find commit by message string</h2>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> log <span class="token parameter variable">--all</span> <span class="token parameter variable">--grep</span><span class="token operator">=</span><span class="token string">"your commit message here (or part of it)"</span></code></pre>
<p>Use <code>--all</code> to search across all branches. Don’t use it if you want to restrict the search to the branch you are currently on.</p>
<p>Reference <a href="https://stackoverflow.com/a/7124949">here</a></p>
<h2 id="h-show-all-commits-from-an-author-in-a-specified-date-range" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-show-all-commits-from-an-author-in-a-specified-date-range"><span aria-hidden="true">#</span></a> Show all commits from an author, in a specified date range</h2>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> log <span class="token parameter variable">--pretty</span><span class="token operator">=</span>format:<span class="token string">"%ad - %an: %s"</span> <span class="token parameter variable">--after</span><span class="token operator">=</span><span class="token string">"2016-01-31"</span> <span class="token parameter variable">--until</span><span class="token operator">=</span><span class="token string">"2017-12-01"</span> <span class="token parameter variable">--author</span><span class="token operator">=</span><span class="token string">"John Doe"</span></code></pre>
<p>Reference <a href="https://stackoverflow.com/a/42795304/3036129">here</a> and <a href="https://git-scm.com/book/it/v2/Git-Basics-Viewing-the-Commit-History">here</a></p>
<h2 id="h-fix-gitindexlock" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-fix-gitindexlock"><span aria-hidden="true">#</span></a> Fix <code>.git/index.lock</code></h2>
<p>I get this from time to time, and I still haven’t figured out the reason why it occurs.<br />
You just have to remove the <code>index.lock</code> file (not the index!).</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">rm</span> .git/index.lock</code></pre>
<p>Reference <a href="https://thoughtbot.com/blog/how-to-fix-rm-f-git-index">here</a></p>
<h2 id="h-permanently-remove-a-file-from-a-repo" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-permanently-remove-a-file-from-a-repo"><span aria-hidden="true">#</span></a> Permanently remove a file from a repo</h2>
<p>You should really think twice before performing this operation. Also, keep in mind that this command will override git history.</p>
<p>Anyway, there are some real use cases when this command is useful.</p>
<p>Let’s say that you commit a large file by mistake. You have some stuff to commit and you run a convenient <code>git add .</code> to stash all your changes. Then you forget to review your changes and those changes end up in the commit history. At this point you are still on a local branch, so you can still save the day by removing the file, stashing your changes once again and running a <code>git commit --amend</code>. If you realize your mistake some commits later, but you have not yet pushed them, you can also <a href="https://www.giacomodebidda.com/squashing-git-commits/">squash your commits</a>. But if you have already pushed to your remote repository you are out of luck, and that’s when this command could help you.</p>
<p>Let’s suppose that at some point in time you created a <code>README.md</code> for your project, but for some esoteric reason you want to remove it from your project. You can permanently remove your <code>README.md</code> file and make it disappear from your git history with the following command.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> filter-branch --tree-filter <span class="token string">'rm -rf README.md'</span> HEAD</code></pre>
<p>If you now run <code>git status</code>, you will notice that your local branch and your remote branch differ. That’s because your remote branch still has a <code>README.md</code> in its history.</p>
<pre class="language-shell"><code class="language-shell">jack@ThinkJack:~/Repos/d3-visualizations<span class="token punctuation">(</span>master<span class="token punctuation">)</span>$ <span class="token function">git</span> status
On branch master
Your branch and <span class="token string">'origin/master'</span> have diverged,
and have <span class="token number">26</span> and <span class="token number">26</span> different commits each, respectively.
<span class="token punctuation">(</span>use <span class="token string">"git pull"</span> to merge the remote branch into yours<span class="token punctuation">)</span></code></pre>
<p>Now you have to push your changes to the remote repo. You have to use <code>--force</code>, since this is not a fast-forward commit and you have to rewrite the history of the remote repo.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> push origin master <span class="token parameter variable">--force</span></code></pre>
<p>Reference <a href="https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository">here</a></p>
<h2 id="h-show-branch-name-in-linux-terminal" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-show-branch-name-in-linux-terminal"><span aria-hidden="true">#</span></a> Show branch name in Linux terminal</h2>
<p>For this, you have to edit your <code>.bashrc</code> file. It should be in your <code>home</code> directory.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function-name function">parse_git_branch</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token function">git</span> branch <span class="token operator"><span class="token file-descriptor important">2</span>></span> /dev/null <span class="token operator">|</span> <span class="token function">sed</span> <span class="token parameter variable">-e</span> <span class="token string">'/^[^*]/d'</span> <span class="token parameter variable">-e</span> <span class="token string">'s/* \(.*\)/(\1)/'</span>
<span class="token punctuation">}</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token string">"<span class="token variable">$color_prompt</span>"</span> <span class="token operator">=</span> <span class="token function">yes</span> <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token assign-left variable"><span class="token environment constant">PS1</span></span><span class="token operator">=</span><span class="token string">'${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[01;31m\]$(parse_git_branch)\[\033[00m\]\$ '</span>
<span class="token keyword">else</span>
<span class="token assign-left variable"><span class="token environment constant">PS1</span></span><span class="token operator">=</span><span class="token string">'${debian_chroot:+($debian_chroot)}\u@\h:\w$(parse_git_branch)\$ '</span>
<span class="token keyword">fi</span>
<span class="token builtin class-name">unset</span> color_prompt force_color_prompt</code></pre>
<p>Reference <a href="https://www.leaseweb.com/labs/2013/08/git-tip-show-your-branch-name-on-the-linux-prompt/">here</a></p>
<h2 id="h-compare-a-file-across-2-branches" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-compare-a-file-across-2-branches"><span aria-hidden="true">#</span></a> Compare a file across 2 branches</h2>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> difftool <span class="token operator"><</span>target branch<span class="token operator">></span> -- <span class="token operator"><</span>your file<span class="token operator">></span></code></pre>
<p>For example, if you are on <code>master</code> and want to check a file on your <code>release-01.03.02</code> branch, run:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> difftool release-01.03.02 -- src/js/index.js</code></pre>
<p>Reference <a href="https://stackoverflow.com/a/4099805/3036129">here</a>, and if your file has a different name in 2 different branches, see <a href="https://stackoverflow.com/a/8131164/3036129">here</a>.</p>
<h2 id="h-list-all-commits-for-a-specific-file" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-list-all-commits-for-a-specific-file"><span aria-hidden="true">#</span></a> List all commits for a specific file</h2>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> log <span class="token parameter variable">--follow</span> filename</code></pre>
<p>Reference <a href="https://stackoverflow.com/a/8808453/3036129">here</a></p>
<p>Configure Git aliases<br />
You can use <code>git config</code> to create your aliases, but there is a faster way.<br />
Add (or edit) the <code>alias</code> section of your <code>.gitconfig</code> file. It should be in your <code>home</code> directory.</p>
<p>These is the <code>alias</code> section in my <code>.gitconfig</code>.</p>
<pre class="language-shell"><code class="language-shell"><span class="token punctuation">[</span>alias<span class="token punctuation">]</span>
st <span class="token operator">=</span> status
au <span class="token operator">=</span> <span class="token function">add</span> <span class="token parameter variable">-u</span>
dt <span class="token operator">=</span> difftool
cm <span class="token operator">=</span> commit <span class="token parameter variable">-m</span>
cma <span class="token operator">=</span> commit <span class="token parameter variable">--amend</span> --no-edit
co <span class="token operator">=</span> checkout
<span class="token comment"># list aliases https://gist.github.com/mwhite/6887990</span>
la <span class="token operator">=</span> <span class="token string">"!git config -l | grep alias | cut -c 7-"</span></code></pre>
<p>For some other Git aliases, see <a href="https://git.wiki.kernel.org/index.php/Aliases">this wiki</a>.</p>
<p>Reference <a href="https://gist.github.com/mwhite/6887990">here</a> and <a href="https://git-scm.com/book/it/v2/Git-Basics-Git-Aliases">here</a></p>
<h2 id="h-commit-only-part-of-a-file" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-commit-only-part-of-a-file"><span aria-hidden="true">#</span></a> Commit only part of a file</h2>
<p>You can use <code>git gui</code> for this. If you don’t have it already, install it with <code>sudo apt-get install git-gui</code>. Here is what I usually do:</p>
<ol>
<li><code>git gui</code></li>
<li>right click on the code you want to commit and select <code>stage lines for commit</code> (or <code>stage hunk for commit</code>)</li>
<li><code>git stash</code>, to save in the stash all changes that I am not committing right now</li>
<li><code>git commit</code></li>
<li><code>git stash apply</code></li>
<li><code>git stash drop</code></li>
</ol>
<p>If you like (I personally don’t), you can replace</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> stash apply
<span class="token function">git</span> stash drop</code></pre>
<p>with</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> stash pop</code></pre>
<p>Reference <a href="https://stackoverflow.com/a/16137932">here</a>.</p>
<p>In alternative to <code>git gui</code>, you can use <a href="https://git-scm.com/book/en/v2/Git-Tools-Interactive-Staging">interactive staging from the terminal</a>. I have never tried this.</p>
<h2 id="h-git-hooks" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-git-hooks"><span aria-hidden="true">#</span></a> Git hooks</h2>
<p>I am not an expert on hooks in Git, but I found this really <a href="https://blog.ittybittyapps.com/blog/2013/09/03/git-pre-push/">nice article about them</a>.</p>
<p>I think the most useful hooks are</p>
<ul>
<li>pre-push</li>
<li>prepare-commit-msg</li>
<li>post-commit</li>
</ul>
<p>It would be nice to have a pre-merge hook, so we could prevent a feature branch from merging into master if certain conditions are not met (e.g. your tests fail). Since a pre-merge hook is not available, you’ll have to [write it]((<a href="https://stackoverflow.com/questions/19102714/how-would-i-write-a-pre-merge-hook-in-git">https://stackoverflow.com/questions/19102714/how-would-i-write-a-pre-merge-hook-in-git</a>).</p>
<p>Reference <a href="https://www.atlassian.com/git/tutorials/git-hooks">here</a>.</p>
<h2 id="h-discard-changes-to-git-submodules" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/my-personal-git-memo/#h-discard-changes-to-git-submodules"><span aria-hidden="true">#</span></a> Discard changes to Git submodules</h2>
<p>I think it happened only once to me, but I had to reset all submodules with this line:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> submodule foreach <span class="token function">git</span> reset <span class="token parameter variable">--hard</span></code></pre>
<p>Reference <a href="https://kalyanchakravarthy.net/blog/git-discard-submodule-changes/">here</a>.</p>
https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/Getting started with Webpack and ES62017-05-12T00:00:00Z2017-05-12T00:00:00Z
<p>Project starters like <a href="https://github.com/AngularClass/angular-starter">Angular starter</a> and <a href="https://github.com/pydanny/cookiecutter-django">Cookiecutter-Django</a> are really amazing when you are experienced with the technology they use and you want to develop applications that follow best practices and are ready for production. On the other hand, if you are just starting out, trying to wrap your head around these complex setups will be a daunting task and most likely a waste of your time. It’s fundamental to know why you need a given package in your project, and why it’s good to have your files in a given location in your project directory tree.</p>
<p>Here I want to list all the basic requirements to get you started with a Webpack project that uses ES6, and I going to explain, very briefly, why you need each package and configuration.</p>
<p><em>Note: If you use npm, replace all yarn commands with theit npm equivalent</em>.</p>
<p>Here is a nice checklist:</p>
<ol>
<li>Think about your project</li>
<li>ESLint</li>
<li>The bare minimum</li>
<li>babel-core</li>
<li>babel-preset-es2015</li>
<li>webpack</li>
<li>webpack-dev-server</li>
<li>Loaders</li>
<li>package.json</li>
<li>webpack.config.js</li>
</ol>
<hr />
<h2 id="h-1---think-about-your-project" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-1---think-about-your-project"><span aria-hidden="true">#</span></a> 1 - Think about your project</h2>
<p>Take some time to think about your project. Will it be a small project? A big one? Are you going to use third-party Javascript / CSS libraries? What about a CSS preprocessor like SASS? Do you need different environments (e.g. development, production, test)?</p>
<p>The structure of your project depends on the type of application you are developing, the size of the project itself, and of course on personal preferences. You can use some common principles and general guidelines though. A popular approach is to follow <a href="https://12factor.net/">the Twelve Factor App</a> (see also <a href="https://will.koffel.org/post/2014/12-factor-apps-in-plain-english/">here</a> for an explanation easier to digest than the original one). Probably for small projects - like this one - such considerations are a bit an overkill, but I think it’s important to keep that in mind anyway.</p>
<p>In this article we are going to create a very simple project. In the end it will look like this:</p>
<pre class="language-shell"><code class="language-shell"><span class="token builtin class-name">.</span>
├── dist
│ ├── bundle.js <span class="token comment"># generated by webpack</span>
│ ├── bundle.js.map <span class="token comment"># generated by webpack</span>
│ └── index.html
├── node_modules
├── package.json
├── src
│ ├── index.js
│ └── style.css
├── webpack.config.js
└── yarn.lock</code></pre>
<p>To get started, open a terminal and run:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">yarn</span> init</code></pre>
<p>This will create a <code>package.json</code> file containing the information about your project and its dependencies. You don’t really need to type anything, just press enter and accept the default choices.</p>
<p><em>Note: if you created this project on GitHub and you cloned it on your machine, the yarn init wizard will fill in the correct author and url of your repository.</em></p>
<h2 id="h-2---eslint" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-2---eslint"><span aria-hidden="true">#</span></a> 2 - ESLint</h2>
<p>Using a linter will certainly improve the quality of the code you write, so if you don’t have it yet, I strongly suggest you to install ESLint <strong>globally</strong> with:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">yarn</span> global <span class="token function">add</span> eslint</code></pre>
<p>In your project folder, setup the linter with:</p>
<pre class="language-shell"><code class="language-shell">eslint <span class="token parameter variable">--init</span></code></pre>
<p>Answer the questions on the screen. I like to use the <a href="https://github.com/airbnb/javascript">AirBnB style guide</a> and to save the linter configuration file as JSON: <code>.eslintrc.json</code>.</p>
<h2 id="h-3---the-bare-minimum" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-3---the-bare-minimum"><span aria-hidden="true">#</span></a> 3 - The bare minimum</h2>
<p>You are going to need an HTML file that contains your webpack bundle, a Javascript file that represents the <em>entry point</em> of your application, and a CSS file for styling.</p>
<p>Create an <code>index.html</code> file in your <code>dist</code> folder. As you can see, the <code>bundle.js</code> generated by webpack is included in the <code><script></code> tag. I added some Google fonts too.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>html</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>head</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">charset</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">'</span>utf-8<span class="token punctuation">'</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>title</span><span class="token punctuation">></span></span>Getting started with Webpack and ES6<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>title</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://fonts.googleapis.com/css?family=Open+Sans|Raleway<span class="token punctuation">"</span></span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>stylesheet<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>head</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>body</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h1</span><span class="token punctuation">></span></span>Getting started with Webpack and ES6<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h1</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span> <span class="token attr-name">src</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>./bundle.js<span class="token punctuation">"</span></span> <span class="token attr-name">charset</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>utf-8<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token script"></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>body</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>html</span><span class="token punctuation">></span></span></code></pre>
<p>Create a <code>style.css</code> file in your <code>src</code> folder. This file will be processed by the <code>css-loader</code> first, and by the <code>style-loader</code> second.</p>
<pre class="language-css"><code class="language-css"><span class="token selector">h1</span> <span class="token punctuation">{</span>
<span class="token property">font-family</span><span class="token punctuation">:</span> <span class="token string">'Open Sans'</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token selector">p</span> <span class="token punctuation">{</span>
<span class="token property">font-family</span><span class="token punctuation">:</span> <span class="token string">'Raleway'</span><span class="token punctuation">;</span>
<span class="token property">font-style</span><span class="token punctuation">:</span> italic<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>Lastly, write an <code>index.js</code> in your <code>src</code> folder. This file will be the entry point for your app.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'./style.css'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> name <span class="token operator">=</span> <span class="token string">'Webpack'</span>
<span class="token keyword">const</span> p <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">createElement</span><span class="token punctuation">(</span><span class="token string">'p'</span><span class="token punctuation">)</span>
p<span class="token punctuation">.</span>textContent <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Hello </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>name<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span>
document<span class="token punctuation">.</span>body<span class="token punctuation">.</span><span class="token function">appendChild</span><span class="token punctuation">(</span>p<span class="token punctuation">)</span></code></pre>
<h2 id="h-4---babel-core" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-4---babel-core"><span aria-hidden="true">#</span></a> 4 - babel-core</h2>
<p>babel-core is the Babel compiler itself, or maybe it’s more appropriate calling it <em>transpiler</em>, because it transforms javascript code into javascript code. It can be broken down into 3 parts: a parser, a transformer and a generator. You can find more about how it works in the <a href="https://github.com/babel/babel">documentation</a>. Install it with:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">yarn</span> <span class="token function">add</span> <span class="token parameter variable">--dev</span> babel-core</code></pre>
<p><em>Note: <a href="https://babeljs.io/setup">this page on babeljs.io</a> tells you which packages you need in order to integrate Babel with your tools of choice</em>.</p>
<h2 id="h-5--babel-preset-es2015" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-5--babel-preset-es2015"><span aria-hidden="true">#</span></a> 5- babel-preset-es2015</h2>
<p>As the name implies, it’s a preset that automatically determines the Babel plugins you need, based on your supported environments, so Babel can perform its syntax transformation. If you are developing a web application then you want the es2015 preset because you need to target ECMAScript 2015 (a.k.a ES6). You enter ES6 code, and thanks to all Babel plugins installed (because such plugins are required by this preset) the Babel compiler will generate code that can run in all major browsers. You can find more about this process <a href="https://babeljs.io/docs/en/babel-preset-env/">here</a>.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">yarn</span> <span class="token function">add</span> <span class="token parameter variable">--dev</span> "babel-preset-es2015</code></pre>
<p>You could specify the preset in a <code>.babelrc</code> file (you need to put it under version control) or in the <code>webpack.confg.js</code>. For more information about <code>babelrc</code> see <a href="https://babeljs.io/docs/en/config-files/">here</a>.</p>
<p><em>Note: Keep in mind that ECMAScript is a standard, and Javascript is an implementation of such standard. For a nice, short overview of major ECMAScript versions see <a href="https://benmccormick.org/2015/09/14/es5-es6-es2016-es-next-whats-going-on-with-javascript-versioning/">this article</a></em>.</p>
<h2 id="h-6---webpack" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-6---webpack"><span aria-hidden="true">#</span></a> 6 - webpack</h2>
<p>It’s the Javascript module bundler, currently at version 2. It lets you create bundles for your application. You can have one or more bundles for all your Javascript files, and one or more bundles for your stylesheets. The number of bundles depends on your webpack configuration. Even if you are going to use Webpack in all of your projects, install it <strong>locally</strong>:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">yarn</span> <span class="token function">add</span> <span class="token parameter variable">--dev</span> webpack</code></pre>
<h2 id="h-7---webpack-dev-server" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-7---webpack-dev-server"><span aria-hidden="true">#</span></a> 7 - webpack-dev-server</h2>
<p>It’s a little Node.js Express server for development. It’s really fast because it serves your bundles from the memory, and since it has <em>live reload</em> it’s also very handy to use.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">yarn</span> <span class="token function">add</span> <span class="token parameter variable">--dev</span> webpack-dev-server</code></pre>
<h2 id="h-8---loaders" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-8---loaders"><span aria-hidden="true">#</span></a> 8 - Loaders</h2>
<p>Webpack treats every file as a <em>module</em>. Loaders convert files into modules, so you can <em>require</em> these files in your application. In order to tell Webpack which loader to use, you need to write a set of rules in your <code>webpack.config.js</code> file. For each rule you need to specify a regular expression (the <code>test</code> field) and the loader you want to use (the <code>use</code> field).</p>
<p>You will need <code>babel-loader</code> to convert ES6 files into modules, <code>css-loader</code> to convert CSS files into modules, and <code>style-loader</code> to create a <code><style></code> node and append it in the <code><head></code> of your <code>index.html</code>.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">yarn</span> <span class="token function">add</span> <span class="token parameter variable">--dev</span> babel-loader style-loader css-loader</code></pre>
<h2 id="h-9---packagejson" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-9---packagejson"><span aria-hidden="true">#</span></a> 9 - package.json</h2>
<p>Now that all the dependencies are set, the next step is to open your <code>package.json</code> and write the commands for running the webpack dev server and build the bundle/s. These commands go to the <code>scripts</code> field in <code>package.json</code>.</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
<span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"webpack-starter"</span><span class="token punctuation">,</span>
<span class="token property">"version"</span><span class="token operator">:</span> <span class="token string">"1.0.0"</span><span class="token punctuation">,</span>
<span class="token property">"main"</span><span class="token operator">:</span> <span class="token string">"index.js"</span><span class="token punctuation">,</span>
<span class="token property">"scripts"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"server"</span><span class="token operator">:</span> <span class="token string">"webpack-dev-server --config webpack.config.js"</span><span class="token punctuation">,</span>
<span class="token property">"lint"</span><span class="token operator">:</span> <span class="token string">"eslint src"</span><span class="token punctuation">,</span>
<span class="token property">"build"</span><span class="token operator">:</span> <span class="token string">"webpack --config webpack.config.js --progress"</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token property">"license"</span><span class="token operator">:</span> <span class="token string">"MIT"</span><span class="token punctuation">,</span>
<span class="token property">"devDependencies"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token property">"babel-core"</span><span class="token operator">:</span> <span class="token string">"^6.24.1"</span><span class="token punctuation">,</span>
<span class="token property">"babel-loader"</span><span class="token operator">:</span> <span class="token string">"^7.0.0"</span><span class="token punctuation">,</span>
<span class="token property">"babel-preset-es2015"</span><span class="token operator">:</span> <span class="token string">"^6.24.1"</span><span class="token punctuation">,</span>
<span class="token property">"css-loader"</span><span class="token operator">:</span> <span class="token string">"^0.28.1"</span><span class="token punctuation">,</span>
<span class="token property">"eslint"</span><span class="token operator">:</span> <span class="token string">"^3.19.0"</span><span class="token punctuation">,</span>
<span class="token property">"eslint-config-airbnb-base"</span><span class="token operator">:</span> <span class="token string">"^11.1.3"</span><span class="token punctuation">,</span>
<span class="token property">"eslint-plugin-import"</span><span class="token operator">:</span> <span class="token string">"^2.2.0"</span><span class="token punctuation">,</span>
<span class="token property">"style-loader"</span><span class="token operator">:</span> <span class="token string">"^0.17.0"</span><span class="token punctuation">,</span>
<span class="token property">"webpack"</span><span class="token operator">:</span> <span class="token string">"^2.5.1"</span><span class="token punctuation">,</span>
<span class="token property">"webpack-dev-server"</span><span class="token operator">:</span> <span class="token string">"^2.4.5"</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="h-10---webpackconfigjs" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-10---webpackconfigjs"><span aria-hidden="true">#</span></a> 10 - webpack.config.js</h2>
<p>There is only one thing left to do: writing a configuration file for Webpack. You will need to set the entry point of your application, the output of your <code>bundle.js</code>, the rules for your loaders, and the configuration for the webpack-dev-server.</p>
<p>For this very simple project, the <code>webpack.config.js</code> will look like this:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> path <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'path'</span><span class="token punctuation">)</span>
module<span class="token punctuation">.</span>exports <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token literal-property property">entry</span><span class="token operator">:</span> <span class="token punctuation">[</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'index.js'</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token literal-property property">output</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">path</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'dist'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">filename</span><span class="token operator">:</span> <span class="token string">'bundle.js'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">module</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">rules</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token comment">// rule for .js/.jsx files</span>
<span class="token punctuation">{</span>
<span class="token literal-property property">test</span><span class="token operator">:</span> <span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">\.(js|jsx)$</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">include</span><span class="token operator">:</span> <span class="token punctuation">[</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token literal-property property">exclude</span><span class="token operator">:</span> <span class="token punctuation">[</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'node_modules'</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token literal-property property">use</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">loader</span><span class="token operator">:</span> <span class="token string">'babel-loader'</span><span class="token punctuation">,</span>
<span class="token literal-property property">options</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">presets</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">'es2015'</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token comment">// rule for .css files</span>
<span class="token punctuation">{</span>
<span class="token literal-property property">test</span><span class="token operator">:</span> <span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">\.css$</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">include</span><span class="token operator">:</span> <span class="token punctuation">[</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token literal-property property">use</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token punctuation">{</span> <span class="token literal-property property">loader</span><span class="token operator">:</span> <span class="token string">'style-loader'</span> <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">loader</span><span class="token operator">:</span> <span class="token string">'css-loader'</span> <span class="token punctuation">}</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token literal-property property">devtool</span><span class="token operator">:</span> <span class="token string">'source-map'</span><span class="token punctuation">,</span>
<span class="token literal-property property">devServer</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">host</span><span class="token operator">:</span> <span class="token string">'localhost'</span><span class="token punctuation">,</span>
<span class="token literal-property property">port</span><span class="token operator">:</span> <span class="token number">8080</span><span class="token punctuation">,</span>
<span class="token literal-property property">contentBase</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'dist'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">inline</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> <span class="token comment">// live reloading</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span></code></pre>
<p><em>Note: it’s not mandatory to set</em> <code>devtool</code> <em>and generate a source map, but since it really helps when debugging your application I decided to include it in the configuration</em>.</p>
<h2 id="h-next-steps" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/getting-started-with-webpack-and-es6/#h-next-steps"><span aria-hidden="true">#</span></a> Next steps</h2>
<p>If you managed to run this project, congratulations! As you can see, even for a small project like this one the <code>webpack.config.js</code> file is not that small. If your application gets bigger, you might want to create multiple entry points for your app, organize your project tree in a different way (e.g. create a folder for js files, another one for css files, etc), or use some Webpack plugin.</p>
<p>Webpack is a great tool, and has a vast ecosystem of plugins. In this article I wanted to keep things as simple as possible, so I didn’t include any, but in a real project you will likely use several plugins. If you are curious about which Webpack plugins I am currently using you may want to check <a href="https://giacomodebidda.com/posts/webpack-plugins-that-i-am-currently-using/">this article</a>.</p>
https://www.giacomodebidda.com/posts/webpack-plugins-that-i-am-currently-using/Webpack plugins that I am currently using2017-05-11T00:00:00Z2017-05-11T00:00:00Z
<p>Webpack is probably the most popular module bundler for Javascript applications. Its idea is to have a <strong>single tool</strong> that manages all the assets: Javascript files, CSS files, Images, etc. Webpack lets you also perform all the necessary tasks for managing and optimizing code dependencies, such as compilation, concatenation, minification, and compression.</p>
<p>Webpack is based on these 4 key concepts:</p>
<ol>
<li>entry</li>
<li>output</li>
<li>loaders</li>
<li>plugins</li>
</ol>
<p>The <a href="https://webpack.js.org/concepts/">official documentation</a> does a great job in explaining these concept in a clear, concise way, so I suggest you to check it out. Note that you should install Webpack locally, as they suggest <a href="https://webpack.js.org/guides/installation/#global-installation">here</a>. Other great resources to learn how to use and configure Webpack are the <a href="https://www.youtube.com/channel/UCblk3IlXm_ZXkR5OYLuYWaQ/playlists">Matthew Hsiung’s Webpack playlists on Youtube</a> and <a href="https://medium.com/@rajaraodv/webpack-the-confusing-parts-58712f8fcad9">this article on Medium</a>.</p>
<p>One of the best features of Webpack is its huge ecosystem of plugins. While loaders can only execute transforms on a per-file basis, plugins add custom functionality and let you perform actions on “compilations” or “chunks” of your bundled modules.</p>
<p>Here I am going to list all plugins that I am currently using and include a sample configuration for each one of them.</p>
<ul>
<li>clean-webpack-plugin</li>
<li>copy-webpack-plugin</li>
<li>extract-text-webpack-plugin</li>
<li>favicons-webpack-plugin</li>
<li>html-webpack-plugin</li>
<li>Other plugins</li>
</ul>
<h2 id="h-clean-webpack-plugin" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/webpack-plugins-that-i-am-currently-using/#h-clean-webpack-plugin"><span aria-hidden="true">#</span></a> clean-webpack-plugin</h2>
<p><a href="https://github.com/johnagan/clean-webpack-plugin">This plugin</a> removes your build folder(s) before building. I use it to remove my <strong>dist/</strong> folder.</p>
<pre class="language-javascript"><code class="language-javascript">plugins <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token comment">// more plugins</span>
<span class="token keyword">new</span> <span class="token class-name">CleanWebpackPlugin</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'dist'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">root</span><span class="token operator">:</span> __dirname<span class="token punctuation">,</span> <span class="token literal-property property">verbose</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token comment">// more plugins</span>
<span class="token punctuation">]</span></code></pre>
<h2 id="h-copy-webpack-plugin" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/webpack-plugins-that-i-am-currently-using/#h-copy-webpack-plugin"><span aria-hidden="true">#</span></a> copy-webpack-plugin</h2>
<p>Some people consider the <a href="https://github.com/kevlened/copy-webpack-plugin">Copy Webpack Plugin</a> to be an antipattern, because it goes against the main philosophy of Webpack: if your application needs a given asset, you should <code>require</code> it where it’s needed, use a file loader to convert it into a module, and let Webpack manage it. I guess they have a point, but at the moment I find it very convenient.</p>
<pre class="language-javascript"><code class="language-javascript">plugins <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token comment">// more plugins</span>
<span class="token keyword">new</span> <span class="token class-name">CopyWebpackPlugin</span><span class="token punctuation">(</span>
<span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token literal-property property">from</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'data'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">to</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'dist'</span><span class="token punctuation">,</span> <span class="token string">'data'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span> <span class="token literal-property property">debug</span><span class="token operator">:</span> <span class="token string">'warning'</span> <span class="token punctuation">}</span>
<span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token comment">// more plugins</span>
<span class="token punctuation">]</span></code></pre>
<h2 id="h-extract-text-webpack-plugin" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/webpack-plugins-that-i-am-currently-using/#h-extract-text-webpack-plugin"><span aria-hidden="true">#</span></a> extract-text-webpack-plugin</h2>
<p>The <a href="https://github.com/webpack-contrib/extract-text-webpack-plugin">Extract Text Plugin</a> takes all files that match the specified regular expressions, and bundle them all into a single file. You can use it to collect all stylesheets required by your application and move them into a single file. This way your stylesheets are no longer inlined into the JS bundle. If your total stylesheet volume is big, it will be faster because the CSS bundle is loaded in parallel to the JS bundle. You will need <code>css-loader</code> to use this plugin. If you use Sass, you will need <code>sass-loader</code> too. A nice benefit of using this plugin is that it allows you to avoid using the <code>style-loader</code>, which has the disadvantage of creating a <code><style></code> node in the DOM.</p>
<p>If your app has multiple entry points (e.g. home.js and about.js), this pluging plays well together with the CommonsChunkPlugin, but bear in mind that the configuration will be different from the one I included down here.</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token literal-property property">module</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">rules</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token comment">// rule for .css files</span>
<span class="token punctuation">{</span>
<span class="token literal-property property">test</span><span class="token operator">:</span> <span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">\.css$</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">include</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'css'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">use</span><span class="token operator">:</span> ExtractTextPlugin<span class="token punctuation">.</span><span class="token function">extract</span><span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">fallback</span><span class="token operator">:</span> <span class="token string">'style-loader'</span><span class="token punctuation">,</span> <span class="token literal-property property">use</span><span class="token operator">:</span> <span class="token string">'css-loader'</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token comment">// rule for .sass/.scss files</span>
<span class="token punctuation">{</span>
<span class="token literal-property property">test</span><span class="token operator">:</span> <span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">\.(sass|scss)$</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">include</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'sass'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">use</span><span class="token operator">:</span> ExtractTextPlugin<span class="token punctuation">.</span><span class="token function">extract</span><span class="token punctuation">(</span><span class="token punctuation">{</span> <span class="token literal-property property">fallback</span><span class="token operator">:</span> <span class="token string">'style-loader'</span><span class="token punctuation">,</span> <span class="token literal-property property">use</span><span class="token operator">:</span> <span class="token punctuation">[</span><span class="token string">'css-loader'</span><span class="token punctuation">,</span> <span class="token string">'sass-loader'</span><span class="token punctuation">]</span> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
plugins <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token comment">// more plugins</span>
<span class="token keyword">new</span> <span class="token class-name">ExtractTextPlugin</span><span class="token punctuation">(</span><span class="token string">'styles.css'</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token comment">// name of the bundle for all extracted files</span>
<span class="token comment">// more plugins</span>
<span class="token punctuation">]</span></code></pre>
<h2 id="h-favicons-webpack-plugin" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/webpack-plugins-that-i-am-currently-using/#h-favicons-webpack-plugin"><span aria-hidden="true">#</span></a> favicons-webpack-plugin</h2>
<p><a href="https://github.com/jantimon/favicons-webpack-plugin">This plugin</a> automatically creates all the favicons you need for your application. Since its basic configuration creates 37 different favicons, optimized for different devices, it takes a while. That’s why it might be a good idea to use this plugin only in production, not in development.</p>
<pre class="language-javascript"><code class="language-javascript">plugins <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token comment">// more plugins</span>
<span class="token keyword">new</span> <span class="token class-name">FaviconsWebpackPlugin</span><span class="token punctuation">(</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'images'</span><span class="token punctuation">,</span> <span class="token string">'logo.png'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token comment">// more plugins</span>
<span class="token punctuation">]</span></code></pre>
<h2 id="h-html-webpack-plugin" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/webpack-plugins-that-i-am-currently-using/#h-html-webpack-plugin"><span aria-hidden="true">#</span></a> html-webpack-plugin</h2>
<p><a href="https://github.com/jantimon/html-webpack-plugin">This plugin</a> is a <em>must have</em>. It generates an HTML5 file that includes all your webpack bundles in the body using script tags. This way in your <code>index.html</code> file you don’t have to include any <code><link></code> tag in the <code><head></code> for your stylesheet bundle, nor you have to include any <code><script></code> tag in the <code><body></code> for your javascript bundle.<br />
You can either let the plugin generate an HTML file for you, supply your own template using lodash templates or use your own loader if you are using different templates (e.g. handlebars). If you have the appropriate loaders, <a href="https://stackoverflow.com/questions/42861651/can-i-use-html-webpack-plugin-with-a-django-base-template">you can even use Django templates</a> with this plugin.</p>
<p>If you have multiple entry points, create an instance of this plugin for each one of them.</p>
<p><em>A small gotcha</em>: if you forget to remove the <code><link></code> tag or your <code><script></code> tag in your <code>index.html</code> , Webpack <a href="https://stackoverflow.com/questions/37081559/all-my-code-runs-twice-when-compiled-by-webpack">executes twice</a> all the code required by your entry point.</p>
<pre class="language-javascript"><code class="language-javascript">plugins <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token comment">// more plugins</span>
<span class="token keyword">new</span> <span class="token class-name">HtmlWebpackPlugin</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token literal-property property">template</span><span class="token operator">:</span> path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'src'</span><span class="token punctuation">,</span> <span class="token string">'templates'</span><span class="token punctuation">,</span> <span class="token string">'index.html'</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token literal-property property">filename</span><span class="token operator">:</span> <span class="token string">'index.html'</span><span class="token punctuation">,</span>
<span class="token literal-property property">hash</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token comment">// more plugins</span>
<span class="token punctuation">]</span></code></pre>
<h2 id="h-other-plugins" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/webpack-plugins-that-i-am-currently-using/#h-other-plugins"><span aria-hidden="true">#</span></a> Other plugins</h2>
<p>Here are a few plugins that I am not using at the moment, but that I am pretty sure I will use sooner or later.</p>
<p><strong>CommonsChunkPlugin</strong><br />
The <a href="https://webpack.js.org/plugins/commons-chunk-plugin/">CommonsChunkPlugin</a> is basically an <a href="https://github.com/webpack/docs/wiki/optimization">optimization</a> plugin. It creates a separate file (known as a chunk), consisting of common modules shared between multiple entry points. By separating common modules from bundles, the resulting chunked file can be loaded once initially, and stored in cache for later use. Its configuration requires definitely some effort, but as they say in <a href="https://www.youtube.com/watch?v=-xzWMKuiS2o&list=PLnUE-7Cz5mHERezkTJfh0iU0LESkHmSxA">these two videos</a>, this plugin is a killer feature of Webpack.</p>
<p><strong>Dotenv-webpack</strong><br />
Apps sometimes store config as constants in the code. This is a violation of <a href="https://12factor.net/config">twelve-factor</a>, which requires strict separation of config from code. <a href="https://github.com/mrsteele/dotenv-webpack">This plugin</a> lets you store your configuration in an <code>.env</code> file - which you should add to <code>.gitignore</code> - so you can easily switch between different environments (e.g. development, test, production).</p>
<p><strong>HotModuleReplacementPlugin</strong><br />
The HotModuleReplacementPlugin is basically a LiveReload, for every module. As they say in <a href="https://github.com/webpack/docs/wiki/hot-module-replacement-with-webpack">the documentation</a>, this plugin is still an experimental feature.</p>
<p><strong>Webpack Shell Plugin</strong><br />
The <a href="https://github.com/1337programming/webpack-shell-plugin">Webpack Shell Plugin</a> allows you to run any shell command before or after webpack builds. This will work for both webpack and webpack-dev-server.</p>
https://www.giacomodebidda.com/posts/threejs-project-starter-for-es6-and-webpack-2/Three.js project starter for ES6 and Webpack 22017-04-30T00:00:00Z2017-04-30T00:00:00Z
<p>Some months ago I came across this hilarious article on HackerNoon: <a href="https://hackernoon.com/how-it-feels-to-learn-javascript-in-2016-d3a717dd577f">how it feels to learn javascript in 2016</a> (if you missed it, I suggest you to go and read it after you’re done with this blog post). I mostly code in Python, but sometimes I have to do some frontend stuff, or write some visualizations, so I need to use Javascript.</p>
<p>I have been wanting to improve my Javascript skills for some time, but it seemed a daunting task and to be honest I felt a bit overwhelmed. I didn’t know where to start: Node.js? ES6? React? Vue.js?</p>
<p>As most developers know, the best way to learn is to build some stuff, so I decided to focus on learning the basics of ES6 and write a small project with a library that I wanted to use: <a href="https://threejs.org/">Three.js</a>.</p>
<h2 id="h-es6-babel-and-webpack" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/threejs-project-starter-for-es6-and-webpack-2/#h-es6-babel-and-webpack"><span aria-hidden="true">#</span></a> ES6, Babel and Webpack</h2>
<p>ES6, also known as Ecma Script 2015, is the next generation Javascript and introduces a lot of <a href="https://babeljs.io/learn-es2015/">great features</a> into the language. The <a href="https://nodejs.org/en/docs/es6/">latest versions of Node.js already support many features of this syntax</a>, but several browsers, especially the older ones, are still behind in adopting this new standard and some features are still <a href="https://caniuse.com/#search=es6">not fully supported</a>. Luckily there are some great tools, like <a href="https://babeljs.io/">Babel</a>, that let you write ES6 Javascript and <em>transpile</em> it to ES5.1 Javascript (or any other version you want).</p>
<p>You can use a tool like <code>babel-cli</code> to <em>transpile</em> your ES6 Javascript to ES5.1 Javascript manually, but there are better options that let you automate this process. In the Javascript ecosystem there are many task runners suited for this job. Until some years ago the most popular options were <a href="https://gruntjs.com/">Grunt</a>, <a href="https://gulpjs.com/">Gulp</a> and <a href="https://broccolijs.com/">Broccoli</a>. Nowadays it seems that everyone is using a module bundler like <a href="https://www.npmjs.com/package/neutrino">Neutrino</a>, <a href="https://rollupjs.org/">Rollup</a> or <a href="https://webpack.github.io/">Webpack</a>. I opted for Webpack version 2.</p>
<p>It took me a while to write the configuration file for Webpack, but at least it’s a <em>set-and-forget</em> thing.<br />
After you have done it, you can write ES6 code, and your Webpack toolchain will take care of transpiling it to ES5.1 and bundling it into a single file (usually called <code>bundle.js</code>).</p>
<h2 id="h-eslint-and-airbnb-style-guide" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/threejs-project-starter-for-es6-and-webpack-2/#h-eslint-and-airbnb-style-guide"><span aria-hidden="true">#</span></a> ESLint and AirBnB style guide</h2>
<p>I think that one of the best way to write clean and readable code is to use a linter, and for ES6 there is <a href="https://eslint.org/">ESLint</a>. Writing all the rules for your linter is just insane and counterproductive, so I decided to use a popular style guide and stick to it. After a quick search I found out that the <a href="https://github.com/airbnb/javascript">AirBnB Javascript style guide</a> is one of - if not <em>the</em> - most popular style guide. I tried it for a while and I immediately liked it. It seems pretty strict, and I’ve already learned several things thanks to the warnings given me by the linter.</p>
<h2 id="h-sass" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/threejs-project-starter-for-es6-and-webpack-2/#h-sass"><span aria-hidden="true">#</span></a> SASS</h2>
<p>Variables, nesting selectors, partials, mixins? Yes please! CSS preprocessors like Sass or Less offer you many features that really speed you up. Your browser still needs CSS files, but you can use a Webpack loader to preprocess files as you <code>require()</code> them in your Javascript code.<br />
I started using SASS with the <code>.scss</code> syntax, but after watching a couple of videos of <a href="https://www.youtube.com/channel/UCyIe-61Y8C4_o-zZCtO4ETQ">DevTips</a> I switched to <code>.sass</code>. It feels so much better to have clean, nice formatted code. I’m a Python developer after all. :-)<br />
I like to use a linter when I write <code>.sass</code> files too. It’s not as important as using a linter when writing Javascript code, but it doesn’t hurt.</p>
<h2 id="h-glsl" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/threejs-project-starter-for-es6-and-webpack-2/#h-glsl"><span aria-hidden="true">#</span></a> GLSL</h2>
<p>GLSL stands for OpenGL shading language. It’s a strongly typed, C/C++ like language, and it’s used to write shaders.<br />
A shader is a small program that run on th GPU of your computer, and every WebGL program needs two shaders: a <strong>vertex shader</strong> and a <strong>fragment shader</strong>.</p>
<p>If you write pure WebGL code, you need to write your own shaders every time. In Three.js you have to do it only if you want to define a custom material which is not available in the library. You can do it by defining your own <a href="https://threejs.org/docs/#api/materials/ShaderMaterial">ShaderMaterial</a>.</p>
<p>But where do you write this GLSL code?</p>
<p>Usually the GLSL code is written directly in the <code>index.html</code> and appended to the DOM as text node of a script tag, so they can be easily accessed by Javascript with <code>document.getElementByID</code>.</p>
<p>This might be a convenient way to access the shader code if your project is small, but as your project gets bigger this solution becomes unpractical and messy pretty fast</p>
<p>I found a <a href="https://www.npmjs.com/package/webpack-glsl-loader">Webpack loader for glsl files</a>, so I decided to write both shaders in independent files that I called <code>vertexShader.glsl</code> and <code>fragmentShader.glsl</code>. Then I used <code>require()</code> statements to store the content of these scripts in Javascript variables.</p>
<h2 id="h-threejs-es6-webpack-starter" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/threejs-project-starter-for-es6-and-webpack-2/#h-threejs-es6-webpack-starter"><span aria-hidden="true">#</span></a> threejs-es6-webpack-starter</h2>
<p>So, here is a preview of this project starter. Here you can see a grid, the XYZ axes, a directional light, a spotlight, a particle system (the stars), a group of green cubes (it’s treated as a single object) and an object that uses a custom <code>ShaderMaterial</code>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303234/preview_puzvpq.gif">https://res.cloudinary.com/jackdbd/image/upload/v1599303234/preview_puzvpq.gif</a></p>
<p>If you want to give Three.js a try, clone the repo <a href="https://github.com/jackdbd/threejs-es6-webpack-starter">threejs-es6-webpack-starter</a> and have some 3D fun!</p>
<h2 id="h-additional-resources" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/threejs-project-starter-for-es6-and-webpack-2/#h-additional-resources"><span aria-hidden="true">#</span></a> Additional resources</h2>
<p>You can find some really nice tutorials about shaders and Custom Materials in Three.js <a href="https://blog.cjgammon.com/threejs-custom-shader-material">here</a>, <a href="https://github.com/Jam3/jam3-lesson-webgl-shader-threejs">here</a> and <a href="https://aerotwist.com/tutorials/an-introduction-to-shaders-part-1/">here</a>.</p>
<p>For some more advanced stuff on shaders see these websites:</p>
<ul>
<li><a href="https://www.shadertoy.com/">Shadertoy</a></li>
<li><a href="https://glslsandbox.com/">GLSL Sandbox</a></li>
<li><a href="https://thebookofshaders.com/">The Book of Shaders</a></li>
<li><a href="https://www.kickjs.org/example/shader_editor/shader_editor.html">Kick.js shader editor</a></li>
</ul>
https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/MVC pattern in Python: Dataset2017-04-15T00:00:00Z2017-04-15T00:00:00Z
<p>This is the <strong>third</strong> article of a series of blog posts related to the MVC pattern. In the first article we saw how to divide business logic, presentation layer and user interaction into three components: Model, View and Controller. Last time we replaced the Model without touching a single line of code neither in the View, nor in the Controller.</p>
<p>This time we are going to replace the Model once again, but instead of using a database directly, we are going to use a small <a href="https://en.wikipedia.org/wiki/Object-relational_mapping">ORM</a> called <a href="https://dataset.readthedocs.io/en/latest/">Dataset</a>.</p>
<p>Here are the links to the other articles in the series:</p>
<ol>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/">MVC pattern in Python: Introduction and BasicModel</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-sqlite/">MVC pattern in Python: SQLite</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/">MVC pattern in Python: Dataset</a></li>
</ol>
<p><em>All code was written in Python 3.5. If you are using Python 2.7 you should be able to run it with a few minor changes.</em></p>
<hr />
<p>Table of contents</p>
<ol>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/#intro">Introduction</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/#crud">CRUD</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/#postgres">Switch to PostgreSQL</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/#model">Model</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/#view-controller">View and Controller</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/#conclusion">Conclusion</a></li>
</ol>
<p><a><h2>Introduction</h2></a><br />
First of all, if you haven’t read the first article in the MVC series, I suggest you to read that one first, otherwise many of the things here will not make much sense. Moreover, you will need the code for the <code>View</code> and the <code>Controller</code>.</p>
<p>Dataset is a small abstraction layer built on top of the most popular Python ORM, <a href="https://www.sqlalchemy.org/">SqlAlchemy</a> (interestingly enough, on GitHub <a href="https://github.com/pudo/dataset/">Dataset</a> has even more stars than <a href="https://github.com/zzzeek/sqlalchemy">SqlAlchemy</a> itself!). I stumbled upon this project when I was playing around with <a href="https://kivy.org/#home">Kivy</a> and I needed to store a few records. It was just a small application and I didn’t want to use a database, so I thought about using the <a href="https://kivy.org/docs/api-kivy.storage.jsonstore.html">JSON Storage</a> module of the Kivy framework itself. That worked, but I didn’t like it too much, so I started looking for a better alternative.</p>
<p>As they say in their <a href="https://dataset.readthedocs.io/en/latest/">awesome documentation</a>, with Dataset you can use databases just like you would use a JSON file or a NoSQL store. And the cool thing is that your code will stay basically the same, no matter which database engine you want to use (at this time Dataset supports SQLite, PostgreSQL and MySQL).</p>
<p>In this article I will show you how to use SQLite and PostgreSQL with Dataset.</p>
<p><a><h2>CRUD</h2></a><br />
As we did last time, let’s implement each CRUD functionality in the simplest way possible.</p>
<p>Let’s review the inventory of a small grocery store. A typical product list would look like this:</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption></caption>
<thead class="">
<tr class=""><th scope="col" class="">Name</th><th scope="col" class="">Price</th><th scope="col" class="">Quantity</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">Bread</td><td class="">0.5</td><td class="">20</td></tr><tr class=""><td class="">Milk</td><td class="">1.0</td><td class="">10</td></tr><tr class=""><td class="">Wine</td><td class="">10.0</td><td class="">5</td></tr>
</tbody>
</table>
</div>
</div>
<p>Create a Python script and call it <code>dataset_backend.py</code>.</p>
<p>The first thing to do is to connect to a database. With Dataset you just need a <a href="https://dataset.readthedocs.io/en/latest/api.html#dataset.connect">single line of code</a>. Let’s connect to an in-memory SQLite databse with:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> dataset
conn <span class="token operator">=</span> dataset<span class="token punctuation">.</span>connect<span class="token punctuation">(</span><span class="token string">'sqlite:///:memory:'</span><span class="token punctuation">)</span></code></pre>
<p><code>dataset.connect</code> returns an instance of class <code>Database</code>, an object that represents a SQL database with multiple tables, and opens a new connection to this database. No need to worry about connection timeouts or disconnections.</p>
<p>Ok, now you need to create a table. Forget about SQL statements and Data Definition Language: with Dataset you have <em>automatic schema</em>, so you don’t have to specify any datatype in advance.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># dataset_backend.py</span>
<span class="token keyword">from</span> sqlalchemy<span class="token punctuation">.</span>exc <span class="token keyword">import</span> NoSuchTableError
<span class="token keyword">def</span> <span class="token function">create_table</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Load a table or create it if it doesn't exist yet.
The function load_table can only load a table if exist, and raises a NoSuchTableError if the table does not already exist in the database.
The function get_table either loads a table or creates it if it doesn't exist yet. The new table will automatically have an id column unless specified via optional parameter primary_id, which will be used as the primary key of the table.
Parameters
----------
table_name : str
conn : dataset.persistence.database.Database
"""</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
conn<span class="token punctuation">.</span>load_table<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
<span class="token keyword">except</span> NoSuchTableError <span class="token keyword">as</span> e<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Table {} does not exist. It will be created now'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>e<span class="token punctuation">)</span><span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>get_table<span class="token punctuation">(</span>table_name<span class="token punctuation">,</span> primary_id<span class="token operator">=</span><span class="token string">'name'</span><span class="token punctuation">,</span> primary_type<span class="token operator">=</span><span class="token string">'String'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Created table {} on database {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">,</span> DB_name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>Here is the code for CRUD operations.</p>
<p><em>Create</em></p>
<pre class="language-python"><code class="language-python"><span class="token comment"># dataset_backend.py</span>
<span class="token keyword">from</span> sqlalchemy<span class="token punctuation">.</span>exc <span class="token keyword">import</span> IntegrityError<span class="token punctuation">,</span> NoSuchTableError
<span class="token keyword">import</span> mvc_exceptions <span class="token keyword">as</span> mvc_exc
<span class="token keyword">def</span> <span class="token function">insert_one</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Insert a single item in a table.
Parameters
----------
name : str
price : float
quantity : int
table_name : dataset.persistence.table.Table
conn : dataset.persistence.database.Database
Raises
------
mvc_exc.ItemAlreadyStored: if the record is already stored in the table.
"""</span>
table <span class="token operator">=</span> conn<span class="token punctuation">.</span>load_table<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
table<span class="token punctuation">.</span>insert<span class="token punctuation">(</span><span class="token builtin">dict</span><span class="token punctuation">(</span>name<span class="token operator">=</span>name<span class="token punctuation">,</span> price<span class="token operator">=</span>price<span class="token punctuation">,</span> quantity<span class="token operator">=</span>quantity<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">except</span> IntegrityError <span class="token keyword">as</span> e<span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemAlreadyStored<span class="token punctuation">(</span>
<span class="token string">'"{}" already stored in table "{}".\nOriginal Exception raised: {}'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> table<span class="token punctuation">.</span>table<span class="token punctuation">.</span>name<span class="token punctuation">,</span> e<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">insert_many</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> items<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Insert all items in a table.
Parameters
----------
items : list
list of dictionaries
table_name : str
conn : dataset.persistence.database.Database
"""</span>
<span class="token comment"># TODO: check what happens if 1+ records can be inserted but 1 fails</span>
table <span class="token operator">=</span> conn<span class="token punctuation">.</span>load_table<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
<span class="token keyword">for</span> x <span class="token keyword">in</span> items<span class="token punctuation">:</span>
table<span class="token punctuation">.</span>insert<span class="token punctuation">(</span><span class="token builtin">dict</span><span class="token punctuation">(</span>
name<span class="token operator">=</span>x<span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> price<span class="token operator">=</span>x<span class="token punctuation">[</span><span class="token string">'price'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span>x<span class="token punctuation">[</span><span class="token string">'quantity'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">except</span> IntegrityError <span class="token keyword">as</span> e<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'At least one in {} was already stored in table "{}".\nOriginal '</span>
<span class="token string">'Exception raised: {}'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span><span class="token punctuation">[</span>x<span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span> <span class="token keyword">for</span> x <span class="token keyword">in</span> items<span class="token punctuation">]</span><span class="token punctuation">,</span> table<span class="token punctuation">.</span>table<span class="token punctuation">.</span>name<span class="token punctuation">,</span> e<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p><em>Read</em></p>
<pre class="language-python"><code class="language-python"><span class="token comment"># dataset_backend.py</span>
<span class="token keyword">def</span> <span class="token function">select_one</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> name<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Select a single item in a table.
The dataset library returns a result as an OrderedDict.
Parameters
----------
name : str
name of the record to look for in the table
table_name : str
conn : dataset.persistence.database.Database
Raises
------
mvc_exc.ItemNotStored: if the record is not stored in the table.
"""</span>
table <span class="token operator">=</span> conn<span class="token punctuation">.</span>load_table<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
row <span class="token operator">=</span> table<span class="token punctuation">.</span>find_one<span class="token punctuation">(</span>name<span class="token operator">=</span>name<span class="token punctuation">)</span>
<span class="token keyword">if</span> row <span class="token keyword">is</span> <span class="token keyword">not</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token builtin">dict</span><span class="token punctuation">(</span>row<span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored<span class="token punctuation">(</span>
<span class="token string">'Can\'t read "{}" because it\'s not stored in table "{}"'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> table<span class="token punctuation">.</span>table<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">select_all</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Select all items in a table.
The dataset library returns results as OrderedDicts.
Parameters
----------
table_name : str
conn : dataset.persistence.database.Database
Returns
-------
list
list of dictionaries. Each dict is a record.
"""</span>
table <span class="token operator">=</span> conn<span class="token punctuation">.</span>load_table<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
rows <span class="token operator">=</span> table<span class="token punctuation">.</span><span class="token builtin">all</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token builtin">map</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span> <span class="token builtin">dict</span><span class="token punctuation">(</span>x<span class="token punctuation">)</span><span class="token punctuation">,</span> rows<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p><em>Update</em></p>
<pre class="language-python"><code class="language-python"><span class="token comment"># dataset_backend.py</span>
<span class="token keyword">def</span> <span class="token function">update_one</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Update a single item in the table.
Note: dataset update method is a bit counterintuitive to use. Read the docs here: https://dataset.readthedocs.io/en/latest/quickstart.html#storing-data
Dataset has also an upsert functionality: if rows with matching keys exist they will be updated, otherwise a new row is inserted in the table.
Parameters
----------
name : str
price : float
quantity : int
table_name : str
conn : dataset.persistence.database.Database
Raises
------
mvc_exc.ItemNotStored: if the record is not stored in the table.
"""</span>
table <span class="token operator">=</span> conn<span class="token punctuation">.</span>load_table<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
row <span class="token operator">=</span> table<span class="token punctuation">.</span>find_one<span class="token punctuation">(</span>name<span class="token operator">=</span>name<span class="token punctuation">)</span>
<span class="token keyword">if</span> row <span class="token keyword">is</span> <span class="token keyword">not</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
item <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> name<span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> price<span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> quantity<span class="token punctuation">}</span>
table<span class="token punctuation">.</span>update<span class="token punctuation">(</span>item<span class="token punctuation">,</span> keys<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored<span class="token punctuation">(</span>
<span class="token string">'Can\'t update "{}" because it\'s not stored in table "{}"'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> table<span class="token punctuation">.</span>table<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p><em>Delete</em></p>
<pre class="language-python"><code class="language-python"><span class="token comment"># dataset_backend.py</span>
<span class="token keyword">def</span> <span class="token function">delete_one</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> item_name<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Delete a single item in a table.
Parameters
----------
item_name : str
table_name : str
conn : dataset.persistence.database.Database
Raises
------
mvc_exc.ItemNotStored: if the record is not stored in the table.
"""</span>
table <span class="token operator">=</span> conn<span class="token punctuation">.</span>load_table<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
row <span class="token operator">=</span> table<span class="token punctuation">.</span>find_one<span class="token punctuation">(</span>name<span class="token operator">=</span>item_name<span class="token punctuation">)</span>
<span class="token keyword">if</span> row <span class="token keyword">is</span> <span class="token keyword">not</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
table<span class="token punctuation">.</span>delete<span class="token punctuation">(</span>name<span class="token operator">=</span>item_name<span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored<span class="token punctuation">(</span>
<span class="token string">'Can\'t delete "{}" because it\'s not stored in table "{}"'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item_name<span class="token punctuation">,</span> table<span class="token punctuation">.</span>table<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>Let’s put everything together and see if these CRUD operations are correct!</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># dataset_backend.py</span>
<span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
conn <span class="token operator">=</span> dataset<span class="token punctuation">.</span>connect<span class="token punctuation">(</span><span class="token string">'sqlite:///:memory:'</span><span class="token punctuation">)</span>
table_name <span class="token operator">=</span> <span class="token string">'items'</span>
create_table<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span>
<span class="token comment"># CREATE</span>
my_items <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">20</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'wine'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">10.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">5</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
insert_many<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> items<span class="token operator">=</span>my_items<span class="token punctuation">,</span> table_name<span class="token operator">=</span>table_name<span class="token punctuation">)</span>
insert_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'beer'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">2.0</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">5</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span>table_name<span class="token punctuation">)</span>
<span class="token comment"># if we try to insert an object already stored we get an ItemAlreadyStored exception</span>
<span class="token comment"># insert_one(conn, 'beer', 2.0, 5, table_name=table_name)</span>
<span class="token comment"># READ</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'SELECT milk'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>select_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span>table_name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'SELECT all'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>select_all<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token operator">=</span>table_name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to select an object not stored we get an ItemNotStored exception</span>
<span class="token comment"># print(select_one(conn, 'pizza', table_name=table_name))</span>
<span class="token comment"># UPDATE</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'UPDATE bread, SELECT bread'</span><span class="token punctuation">)</span>
update_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">1.5</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">5</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span>table_name<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>select_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span>table_name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to update an object not stored we get an ItemNotStored exception</span>
<span class="token comment"># print('UPDATE pizza')</span>
<span class="token comment"># update_one(conn, 'pizza', 9.5, 5, table_name=table_name)</span>
<span class="token comment"># DELETE</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'DELETE beer, SELECT all'</span><span class="token punctuation">)</span>
delete_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'beer'</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span>table_name<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>select_all<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token operator">=</span>table_name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to delete an object not stored we get an ItemNotStored exception</span>
<span class="token comment"># print('DELETE fish')</span>
<span class="token comment"># delete_one(conn, 'fish', table_name=table_name)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p><a><h2>Switch to PostgreSQL</h2></a><br />
OK cool, now that we tested all CRUD operations on a SQLite database, let’s try to switch to PostgreSQL.</p>
<p>If you need to setup PostgreSQL on your machine have a look at <a href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/">this post</a>, otherwise just open a terminal and create a new Postgres user:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">sudo</span> <span class="token parameter variable">-i</span> <span class="token parameter variable">-u</span> postgres
createuser <span class="token parameter variable">--interactive</span>
<span class="token operator">>></span><span class="token operator">></span> Enter name of role to add: test_user
<span class="token operator">>></span><span class="token operator">></span> Shall the new role be a superuser? <span class="token punctuation">(</span>y/n<span class="token punctuation">)</span> y</code></pre>
<p>then go the <code>psql</code> shell, assign a password to this user and create a database.</p>
<pre class="language-shell"><code class="language-shell">ALTER <span class="token environment constant">USER</span> test_user WITH PASSWORD <span class="token string">'test_password'</span><span class="token punctuation">;</span>
CREATE DATABASE test_db<span class="token punctuation">;</span></code></pre>
<p>In the <code>psql</code> shell you can double check that this new database and user are available by typing <code>\l</code> and <code>\du</code>, respectively. Exit the <code>psql</code> shell with <code>\q</code>.</p>
<p>In the <code>dataset_backend.py</code> file, create a new function to connect to the database, so you can easily switch back and forth between SQLite and PostgreSQL.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># dataset_backend.py</span>
DB_name <span class="token operator">=</span> <span class="token string">'test_db'</span>
<span class="token keyword">class</span> <span class="token class-name">UnsupportedDatabaseEngine</span><span class="token punctuation">(</span>Exception<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token keyword">def</span> <span class="token function">connect_to_db</span><span class="token punctuation">(</span>db_name<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">,</span> db_engine<span class="token operator">=</span><span class="token string">'sqlite'</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Connect to a database. Create the database if there isn't one yet.
The database can be a SQLite DB (either a DB file or an in-memory DB), or a PostgreSQL DB. In order to connect to a PostgreSQL DB you have first to create a database, create a user, and finally grant him all necessary privileges on that database and tables.
'postgresql://<username>:<password>@localhost:<PostgreSQL port>/<db name>'
Note: at the moment it looks it's not possible to close a connection manually (e.g. like calling conn.close() in sqlite3).
Parameters
----------
db_name : str or None
database name (without file extension .db)
db_engine : str
database engine ('sqlite' or 'postgres')
Returns
-------
dataset.persistence.database.Database
connection to a database
"""</span>
engines <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token string">'sqlite'</span><span class="token punctuation">,</span> <span class="token string">'postgres'</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> db_name <span class="token keyword">is</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
db_string <span class="token operator">=</span> <span class="token string">'sqlite:///:memory:'</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'New connection to in-memory SQLite DB...'</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> db_engine <span class="token operator">==</span> <span class="token string">'sqlite'</span><span class="token punctuation">:</span>
db_string <span class="token operator">=</span> <span class="token string">'sqlite:///{}.db'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>DB_name<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'New connection to SQLite DB...'</span><span class="token punctuation">)</span>
<span class="token keyword">elif</span> db_engine <span class="token operator">==</span> <span class="token string">'postgres'</span><span class="token punctuation">:</span>
db_string <span class="token operator">=</span> \
<span class="token string">'postgresql://test_user:test_password@localhost:5432/{}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>DB_NAME<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'New connection to PostgreSQL DB...'</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> UnsupportedDatabaseEngine<span class="token punctuation">(</span>
<span class="token string">'No database engine with this name. '</span>
<span class="token string">'Choose one of the following: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>engines<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> dataset<span class="token punctuation">.</span>connect<span class="token punctuation">(</span>db_string<span class="token punctuation">)</span></code></pre>
<p>If you now replace this line in the <code>main</code> function:</p>
<pre class="language-python"><code class="language-python">conn <span class="token operator">=</span> dataset<span class="token punctuation">.</span>connect<span class="token punctuation">(</span><span class="token string">'sqlite:///:memory:'</span><span class="token punctuation">)</span></code></pre>
<p>with this line</p>
<pre class="language-python"><code class="language-python">conn <span class="token operator">=</span> connect_to_db<span class="token punctuation">(</span>db_name<span class="token operator">=</span><span class="token string">'test_db'</span><span class="token punctuation">,</span> db_engine<span class="token operator">=</span><span class="token string">'postgres'</span><span class="token punctuation">)</span></code></pre>
<p>you should be able to perform all CRUD operations on a PostgreSQL database, instead of SQLite.</p>
<p>How cool is that? With a single line we completely switched database engine!</p>
<p><a><h2>Model</h2></a><br />
Now that all CRUD operations are implemented as simple functions, creating a class for a Model that uses a SQLite database as persistence layer is pretty straightforward.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># model_view_controller.py</span>
<span class="token keyword">import</span> dataset_backend
<span class="token keyword">import</span> mvc_exceptions <span class="token keyword">as</span> mvc_exc
<span class="token keyword">class</span> <span class="token class-name">ModelDataset</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> application_items<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_item_type <span class="token operator">=</span> <span class="token string">'product'</span>
self<span class="token punctuation">.</span>_connection <span class="token operator">=</span> dataset_backend<span class="token punctuation">.</span>connect_to_db<span class="token punctuation">(</span>
dataset_backend<span class="token punctuation">.</span>DB_name<span class="token punctuation">,</span> db_engine<span class="token operator">=</span><span class="token string">'postgres'</span><span class="token punctuation">)</span>
dataset_backend<span class="token punctuation">.</span>create_table<span class="token punctuation">(</span>self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> self<span class="token punctuation">.</span>_item_type<span class="token punctuation">)</span>
self<span class="token punctuation">.</span>create_items<span class="token punctuation">(</span>application_items<span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">item_type</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>_item_type
<span class="token decorator annotation punctuation">@item_type<span class="token punctuation">.</span>setter</span>
<span class="token keyword">def</span> <span class="token function">item_type</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> new_item_type<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_item_type <span class="token operator">=</span> new_item_type
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">connection</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>_connection
<span class="token keyword">def</span> <span class="token function">create_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
dataset_backend<span class="token punctuation">.</span>insert_one<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">create_items</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> items<span class="token punctuation">)</span><span class="token punctuation">:</span>
dataset_backend<span class="token punctuation">.</span>insert_many<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> items<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">read_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> dataset_backend<span class="token punctuation">.</span>select_one<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> name<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">read_items</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> dataset_backend<span class="token punctuation">.</span>select_all<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">update_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
dataset_backend<span class="token punctuation">.</span>update_one<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">delete_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">:</span>
dataset_backend<span class="token punctuation">.</span>delete_one<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> name<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span></code></pre>
<p><a><h2>View and Controller</h2></a><br />
<code>View</code> and <code>Controller</code> are completely <strong>decoupled</strong> from the <code>Model</code> (and between themselves), so you don’t need to change anything in their implementation. If you need the code for these classes, see the <a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/">first article</a> in the series.</p>
<p>The only thing to do is to plug the <code>ModelDataset</code> in the <code>Controller</code>.</p>
<p>Here is a snippet to test our small MVC application:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
my_items <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">20</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'wine'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">10.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">5</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
c <span class="token operator">=</span> Controller<span class="token punctuation">(</span>ModelDataset<span class="token punctuation">(</span>my_items<span class="token punctuation">)</span><span class="token punctuation">,</span> View<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_items<span class="token punctuation">(</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_items<span class="token punctuation">(</span>bullet_points<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_item<span class="token punctuation">(</span><span class="token string">'chocolate'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_item<span class="token punctuation">(</span><span class="token string">'bread'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>insert_item<span class="token punctuation">(</span><span class="token string">'bread'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">1.0</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">5</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>insert_item<span class="token punctuation">(</span><span class="token string">'chocolate'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">2.0</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">10</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_item<span class="token punctuation">(</span><span class="token string">'chocolate'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>update_item<span class="token punctuation">(</span><span class="token string">'milk'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">1.2</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">20</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>update_item<span class="token punctuation">(</span><span class="token string">'ice cream'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">3.5</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">20</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>delete_item<span class="token punctuation">(</span><span class="token string">'fish'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>delete_item<span class="token punctuation">(</span><span class="token string">'bread'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_items<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p><a><h2>Conclusion</h2></a><br />
In this article we implemented an ORM-based backend for the <em>Model</em> component of the MVC architecture. Thanks to the Dataset package, we can switch from SQLite (maybe for <em>development</em>) to PostgreSQL (maybe for <em>production</em>) very easily.</p>
<p>Dataset is a really cool project and I strongly suggest you to check it out. You can go through the awesome <a href="https://dataset.readthedocs.io/en/latest/quickstart.html">quickstart in 12 minutes</a>.</p>
<p>Ah, just in case you want to cleanup your postgres, open the <code>psql</code> shell as user <code>postgres</code> and drop the test database and test user with the following statements:</p>
<pre class="language-shell"><code class="language-shell">DROP DATABASE test_db<span class="token punctuation">;</span>
DROP <span class="token environment constant">USER</span> test_user<span class="token punctuation">;</span></code></pre>
https://www.giacomodebidda.com/posts/mvc-pattern-in-python-sqlite/MVC pattern in Python: SQLite2017-04-10T00:00:00Z2017-04-10T00:00:00Z
<p>This is the <strong>second</strong> article of a series of blog posts related to the MVC pattern. Last time we saw how to divide business logic, presentation layer and user interaction into three components: Model, View and Controller.</p>
<p>This time we are going to replace the Model and implement a persistance layer with a SQLite database.</p>
<p>Here are the links to the other articles in the series:</p>
<ol>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/">MVC pattern in Python: Introduction and BasicModel</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-sqlite/">MVC pattern in Python: SQLite</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/">MVC pattern in Python: Dataset</a></li>
</ol>
<p><em>All code was written in Python 3.5. If you are using Python 2.7 you should be able to run it with a few minor changes.</em></p>
<hr />
<p>Table of contents</p>
<ol>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-sqlite/#intro">Introduction</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-sqlite/#crud">CRUD</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-sqlite/#model">Model</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-sqlite/#view-controller">View and Controller</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-sqlite/#conclusion">Conclusion</a></li>
</ol>
<p><a><h2>Introduction</h2></a><br />
First of all, if you haven’t read my previous article in the MVC series, I suggest you to read that one first, otherwise many of the things here will not make much sense. Moreover, you will need the code for the <code>View</code> and the <code>Controller</code>.</p>
<p><a><h2>CRUD</h2></a><br />
Let’s review the inventory of a small grocery store. A typical product list would look like this:</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption></caption>
<thead class="">
<tr class=""><th scope="col" class="">Name</th><th scope="col" class="">Price</th><th scope="col" class="">Quantity</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">Bread</td><td class="">0.5</td><td class="">20</td></tr><tr class=""><td class="">Milk</td><td class="">1.0</td><td class="">10</td></tr><tr class=""><td class="">Wine</td><td class="">10.0</td><td class="">5</td></tr>
</tbody>
</table>
</div>
</div>
<p>In this article we will use SQLite and store all these products in a database table.</p>
<p>As we did last time, let’s implement each CRUD functionality in the simplest way possible. Create a python script and call it <code>sqlite_backend.py</code>.<br />
Actually, even before writing any code for CRUD operations, we have to write some code to handle database connections.</p>
<p>A great feature of SQLite is that you can create in-memory databases. An in-memory database runs in the RAM of your computer, so it lets you develop and test your code much faster than a “normal” database.</p>
<p>A “normal”, physical SQLite database is just a file, and this makes using SQLite a joy: even if you mess up, you can simply delete your <code>.db</code> file and start over.</p>
<p>The code to establish a connection to SQLite3 is pretty straightforward and doesn’t change either if you are using an in-memory database or a physical one.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> sqlite3
DB_name <span class="token operator">=</span> <span class="token string">'myDB'</span>
<span class="token keyword">def</span> <span class="token function">connect_to_db</span><span class="token punctuation">(</span>db<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Connect to a sqlite DB. Create the database if there isn't one yet.
Open a connection to a SQLite DB (either a DB file or an in-memory DB).
When a database is accessed by multiple connections, and one of the
processes modifies the database, the SQLite database is locked until that
transaction is committed.
Parameters
----------
db : str
database name (without .db extension). If None, create an In-Memory DB.
Returns
-------
connection : sqlite3.Connection
connection object
"""</span>
<span class="token keyword">if</span> db <span class="token keyword">is</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
mydb <span class="token operator">=</span> <span class="token string">':memory:'</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'New connection to in-memory SQLite DB...'</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
mydb <span class="token operator">=</span> <span class="token string">'{}.db'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>db<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'New connection to SQLite DB...'</span><span class="token punctuation">)</span>
connection <span class="token operator">=</span> sqlite3<span class="token punctuation">.</span>connect<span class="token punctuation">(</span>mydb<span class="token punctuation">)</span>
<span class="token keyword">return</span> connection</code></pre>
<p>As you can see, <code>connect_to_db</code> returns a <code>connection</code>, an object tha you will need to pass as argument to each database operation.</p>
<p>Let’s say that you have the following requirement for your application: each database operation should be able to open a connection if there isn’t one already. How would you do it?</p>
<p>You could call <code>connect_to_db</code> at the beginning of each database operation, but this would open a new database connection for each operation, every time. This doesn’t sound too smart, and you should try to reuse a connection that already exists.</p>
<p>You could place a <code>try/except</code> block at the beginning of each database operation, but you would end up with a lot of ugly, <a href="https://stackoverflow.com/a/2298357">duplicate code</a>.</p>
<p>Luckily, in Python there is a better alternative: a decorator.<br />
The <code>try/except</code> block in the code below is dead simple. We just try a very fast query. If it succeeds, it means that there is an open connection that we can use. If it fails, it means that there is no connection or that the connection is closed, and we have to open a new one.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token keyword">from</span> sqlite3 <span class="token keyword">import</span> OperationalError<span class="token punctuation">,</span> IntegrityError<span class="token punctuation">,</span> ProgrammingError
<span class="token comment"># TODO: use this decorator to wrap commit/rollback in a try/except block ?</span>
<span class="token comment"># see https://www.kylev.com/2009/05/22/python-decorators-and-database-idioms/</span>
<span class="token keyword">def</span> <span class="token function">connect</span><span class="token punctuation">(</span>func<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Decorator to (re)open a sqlite database connection when needed.
A database connection must be open when we want to perform a database query
but we are in one of the following situations:
1) there is no connection
2) the connection is closed
Parameters
----------
func : function
function which performs the database query
Returns
-------
inner func : function
"""</span>
<span class="token keyword">def</span> <span class="token function">inner_func</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token operator">*</span>args<span class="token punctuation">,</span> <span class="token operator">**</span>kwargs<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
<span class="token comment"># I don't know if this is the simplest and fastest query to try</span>
conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>
<span class="token string">'SELECT name FROM sqlite_temp_master WHERE type="table";'</span><span class="token punctuation">)</span>
<span class="token keyword">except</span> <span class="token punctuation">(</span>AttributeError<span class="token punctuation">,</span> ProgrammingError<span class="token punctuation">)</span><span class="token punctuation">:</span>
conn <span class="token operator">=</span> connect_to_db<span class="token punctuation">(</span>DB_name<span class="token punctuation">)</span>
<span class="token keyword">return</span> func<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token operator">*</span>args<span class="token punctuation">,</span> <span class="token operator">**</span>kwargs<span class="token punctuation">)</span>
<span class="token keyword">return</span> inner_func</code></pre>
<p>A SQLite database will close a connection automatically after a certain <em>timeout</em> (the default <a href="https://docs.python.org/2/library/sqlite3.html#sqlite3.connect">timeout</a> is 5s). However, sometimes you may want to disconnect from a database explicitly.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token keyword">def</span> <span class="token function">disconnect_from_db</span><span class="token punctuation">(</span>db<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">,</span> conn<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> db <span class="token keyword">is</span> <span class="token keyword">not</span> DB_name<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"You are trying to disconnect from a wrong DB"</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> conn <span class="token keyword">is</span> <span class="token keyword">not</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
conn<span class="token punctuation">.</span>close<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>There is still another thing you need to do before starting to write code to implement any CRUD operation: you need a table!</p>
<p>Your table must contain data about <code>name</code>, <code>price</code> and <code>quantity</code> of every single item. Given the dynamic nature of the Python language, you don’t have to assign a type to any of these three attributes. However, most likely <code>name</code> would be a <code>str</code>, <code>price</code> a <code>float</code> and <code>quantity</code> an <code>int</code>.</p>
<p>In SQLite there are both <a href="https://sqlite.org/datatype3.html">“storage classes” and “datatypes”</a>, but for the most part, “storage class” is indistinguishable from “datatype” and the two terms can be used interchangeably. So, which storage class should you assign to <code>name</code>, <code>price</code>, <code>quantity</code>? I think a good choice is: <code>TEXT</code>, <code>REAL</code> and <code>INTEGER</code>, respectively.</p>
<p><em>Note that here we are defining a table, so we use a <a href="https://en.wikipedia.org/wiki/Data_definition_language">Data Definition Language</a> and <a href="https://stackoverflow.com/questions/730621/do-ddl-statements-always-give-you-an-implicit-commit-or-can-you-get-an-implicit">there is no need to explicitly commit</a></em></p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token decorator annotation punctuation">@connect</span>
<span class="token keyword">def</span> <span class="token function">create_table</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
sql <span class="token operator">=</span> <span class="token string">'CREATE TABLE {} (rowid INTEGER PRIMARY KEY AUTOINCREMENT,'</span> \
<span class="token string">'name TEXT UNIQUE, price REAL, quantity INTEGER)'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>sql<span class="token punctuation">)</span>
<span class="token keyword">except</span> OperationalError <span class="token keyword">as</span> e<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>e<span class="token punctuation">)</span></code></pre>
<p>I’m definitely not an expert in databases, but something which is widely known is that an attacker could insert malicious SQL statements into an entry field of your application, a vulerability called <a href="https://en.wikipedia.org/wiki/SQL_injection">SQL injection</a>. This is just a toy application and I don’t think it makes sense discussing this issue, however some time ago I found a nice snippet to try to prevent SQL injection (actually I don’t remember where I found it, probably Stack Overflow).</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token keyword">def</span> <span class="token function">scrub</span><span class="token punctuation">(</span>input_string<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Clean an input string (to prevent SQL injection).
Parameters
----------
input_string : str
Returns
-------
str
"""</span>
<span class="token keyword">return</span> <span class="token string">''</span><span class="token punctuation">.</span>join<span class="token punctuation">(</span>k <span class="token keyword">for</span> k <span class="token keyword">in</span> input_string <span class="token keyword">if</span> k<span class="token punctuation">.</span>isalnum<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>You can use the <code>scrub</code> function to clean the <code>table_name</code> string.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token decorator annotation punctuation">@connect</span>
<span class="token keyword">def</span> <span class="token function">create_table</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
table_name <span class="token operator">=</span> scrub<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
sql <span class="token operator">=</span> <span class="token string">'CREATE TABLE {} (rowid INTEGER PRIMARY KEY AUTOINCREMENT,'</span> \
<span class="token string">'name TEXT UNIQUE, price REAL, quantity INTEGER)'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>sql<span class="token punctuation">)</span>
<span class="token keyword">except</span> OperationalError <span class="token keyword">as</span> e<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>e<span class="token punctuation">)</span></code></pre>
<p>Now that we finally have a table, let’s start to implement the CRUD functionalities.</p>
<p>Let’s start with the <em>Create</em> functionality.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token decorator annotation punctuation">@connect</span>
<span class="token keyword">def</span> <span class="token function">insert_one</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
table_name <span class="token operator">=</span> scrub<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
sql <span class="token operator">=</span> <span class="token string">"INSERT INTO {} ('name', 'price', 'quantity') VALUES (?, ?, ?)"</span>\
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>sql<span class="token punctuation">,</span> <span class="token punctuation">(</span>name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>commit<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">except</span> IntegrityError <span class="token keyword">as</span> e<span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemAlreadyStored<span class="token punctuation">(</span>
<span class="token string">'{}: "{}" already stored in table "{}"'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>e<span class="token punctuation">,</span> name<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@connect</span>
<span class="token keyword">def</span> <span class="token function">insert_many</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> items<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
table_name <span class="token operator">=</span> scrub<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
sql <span class="token operator">=</span> <span class="token string">"INSERT INTO {} ('name', 'price', 'quantity') VALUES (?, ?, ?)"</span>\
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
entries <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> x <span class="token keyword">in</span> items<span class="token punctuation">:</span>
entries<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token punctuation">(</span>x<span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> x<span class="token punctuation">[</span><span class="token string">'price'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> x<span class="token punctuation">[</span><span class="token string">'quantity'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
conn<span class="token punctuation">.</span>executemany<span class="token punctuation">(</span>sql<span class="token punctuation">,</span> entries<span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>commit<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">except</span> IntegrityError <span class="token keyword">as</span> e<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}: at least one in {} was already stored in table "{}"'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>e<span class="token punctuation">,</span> <span class="token punctuation">[</span>x<span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span> <span class="token keyword">for</span> x <span class="token keyword">in</span> items<span class="token punctuation">]</span><span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>As you can see, <em>Create</em> operations don’t return anything. They just insert data into the database.</p>
<p>Let’s now add a <em>Read</em> functionality, but first there is a small thing to do: if you remember, last time each item was represented as a Python <code>dict</code>;</p>
<pre class="language-python"><code class="language-python">my_items <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">20</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'wine'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">10.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">5</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span></code></pre>
<p>this time, each query that returns an item will return a <code>tuple</code>, and you will need to convert such tuple into a dict.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token keyword">def</span> <span class="token function">tuple_to_dict</span><span class="token punctuation">(</span>mytuple<span class="token punctuation">)</span><span class="token punctuation">:</span>
mydict <span class="token operator">=</span> <span class="token builtin">dict</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
mydict<span class="token punctuation">[</span><span class="token string">'id'</span><span class="token punctuation">]</span> <span class="token operator">=</span> mytuple<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
mydict<span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span> <span class="token operator">=</span> mytuple<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span>
mydict<span class="token punctuation">[</span><span class="token string">'price'</span><span class="token punctuation">]</span> <span class="token operator">=</span> mytuple<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span>
mydict<span class="token punctuation">[</span><span class="token string">'quantity'</span><span class="token punctuation">]</span> <span class="token operator">=</span> mytuple<span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span>
<span class="token keyword">return</span> mydict</code></pre>
<p>In a SQL database, <em>Read</em> operations are performed with <code>SELECT</code> statements.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token decorator annotation punctuation">@connect</span>
<span class="token keyword">def</span> <span class="token function">select_one</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> item_name<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
table_name <span class="token operator">=</span> scrub<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
item_name <span class="token operator">=</span> scrub<span class="token punctuation">(</span>item_name<span class="token punctuation">)</span>
sql <span class="token operator">=</span> <span class="token string">'SELECT * FROM {} WHERE name="{}"'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">,</span> item_name<span class="token punctuation">)</span>
c <span class="token operator">=</span> conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>sql<span class="token punctuation">)</span>
result <span class="token operator">=</span> c<span class="token punctuation">.</span>fetchone<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> result <span class="token keyword">is</span> <span class="token keyword">not</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> tuple_to_dict<span class="token punctuation">(</span>result<span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored<span class="token punctuation">(</span>
<span class="token string">'Can\'t read "{}" because it\'s not stored in table "{}"'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item_name<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@connect</span>
<span class="token keyword">def</span> <span class="token function">select_all</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
table_name <span class="token operator">=</span> scrub<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
sql <span class="token operator">=</span> <span class="token string">'SELECT * FROM {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
c <span class="token operator">=</span> conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>sql<span class="token punctuation">)</span>
results <span class="token operator">=</span> c<span class="token punctuation">.</span>fetchall<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token builtin">map</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span> tuple_to_dict<span class="token punctuation">(</span>x<span class="token punctuation">)</span><span class="token punctuation">,</span> results<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>Let’s now add the <em>Update</em> operation.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token decorator annotation punctuation">@connect</span>
<span class="token keyword">def</span> <span class="token function">update_one</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
table_name <span class="token operator">=</span> scrub<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
sql_check <span class="token operator">=</span> <span class="token string">'SELECT EXISTS(SELECT 1 FROM {} WHERE name=? LIMIT 1)'</span>\
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
sql_update <span class="token operator">=</span> <span class="token string">'UPDATE {} SET price=?, quantity=? WHERE name=?'</span>\
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
c <span class="token operator">=</span> conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>sql_check<span class="token punctuation">,</span> <span class="token punctuation">(</span>name<span class="token punctuation">,</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment"># we need the comma</span>
result <span class="token operator">=</span> c<span class="token punctuation">.</span>fetchone<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> result<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">:</span>
c<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>sql_update<span class="token punctuation">,</span> <span class="token punctuation">(</span>price<span class="token punctuation">,</span> quantity<span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">)</span>
conn<span class="token punctuation">.</span>commit<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored<span class="token punctuation">(</span>
<span class="token string">'Can\'t update "{}" because it\'s not stored in table "{}"'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>And finally, <em>Delete</em>.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token decorator annotation punctuation">@connect</span>
<span class="token keyword">def</span> <span class="token function">delete_one</span><span class="token punctuation">(</span>conn<span class="token punctuation">,</span> name<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
table_name <span class="token operator">=</span> scrub<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
sql_check <span class="token operator">=</span> <span class="token string">'SELECT EXISTS(SELECT 1 FROM {} WHERE name=? LIMIT 1)'</span>\
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
table_name <span class="token operator">=</span> scrub<span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
sql_delete <span class="token operator">=</span> <span class="token string">'DELETE FROM {} WHERE name=?'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>table_name<span class="token punctuation">)</span>
c <span class="token operator">=</span> conn<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>sql_check<span class="token punctuation">,</span> <span class="token punctuation">(</span>name<span class="token punctuation">,</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment"># we need the comma</span>
result <span class="token operator">=</span> c<span class="token punctuation">.</span>fetchone<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> result<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">:</span>
c<span class="token punctuation">.</span>execute<span class="token punctuation">(</span>sql_delete<span class="token punctuation">,</span> <span class="token punctuation">(</span>name<span class="token punctuation">,</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment"># we need the comma</span>
conn<span class="token punctuation">.</span>commit<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored<span class="token punctuation">(</span>
<span class="token string">'Can\'t delete "{}" because it\'s not stored in table "{}"'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>Let’s put everything together and see if these CRUD operations are correct!</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># sqlite_backend.py</span>
<span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
table_name <span class="token operator">=</span> <span class="token string">'items'</span>
conn <span class="token operator">=</span> connect_to_db<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment"># in-memory database</span>
<span class="token comment"># conn = connect_to_db(DB_name) # physical database (i.e. a .db file)</span>
create_table<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token punctuation">)</span>
my_items <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">20</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'wine'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">10.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">5</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
<span class="token comment"># CREATE</span>
insert_many<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> my_items<span class="token punctuation">,</span> table_name<span class="token operator">=</span><span class="token string">'items'</span><span class="token punctuation">)</span>
insert_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'beer'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">2.0</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">5</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span><span class="token string">'items'</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to insert an object already stored we get an ItemAlreadyStored</span>
<span class="token comment"># exception</span>
<span class="token comment"># insert_one(conn, 'milk', price=1.0, quantity=3, table_name='items')</span>
<span class="token comment"># READ</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'SELECT milk'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>select_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span><span class="token string">'items'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'SELECT all'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>select_all<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token operator">=</span><span class="token string">'items'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to select an object not stored we get an ItemNotStored exception</span>
<span class="token comment"># print(select_one(conn, 'pizza', table_name='items'))</span>
<span class="token comment"># conn.close() # the decorator @connect will reopen the connection</span>
<span class="token comment"># UPDATE</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'UPDATE bread, SELECT bread'</span><span class="token punctuation">)</span>
update_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">1.5</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">5</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span><span class="token string">'items'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>select_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span><span class="token string">'items'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to update an object not stored we get an ItemNotStored exception</span>
<span class="token comment"># print('UPDATE pizza')</span>
<span class="token comment"># update_one(conn, 'pizza', price=1.5, quantity=5, table_name='items')</span>
<span class="token comment"># DELETE</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'DELETE beer, SELECT all'</span><span class="token punctuation">)</span>
delete_one<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> <span class="token string">'beer'</span><span class="token punctuation">,</span> table_name<span class="token operator">=</span><span class="token string">'items'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>select_all<span class="token punctuation">(</span>conn<span class="token punctuation">,</span> table_name<span class="token operator">=</span><span class="token string">'items'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to delete an object not stored we get an ItemNotStored exception</span>
<span class="token comment"># print('DELETE fish')</span>
<span class="token comment"># delete_one(conn, 'fish', table_name='items')</span>
<span class="token comment"># save (commit) the changes</span>
<span class="token comment"># conn.commit()</span>
<span class="token comment"># close connection</span>
conn<span class="token punctuation">.</span>close<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>You can execute the <code>main</code> function with this line:</p>
<pre class="language-python"><code class="language-python">conn <span class="token operator">=</span> connect_to_db<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment"># in-memory database</span></code></pre>
<p>or this one:</p>
<pre class="language-python"><code class="language-python">conn <span class="token operator">=</span> connect_to_db<span class="token punctuation">(</span>DB_name<span class="token punctuation">)</span> <span class="token comment"># physical database (i.e. a .db file)</span></code></pre>
<p>Th former creates an in-memory database, so it’s faster and does not create any file. The latter creates a <code>.db</code> file that you can explore with tools like <a href="https://sqlitebrowser.org/">DB Browser for SQLite</a> or even online viewers like <a href="https://inloop.github.io/sqlite-viewer/">this one</a>.</p>
<p><a><h2>Model</h2></a><br />
Now that all CRUD operations are implemented as simple functions, creating a class for a Model that uses a SQLite database as persistence layer is pretty straightforward.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># model_view_controller.py</span>
<span class="token keyword">import</span> sqlite_backend
<span class="token keyword">import</span> mvc_exceptions <span class="token keyword">as</span> mvc_exc
<span class="token keyword">class</span> <span class="token class-name">ModelSQLite</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> application_items<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_item_type <span class="token operator">=</span> <span class="token string">'product'</span>
self<span class="token punctuation">.</span>_connection <span class="token operator">=</span> sqlite_backend<span class="token punctuation">.</span>connect_to_db<span class="token punctuation">(</span>sqlite_backend<span class="token punctuation">.</span>DB_name<span class="token punctuation">)</span>
sqlite_backend<span class="token punctuation">.</span>create_table<span class="token punctuation">(</span>self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> self<span class="token punctuation">.</span>_item_type<span class="token punctuation">)</span>
self<span class="token punctuation">.</span>create_items<span class="token punctuation">(</span>application_items<span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">connection</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>_connection
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">item_type</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>_item_type
<span class="token decorator annotation punctuation">@item_type<span class="token punctuation">.</span>setter</span>
<span class="token keyword">def</span> <span class="token function">item_type</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> new_item_type<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_item_type <span class="token operator">=</span> new_item_type
<span class="token keyword">def</span> <span class="token function">create_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
sqlite_backend<span class="token punctuation">.</span>insert_one<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">create_items</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> items<span class="token punctuation">)</span><span class="token punctuation">:</span>
sqlite_backend<span class="token punctuation">.</span>insert_many<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> items<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">read_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> sqlite_backend<span class="token punctuation">.</span>select_one<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> name<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">read_items</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> sqlite_backend<span class="token punctuation">.</span>select_all<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">update_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
sqlite_backend<span class="token punctuation">.</span>update_one<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">delete_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">:</span>
sqlite_backend<span class="token punctuation">.</span>delete_one<span class="token punctuation">(</span>
self<span class="token punctuation">.</span>connection<span class="token punctuation">,</span> name<span class="token punctuation">,</span> table_name<span class="token operator">=</span>self<span class="token punctuation">.</span>item_type<span class="token punctuation">)</span></code></pre>
<p><a><h2>View and Controller</h2></a><br />
As I said last time, <code>View</code> and <code>Controller</code> are completely <strong>decoupled</strong> from the <code>Model</code> (and between themselves), so you don’t need to change anything in their implementation. If you need the code for these classes, see the <a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/">first article</a> in the series.</p>
<p>The only thing to do is to plug the <code>ModelSQLite</code> in the <code>Controller</code>.</p>
<p>Here is a snippet to test our small MVC application:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
my_items <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">20</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'wine'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">10.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">5</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
c <span class="token operator">=</span> Controller<span class="token punctuation">(</span>ModelSQLite<span class="token punctuation">(</span>my_items<span class="token punctuation">)</span><span class="token punctuation">,</span> View<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_items<span class="token punctuation">(</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_items<span class="token punctuation">(</span>bullet_points<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_item<span class="token punctuation">(</span><span class="token string">'chocolate'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_item<span class="token punctuation">(</span><span class="token string">'bread'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>insert_item<span class="token punctuation">(</span><span class="token string">'bread'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">1.0</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">5</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>insert_item<span class="token punctuation">(</span><span class="token string">'chocolate'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">2.0</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">10</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_item<span class="token punctuation">(</span><span class="token string">'chocolate'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>update_item<span class="token punctuation">(</span><span class="token string">'milk'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">1.2</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">20</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>update_item<span class="token punctuation">(</span><span class="token string">'ice cream'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">3.5</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">20</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>delete_item<span class="token punctuation">(</span><span class="token string">'fish'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>delete_item<span class="token punctuation">(</span><span class="token string">'bread'</span><span class="token punctuation">)</span>
c<span class="token punctuation">.</span>show_items<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token comment"># we close the current sqlite database connection explicitly</span>
<span class="token keyword">if</span> <span class="token builtin">type</span><span class="token punctuation">(</span>c<span class="token punctuation">.</span>model<span class="token punctuation">)</span> <span class="token keyword">is</span> ModelSQLite<span class="token punctuation">:</span>
sqlite_backend<span class="token punctuation">.</span>disconnect_from_db<span class="token punctuation">(</span>
sqlite_backend<span class="token punctuation">.</span>DB_name<span class="token punctuation">,</span> c<span class="token punctuation">.</span>model<span class="token punctuation">.</span>connection<span class="token punctuation">)</span>
<span class="token comment"># the sqlite backend understands that it needs to open a new connection</span>
c<span class="token punctuation">.</span>show_items<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p><a><h2>Conclusion</h2></a><br />
In this article we replaced <code>ModelBasic</code> with <code>ModelSQLite</code>. Thanks to the SQLite database we gained a persistence layer for our application, and thanks to the modular architecture of the MVC pattern we kept the same functionality without having to change a single line of code in the <code>View</code> or in the <code>Controller</code>.</p>
<p>In the next article we will use a very cool package called <a href="https://dataset.readthedocs.io/en/latest/">Dataset</a> to get rid of all these ugly SQL statements! We will be able to simplify all these database operations and make the code more pythonic.</p>
https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/MVC pattern in Python: Introduction and BasicModel2017-04-02T00:00:00Z2017-04-02T00:00:00Z
<p>If you have ever worked with Graphical User Interfaces or web frameworks (e.g. Django), chances are that you heard about the <em>Model-View-Controller</em> pattern.<br />
Since I wanted to understand and implement in Python the most popular patterns, I decided I had to implement a basic MVC from scratch.</p>
<p>This is the <strong>first</strong> article of a series of blog posts related to the MVC pattern. Here are the links to the other articles in the series:</p>
<ol>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/">MVC pattern in Python: Introduction and BasicModel</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-sqlite/">MVC pattern in Python: SQLite</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-dataset/">MVC pattern in Python: Dataset</a></li>
</ol>
<p><em>All code was written in Python 3.5. If you are using Python 2.7 you should be able to run it with a few minor changes.</em></p>
<hr />
<p>Table of contents</p>
<ol>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/#intro">Introduction</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/#crud">CRUD</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/#model">Model</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/#view">View</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/#controller">Controller</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/#test-run">Test Run</a></li>
<li><a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/#conclusion">Conclusion</a></li>
</ol>
<p><a><h2>Introduction</h2></a><br />
The three components of the MVC pattern are <strong>decoupled</strong> and they are responsible for different things:</p>
<ul>
<li>the <strong>Model</strong> manages the data and defines rules and behaviors. It represents the <a href="https://www.giacomodebidda.com/posts/mvc-pattern-in-python-introduction-and-basicmodel/WIKIPEDIA">business logic</a> of the application. The data can be stored in the Model itself or in a database (only the Model has access to the database).</li>
<li>the <strong>View</strong> presents the data to the user. A View can be any kind of output representation: a HTML page, a chart, a table, or even a simple text output. A View should never call its own methods; only a Controller should do it.</li>
<li>the <strong>Controller</strong> accepts user’s inputs and delegates data representation to a View and data handling to a Model.</li>
</ul>
<p>Since Model, View and Controller are <strong>decoupled</strong>, each one of the three can be extended, modified and replaced without having to rewrite the other two.</p>
<p><a><h2>CRUD</h2></a><br />
In order to understand how the MVC works I decided to implement a simple CRUD (Create, Read, Update, Delete) application.</p>
<p><em>A word of caution:</em> according to Wikipedia, <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">create, read, update, and delete</a> are the four basic functions of <em>persistent storage</em>. A persistance layer can be implemented with a database table, a XML file, a JSON, or even a CSV. However, in this first post I want to keep things as simple as possible, so I will create a MVC application that doesn’t have any persistent storage. You could argue that this is not really a CRUD application, but I hope that you will be satisfied with the next article, where I will implement the persistance layer with a SQLite database.</p>
<p>Let’s think about the inventory of a small grocery store. A typical product list would look like this:</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption></caption>
<thead class="">
<tr class=""><th scope="col" class="">Name</th><th scope="col" class="">Price</th><th scope="col" class="">Quantity</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">Bread</td><td class="">0.5</td><td class="">20</td></tr><tr class=""><td class="">Milk</td><td class="">1.0</td><td class="">10</td></tr><tr class=""><td class="">Wine</td><td class="">10.0</td><td class="">5</td></tr>
</tbody>
</table>
</div>
</div>
<p>In Python you can think about these items as a list of dictionaries.</p>
<pre class="language-python"><code class="language-python">my_items <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">20</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'wine'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">10.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">5</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span></code></pre>
<p>The list of items can be changed any time you perform one of the following operations:</p>
<ul>
<li><strong>create</strong> new items</li>
<li><strong>update</strong> existing items</li>
<li><strong>delete</strong> existing items</li>
</ul>
<p>The <strong>read</strong> operation does not modify anything in the list of items.</p>
<p>Instead of jumping straight into creating classes for Model, View and Controller, let’s try to implement each CRUD functionality in the simplest way possible. Keep in mind that we have to use a <code>global</code> variable to store the list of <code>items</code> because its state must be shared across all operations.</p>
<p>Create a python script and call it <code>basic_backend.py</code>.</p>
<p>Let’s start with the <em>Create</em> functionality.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># basic_backend.py</span>
items <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment"># global variable where we keep the data</span>
<span class="token keyword">def</span> <span class="token function">create_items</span><span class="token punctuation">(</span>app_items<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
items <span class="token operator">=</span> app_items
<span class="token keyword">def</span> <span class="token function">create_item</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
items<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> name<span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> price<span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> quantity<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>As you can see, <em>Create</em> operations don’t return anything. They just append new data to the global <code>items</code> list.</p>
<p>Let’s add a <em>Read</em> functionality.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># basic_backend.py</span>
<span class="token keyword">def</span> <span class="token function">read_item</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
myitems <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token builtin">filter</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span> x<span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span> <span class="token operator">==</span> name<span class="token punctuation">,</span> items<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> myitems<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
<span class="token keyword">def</span> <span class="token function">read_items</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
<span class="token keyword">return</span> <span class="token punctuation">[</span>item <span class="token keyword">for</span> item <span class="token keyword">in</span> items<span class="token punctuation">]</span></code></pre>
<p>Actually there are already a couple of problems with this implementation:</p>
<ol>
<li>if you create the same element twice, you get a duplicate in the <code>items</code> list;</li>
<li>if you try to read a non-existing item, you get an <code>IndexError</code> exception.</li>
</ol>
<p>These issues are very easy to fix, but I think it’s important to pause for a moment and think about why they are a problem for your application, and how you want to handle these exceptions.</p>
<ol>
<li><em>duplicate item</em> -> you don’t want duplicates in the list of items. As soon as the user tries to append an item that already exists, you want to prevent this operation and return her a message that the item was <em>already stored</em>.</li>
<li><em>non-existing item</em> -> obviously you can’t read an item which is not currently available, so you want to tell the user that the item is <em>not stored</em>.</li>
</ol>
<p>It’s important to think about these issues right now because we want to create specific exceptions for these situations.</p>
<p>In this example <code>items</code> is just a list, but if it were a table in a SQLite database, these conditions would trigger different exceptions (e.g. adding a duplicate could raise an <code>IntegrityError</code> exception). You want to create exceptions that are at a higher level of abstraction, and implement the exception handling for each persistance layer. If this sounds confusing right now, just bear with me and I hope it will make more sense in the next article.</p>
<p>Let’s create these exceptions in a new file and call it <code>mvc_exceptions.py</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># mvc_exceptions.py</span>
<span class="token keyword">class</span> <span class="token class-name">ItemAlreadyStored</span><span class="token punctuation">(</span>Exception<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token keyword">class</span> <span class="token class-name">ItemNotStored</span><span class="token punctuation">(</span>Exception<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span></code></pre>
<p>Let’s update the code in <code>basic_backend.py</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> mvc_exceptions <span class="token keyword">as</span> mvc_exc
items <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">create_item</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
results <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token builtin">filter</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span> x<span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span> <span class="token operator">==</span> name<span class="token punctuation">,</span> items<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> results<span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemAlreadyStored<span class="token punctuation">(</span><span class="token string">'"{}" already stored!'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
items<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> name<span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> price<span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> quantity<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">create_items</span><span class="token punctuation">(</span>app_items<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
items <span class="token operator">=</span> app_items
<span class="token keyword">def</span> <span class="token function">read_item</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
myitems <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token builtin">filter</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span> x<span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span> <span class="token operator">==</span> name<span class="token punctuation">,</span> items<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> myitems<span class="token punctuation">:</span>
<span class="token keyword">return</span> myitems<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored<span class="token punctuation">(</span>
<span class="token string">'Can\'t read "{}" because it\'s not stored'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">read_items</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
<span class="token keyword">return</span> <span class="token punctuation">[</span>item <span class="token keyword">for</span> item <span class="token keyword">in</span> items<span class="token punctuation">]</span></code></pre>
<p>Now, if you try to create an item that already exists, you get a <code>ItemAlreadyStored</code> exception, and if you try to read an item that is not stored, you get a <code>ItemNotStored</code> exception.</p>
<p>Let’s now add the <em>Update</em> and <em>Delete</em> functionalities.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># basic_backend.py</span>
<span class="token keyword">def</span> <span class="token function">update_item</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
<span class="token comment"># Python 3.x removed tuple parameters unpacking (PEP 3113), so we have to do it manually (i_x is a tuple, idxs_items is a list of tuples)</span>
idxs_items <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span>
<span class="token builtin">filter</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> i_x<span class="token punctuation">:</span> i_x<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span> <span class="token operator">==</span> name<span class="token punctuation">,</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>items<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> idxs_items<span class="token punctuation">:</span>
i<span class="token punctuation">,</span> item_to_update <span class="token operator">=</span> idxs_items<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> idxs_items<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span>
items<span class="token punctuation">[</span>i<span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> name<span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> price<span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> quantity<span class="token punctuation">}</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored<span class="token punctuation">(</span>
<span class="token string">'Can\'t update "{}" because it\'s not stored'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">delete_item</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">global</span> items
<span class="token comment"># Python 3.x removed tuple parameters unpacking (PEP 3113), so we have to do it manually (i_x is a tuple, idxs_items is a list of tuples)</span>
idxs_items <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span>
<span class="token builtin">filter</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> i_x<span class="token punctuation">:</span> i_x<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'name'</span><span class="token punctuation">]</span> <span class="token operator">==</span> name<span class="token punctuation">,</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>items<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> idxs_items<span class="token punctuation">:</span>
i<span class="token punctuation">,</span> item_to_delete <span class="token operator">=</span> idxs_items<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> idxs_items<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span>
<span class="token keyword">del</span> items<span class="token punctuation">[</span>i<span class="token punctuation">]</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored<span class="token punctuation">(</span>
<span class="token string">'Can\'t delete "{}" because it\'s not stored'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>Basically these operations represent the business logic of the application. Let’s test them!</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># basic_backend.py</span>
<span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
my_items <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">20</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'wine'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">10.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">5</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
<span class="token comment"># CREATE</span>
create_items<span class="token punctuation">(</span>my_items<span class="token punctuation">)</span>
create_item<span class="token punctuation">(</span><span class="token string">'beer'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">3.0</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">15</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to re-create an object we get an ItemAlreadyStored exception</span>
<span class="token comment"># create_item('beer', price=2.0, quantity=10)</span>
<span class="token comment"># READ</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'READ items'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>read_items<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to read an object not stored we get an ItemNotStored exception</span>
<span class="token comment"># print('READ chocolate')</span>
<span class="token comment"># print(read_item('chocolate'))</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'READ bread'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>read_item<span class="token punctuation">(</span><span class="token string">'bread'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># UPDATE</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'UPDATE bread'</span><span class="token punctuation">)</span>
update_item<span class="token punctuation">(</span><span class="token string">'bread'</span><span class="token punctuation">,</span> price<span class="token operator">=</span><span class="token number">2.0</span><span class="token punctuation">,</span> quantity<span class="token operator">=</span><span class="token number">30</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>read_item<span class="token punctuation">(</span><span class="token string">'bread'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to update an object not stored we get an ItemNotStored exception</span>
<span class="token comment"># print('UPDATE chocolate')</span>
<span class="token comment"># update_item('chocolate', price=10.0, quantity=20)</span>
<span class="token comment"># DELETE</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'DELETE beer'</span><span class="token punctuation">)</span>
delete_item<span class="token punctuation">(</span><span class="token string">'beer'</span><span class="token punctuation">)</span>
<span class="token comment"># if we try to delete an object not stored we get an ItemNotStored exception</span>
<span class="token comment"># print('DELETE chocolate')</span>
<span class="token comment"># delete_item('chocolate')</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'READ items'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>read_items<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p><a><h2>Model</h2></a><br />
Now that all CRUD operations are implemented as simple functions, it’s very easy to “package” them into a single class. As you can see, there is no mention of <code>View</code> or <code>Controller</code> in the <code>ModelBasic</code> class.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># model_view_controller.py</span>
<span class="token keyword">import</span> basic_backend
<span class="token keyword">import</span> mvc_exceptions <span class="token keyword">as</span> mvc_exc
<span class="token keyword">class</span> <span class="token class-name">ModelBasic</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> application_items<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_item_type <span class="token operator">=</span> <span class="token string">'product'</span>
self<span class="token punctuation">.</span>create_items<span class="token punctuation">(</span>application_items<span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">item_type</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>_item_type
<span class="token decorator annotation punctuation">@item_type<span class="token punctuation">.</span>setter</span>
<span class="token keyword">def</span> <span class="token function">item_type</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> new_item_type<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_item_type <span class="token operator">=</span> new_item_type
<span class="token keyword">def</span> <span class="token function">create_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
basic_backend<span class="token punctuation">.</span>create_item<span class="token punctuation">(</span>name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">create_items</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> items<span class="token punctuation">)</span><span class="token punctuation">:</span>
basic_backend<span class="token punctuation">.</span>create_items<span class="token punctuation">(</span>items<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">read_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> basic_backend<span class="token punctuation">.</span>read_item<span class="token punctuation">(</span>name<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">read_items</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> basic_backend<span class="token punctuation">.</span>read_items<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">update_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
basic_backend<span class="token punctuation">.</span>update_item<span class="token punctuation">(</span>name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">delete_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">:</span>
basic_backend<span class="token punctuation">.</span>delete_item<span class="token punctuation">(</span>name<span class="token punctuation">)</span></code></pre>
<p><a><h2>View</h2></a><br />
Now that the business logic is ready, let’s focus on the presentation layer. In this tutorial the data is presented to the user in a python shell, so this is definitely not something that you would use in a real application. However, the important thing to notice is that there is no logic in the <code>View</code> class, and all of its methods are <strong>normal functions</strong> (see the <code>@staticmethod</code> decorator). Also, there is no mention of the other two components of the MVC pattern. This means that if you want to design a fancy UI for your application, you just have to replace the <code>View</code> class.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># model_view_controller.py</span>
<span class="token keyword">class</span> <span class="token class-name">View</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">show_bullet_point_list</span><span class="token punctuation">(</span>item_type<span class="token punctuation">,</span> items<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- {} LIST ---'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item_type<span class="token punctuation">.</span>upper<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> item <span class="token keyword">in</span> items<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'* {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">show_number_point_list</span><span class="token punctuation">(</span>item_type<span class="token punctuation">,</span> items<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- {} LIST ---'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item_type<span class="token punctuation">.</span>upper<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> i<span class="token punctuation">,</span> item <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>items<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}. {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>i<span class="token operator">+</span><span class="token number">1</span><span class="token punctuation">,</span> item<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">show_item</span><span class="token punctuation">(</span>item_type<span class="token punctuation">,</span> item<span class="token punctuation">,</span> item_info<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'//////////////////////////////////////////////////////////////'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Good news, we have some {}!'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item<span class="token punctuation">.</span>upper<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{} INFO: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item_type<span class="token punctuation">.</span>upper<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> item_info<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'//////////////////////////////////////////////////////////////'</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">display_missing_item_error</span><span class="token punctuation">(</span>item<span class="token punctuation">,</span> err<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'**************************************************************'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'We are sorry, we have no {}!'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item<span class="token punctuation">.</span>upper<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>err<span class="token punctuation">.</span>args<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'**************************************************************'</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">display_item_already_stored_error</span><span class="token punctuation">(</span>item<span class="token punctuation">,</span> item_type<span class="token punctuation">,</span> err<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'**************************************************************'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Hey! We already have {} in our {} list!'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item<span class="token punctuation">.</span>upper<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> item_type<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>err<span class="token punctuation">.</span>args<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'**************************************************************'</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">display_item_not_yet_stored_error</span><span class="token punctuation">(</span>item<span class="token punctuation">,</span> item_type<span class="token punctuation">,</span> err<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'**************************************************************'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'We don\'t have any {} in our {} list. Please insert it first!'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item<span class="token punctuation">.</span>upper<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> item_type<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>err<span class="token punctuation">.</span>args<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'**************************************************************'</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">display_item_stored</span><span class="token punctuation">(</span>item<span class="token punctuation">,</span> item_type<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Hooray! We have just added some {} to our {} list!'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item<span class="token punctuation">.</span>upper<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> item_type<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++'</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">display_change_item_type</span><span class="token punctuation">(</span>older<span class="token punctuation">,</span> newer<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- --- --- --- --- --- --- --- --- --- --'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Change item type from "{}" to "{}"'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>older<span class="token punctuation">,</span> newer<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- --- --- --- --- --- --- --- --- --- --'</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">display_item_updated</span><span class="token punctuation">(</span>item<span class="token punctuation">,</span> o_price<span class="token punctuation">,</span> o_quantity<span class="token punctuation">,</span> n_price<span class="token punctuation">,</span> n_quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- --- --- --- --- --- --- --- --- --- --'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Change {} price: {} --> {}'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item<span class="token punctuation">,</span> o_price<span class="token punctuation">,</span> n_price<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Change {} quantity: {} --> {}'</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>item<span class="token punctuation">,</span> o_quantity<span class="token punctuation">,</span> n_quantity<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--- --- --- --- --- --- --- --- --- --- --'</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">display_item_deletion</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--------------------------------------------------------------'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'We have just removed {} from our list'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'--------------------------------------------------------------'</span><span class="token punctuation">)</span></code></pre>
<p><a><h2>Controller</h2></a><br />
Finally, now that rules and logic (the Model) and information representation (the View) are done, we can focus on the <code>Controller</code>.<br />
As you can see, when you instantiate a <code>Controller</code> you have to specify a Model and a View. However, this is just <em>composition</em>, so whenever you want to use a different Model, and/or a different View, you just have to plug them in when you instantiate the Controller. The Controller accepts user’s inputs and <em>delegates</em> data representation to the View and data handling to the Model.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># model_view_controller.py</span>
<span class="token keyword">class</span> <span class="token class-name">Controller</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> model<span class="token punctuation">,</span> view<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>model <span class="token operator">=</span> model
self<span class="token punctuation">.</span>view <span class="token operator">=</span> view
<span class="token keyword">def</span> <span class="token function">show_items</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> bullet_points<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
items <span class="token operator">=</span> self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>read_items<span class="token punctuation">(</span><span class="token punctuation">)</span>
item_type <span class="token operator">=</span> self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>item_type
<span class="token keyword">if</span> bullet_points<span class="token punctuation">:</span>
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>show_bullet_point_list<span class="token punctuation">(</span>item_type<span class="token punctuation">,</span> items<span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>show_number_point_list<span class="token punctuation">(</span>item_type<span class="token punctuation">,</span> items<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">show_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> item_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
item <span class="token operator">=</span> self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>read_item<span class="token punctuation">(</span>item_name<span class="token punctuation">)</span>
item_type <span class="token operator">=</span> self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>item_type
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>show_item<span class="token punctuation">(</span>item_type<span class="token punctuation">,</span> item_name<span class="token punctuation">,</span> item<span class="token punctuation">)</span>
<span class="token keyword">except</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored <span class="token keyword">as</span> e<span class="token punctuation">:</span>
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>display_missing_item_error<span class="token punctuation">(</span>item_name<span class="token punctuation">,</span> e<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">insert_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">assert</span> price <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token string">'price must be greater than 0'</span>
<span class="token keyword">assert</span> quantity <span class="token operator">>=</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token string">'quantity must be greater than or equal to 0'</span>
item_type <span class="token operator">=</span> self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>item_type
<span class="token keyword">try</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>create_item<span class="token punctuation">(</span>name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span>
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>display_item_stored<span class="token punctuation">(</span>name<span class="token punctuation">,</span> item_type<span class="token punctuation">)</span>
<span class="token keyword">except</span> mvc_exc<span class="token punctuation">.</span>ItemAlreadyStored <span class="token keyword">as</span> e<span class="token punctuation">:</span>
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>display_item_already_stored_error<span class="token punctuation">(</span>name<span class="token punctuation">,</span> item_type<span class="token punctuation">,</span> e<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">update_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">assert</span> price <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token string">'price must be greater than 0'</span>
<span class="token keyword">assert</span> quantity <span class="token operator">>=</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token string">'quantity must be greater than or equal to 0'</span>
item_type <span class="token operator">=</span> self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>item_type
<span class="token keyword">try</span><span class="token punctuation">:</span>
older <span class="token operator">=</span> self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>read_item<span class="token punctuation">(</span>name<span class="token punctuation">)</span>
self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>update_item<span class="token punctuation">(</span>name<span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span>
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>display_item_updated<span class="token punctuation">(</span>
name<span class="token punctuation">,</span> older<span class="token punctuation">[</span><span class="token string">'price'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> older<span class="token punctuation">[</span><span class="token string">'quantity'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> price<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span>
<span class="token keyword">except</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored <span class="token keyword">as</span> e<span class="token punctuation">:</span>
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>display_item_not_yet_stored_error<span class="token punctuation">(</span>name<span class="token punctuation">,</span> item_type<span class="token punctuation">,</span> e<span class="token punctuation">)</span>
<span class="token comment"># if the item is not yet stored and we performed an update, we have</span>
<span class="token comment"># 2 options: do nothing or call insert_item to add it.</span>
<span class="token comment"># self.insert_item(name, price, quantity)</span>
<span class="token keyword">def</span> <span class="token function">update_item_type</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> new_item_type<span class="token punctuation">)</span><span class="token punctuation">:</span>
old_item_type <span class="token operator">=</span> self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>item_type
self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>item_type <span class="token operator">=</span> new_item_type
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>display_change_item_type<span class="token punctuation">(</span>old_item_type<span class="token punctuation">,</span> new_item_type<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">delete_item</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> name<span class="token punctuation">)</span><span class="token punctuation">:</span>
item_type <span class="token operator">=</span> self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>item_type
<span class="token keyword">try</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>model<span class="token punctuation">.</span>delete_item<span class="token punctuation">(</span>name<span class="token punctuation">)</span>
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>display_item_deletion<span class="token punctuation">(</span>name<span class="token punctuation">)</span>
<span class="token keyword">except</span> mvc_exc<span class="token punctuation">.</span>ItemNotStored <span class="token keyword">as</span> e<span class="token punctuation">:</span>
self<span class="token punctuation">.</span>view<span class="token punctuation">.</span>display_item_not_yet_stored_error<span class="token punctuation">(</span>name<span class="token punctuation">,</span> item_type<span class="token punctuation">,</span> e<span class="token punctuation">)</span></code></pre>
<p><a><h2>Test Run</h2></a><br />
Let’s see how everything works together!</p>
<p>Create some items and instantiate a <code>Controller</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># model_view_controller.py</span>
my_items <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'bread'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">20</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'milk'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">10</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span><span class="token string">'name'</span><span class="token punctuation">:</span> <span class="token string">'wine'</span><span class="token punctuation">,</span> <span class="token string">'price'</span><span class="token punctuation">:</span> <span class="token number">10.0</span><span class="token punctuation">,</span> <span class="token string">'quantity'</span><span class="token punctuation">:</span> <span class="token number">5</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
c <span class="token operator">=</span> Controller<span class="token punctuation">(</span>ModelBasic<span class="token punctuation">(</span>my_items<span class="token punctuation">)</span><span class="token punctuation">,</span> View<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>Show all items. The <code>bullet_points</code> parameter controls which view to display. When you call <code>c.show_items()</code> you get this:</p>
<pre class="language-shell"><code class="language-shell">--- PRODUCT LIST ---
<span class="token number">1</span>. <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token builtin class-name">:</span> <span class="token string">'bread'</span>, <span class="token string">'price'</span><span class="token builtin class-name">:</span> <span class="token number">0.5</span>, <span class="token string">'quantity'</span><span class="token builtin class-name">:</span> <span class="token number">20</span><span class="token punctuation">}</span>
<span class="token number">2</span>. <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token builtin class-name">:</span> <span class="token string">'milk'</span>, <span class="token string">'price'</span><span class="token builtin class-name">:</span> <span class="token number">1.0</span>, <span class="token string">'quantity'</span><span class="token builtin class-name">:</span> <span class="token number">10</span><span class="token punctuation">}</span>
<span class="token number">3</span>. <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token builtin class-name">:</span> <span class="token string">'wine'</span>, <span class="token string">'price'</span><span class="token builtin class-name">:</span> <span class="token number">10.0</span>, <span class="token string">'quantity'</span><span class="token builtin class-name">:</span> <span class="token number">5</span><span class="token punctuation">}</span></code></pre>
<p>and when you call <code>c.show_items(bullet_points=True)</code> you get this:</p>
<pre class="language-shell"><code class="language-shell">--- PRODUCT LIST ---
* <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token builtin class-name">:</span> <span class="token string">'bread'</span>, <span class="token string">'price'</span><span class="token builtin class-name">:</span> <span class="token number">0.5</span>, <span class="token string">'quantity'</span><span class="token builtin class-name">:</span> <span class="token number">20</span><span class="token punctuation">}</span>
* <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token builtin class-name">:</span> <span class="token string">'milk'</span>, <span class="token string">'price'</span><span class="token builtin class-name">:</span> <span class="token number">1.0</span>, <span class="token string">'quantity'</span><span class="token builtin class-name">:</span> <span class="token number">10</span><span class="token punctuation">}</span>
* <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token builtin class-name">:</span> <span class="token string">'wine'</span>, <span class="token string">'price'</span><span class="token builtin class-name">:</span> <span class="token number">10.0</span>, <span class="token string">'quantity'</span><span class="token builtin class-name">:</span> <span class="token number">5</span><span class="token punctuation">}</span></code></pre>
<p>When you call <code>c.show_item('chocolate')</code>, but there is no <code>'chocolate'</code>, you get this message:</p>
<pre class="language-shell"><code class="language-shell">**************************************************************
We are sorry, we have no CHOCOLATE<span class="token operator">!</span>
Can<span class="token string">'t read "chocolate" because it'</span>s not stored
**************************************************************</code></pre>
<p>Instead, when you call <code>c.show_item('bread')</code>, a different method of the <code>View</code> class is called, so you see a different output.</p>
<pre class="language-shell"><code class="language-shell">//////////////////////////////////////////////////////////////
Good news, we have some BREAD<span class="token operator">!</span>
PRODUCT INFO: <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token builtin class-name">:</span> <span class="token string">'bread'</span>, <span class="token string">'price'</span><span class="token builtin class-name">:</span> <span class="token number">0.5</span>, <span class="token string">'quantity'</span><span class="token builtin class-name">:</span> <span class="token number">20</span><span class="token punctuation">}</span>
//////////////////////////////////////////////////////////////</code></pre>
<p>You are prevented from inserting the same item a second time (e.g. you type <code>c.insert_item('bread', price=1.0, quantity=5)</code>).</p>
<pre class="language-shell"><code class="language-shell">**************************************************************
Hey<span class="token operator">!</span> We already have BREAD <span class="token keyword">in</span> our product list<span class="token operator">!</span>
<span class="token string">"bread"</span> already stored<span class="token operator">!</span>
**************************************************************</code></pre>
<p>But obviously you can add an item which was is not currently stored, for example with: <code>c.insert_item('chocolate', price=2.0, quantity=10)</code>.</p>
<pre class="language-shell"><code class="language-shell">++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Hooray<span class="token operator">!</span> We have just added some CHOCOLATE to our product list<span class="token operator">!</span>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</code></pre>
<p>So now you can call <code>c.show_item('chocolate')</code></p>
<pre class="language-shell"><code class="language-shell">//////////////////////////////////////////////////////////////
Good news, we have some CHOCOLATE<span class="token operator">!</span>
PRODUCT INFO: <span class="token punctuation">{</span><span class="token string">'name'</span><span class="token builtin class-name">:</span> <span class="token string">'chocolate'</span>, <span class="token string">'price'</span><span class="token builtin class-name">:</span> <span class="token number">2.0</span>, <span class="token string">'quantity'</span><span class="token builtin class-name">:</span> <span class="token number">10</span><span class="token punctuation">}</span>
//////////////////////////////////////////////////////////////</code></pre>
<p>When you update an existing item, for example with <code>c.update_item('milk', price=1.2, quantity=20)</code>, you get:</p>
<pre class="language-shell"><code class="language-shell">--- --- --- --- --- --- --- --- --- --- --
Change milk price: <span class="token number">1.0</span> --<span class="token operator">></span> <span class="token number">1.2</span>
Change milk quantity: <span class="token number">10</span> --<span class="token operator">></span> <span class="token number">20</span>
--- --- --- --- --- --- --- --- --- --- --</code></pre>
<p>And when you try to update some item which is not stored you get a warning. For example, <code>c.update_item('ice cream', price=3.5, quantity=20)</code> will result in the following message:</p>
<pre class="language-shell"><code class="language-shell">**************************************************************
We don<span class="token string">'t have any ICE CREAM in our product list. Please insert it first!
Can'</span>t <span class="token builtin class-name">read</span> <span class="token string">"ice cream"</span> because it's not stored
**************************************************************</code></pre>
<p>You get a warning also when you try to delete some item which is not stored.</p>
<p><code>c.delete_item('fish')</code></p>
<pre class="language-shell"><code class="language-shell">**************************************************************
We don<span class="token string">'t have any FISH in our product list. Please insert it first!
Can'</span>t delete <span class="token string">"fish"</span> because it's not stored
**************************************************************</code></pre>
<p>Finally, when you delete some item which is currently available, for example with <code>c.delete_item('bread')</code>, you get this:</p>
<pre class="language-shell"><code class="language-shell">--------------------------------------------------------------
We have just removed bread from our list
--------------------------------------------------------------</code></pre>
<p><a><h2>Conclusion</h2></a><br />
In this article we saw how to implement a very simple Model-View-Controller pattern. I hope that the implementation of all CRUD operations as simple functions made things a bit easier to understand. However, this MVC application would not be very useful in the real world because there is no <em>persistance layer</em> where to store the data.<br />
In the next article we will replace <code>ModelBasic</code> with a different class that uses a SQLite database. As I said, thanks to the flexible architecture provided by the MVC pattern, nothing is going to change neither in the <code>View</code>, nor in the <code>Controller</code>.</p>
https://www.giacomodebidda.com/posts/multiply-your-python-unit-test-cases-with-ddt/Multiply your Python Unit Test Cases with DDT2017-03-13T00:00:00Z2017-03-13T00:00:00Z
<p>DDT (Data-Driven Tests) is a small python module that allows you to multiply your unit test cases for free.<br />
The idea is pretty simple: you write a single test case and define some data samples, and DDT will generate a test case for each sample you provided.</p>
<p>You might ask: <em>“Why is that useful?”</em></p>
<p>Consider the following example, a simple test case without using DDT.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> unittest
<span class="token keyword">class</span> <span class="token class-name">TestWithoutDDT</span><span class="token punctuation">(</span>unittest<span class="token punctuation">.</span>TestCase<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">test_without_ddt</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">for</span> x <span class="token keyword">in</span> <span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">5</span><span class="token punctuation">]</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>assertGreater<span class="token punctuation">(</span>x<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span></code></pre>
<p>If you run this test you will get the following output:</p>
<pre class="language-python"><code class="language-python">Failure
Traceback <span class="token punctuation">(</span>most recent call last<span class="token punctuation">)</span><span class="token punctuation">:</span>
File <span class="token string">"/home/jack/Repos/design-patterns/test_ddt.py"</span><span class="token punctuation">,</span> line <span class="token number">47</span><span class="token punctuation">,</span> <span class="token keyword">in</span> test_without_ddt
self<span class="token punctuation">.</span>assertGreater<span class="token punctuation">(</span>x<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span>
AssertionError<span class="token punctuation">:</span> <span class="token operator">-</span><span class="token number">2</span> <span class="token keyword">not</span> greater than <span class="token number">0</span></code></pre>
<p>The test failed as soon as it asserted that <code>-2</code> is greater than <code>0</code> and then stopped. It didn’t consider <code>3</code>, <code>4</code>, or <code>-5</code>, so you don’t don’t whether the test would have passed for those inputs or not.</p>
<p>Now take a look at a very similar test with <code>DDT</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> unittest
<span class="token keyword">from</span> ddt <span class="token keyword">import</span> ddt<span class="token punctuation">,</span> data<span class="token punctuation">,</span> idata<span class="token punctuation">,</span> file_data<span class="token punctuation">,</span> unpack
<span class="token decorator annotation punctuation">@ddt</span>
<span class="token keyword">class</span> <span class="token class-name">TestDDTData</span><span class="token punctuation">(</span>unittest<span class="token punctuation">.</span>TestCase<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@data</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">5</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">test_with_ddt_data</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> x<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>assertGreater<span class="token punctuation">(</span>x<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span></code></pre>
<p>If you run this test you will get two distinct failures, for <code>-2</code> and <code>-5</code>.</p>
<pre class="language-python"><code class="language-python">Failure
Traceback <span class="token punctuation">(</span>most recent call last<span class="token punctuation">)</span><span class="token punctuation">:</span>
File <span class="token string">"/home/jack/.virtualenvs/design-patterns/lib/python3.5/site-packages/ddt.py"</span><span class="token punctuation">,</span> line <span class="token number">139</span><span class="token punctuation">,</span> <span class="token keyword">in</span> wrapper
<span class="token keyword">return</span> func<span class="token punctuation">(</span>self<span class="token punctuation">,</span> <span class="token operator">*</span>args<span class="token punctuation">,</span> <span class="token operator">**</span>kwargs<span class="token punctuation">)</span>
File <span class="token string">"/home/jack/Repos/design-patterns/test_ddt.py"</span><span class="token punctuation">,</span> line <span class="token number">15</span><span class="token punctuation">,</span> <span class="token keyword">in</span> test_with_ddt_data
self<span class="token punctuation">.</span>assertGreater<span class="token punctuation">(</span>x<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span>
AssertionError<span class="token punctuation">:</span> <span class="token operator">-</span><span class="token number">2</span> <span class="token keyword">not</span> greater than <span class="token number">0</span>
Failure
Traceback <span class="token punctuation">(</span>most recent call last<span class="token punctuation">)</span><span class="token punctuation">:</span>
File <span class="token string">"/home/jack/.virtualenvs/design-patterns/lib/python3.5/site-packages/ddt.py"</span><span class="token punctuation">,</span> line <span class="token number">139</span><span class="token punctuation">,</span> <span class="token keyword">in</span> wrapper
<span class="token keyword">return</span> func<span class="token punctuation">(</span>self<span class="token punctuation">,</span> <span class="token operator">*</span>args<span class="token punctuation">,</span> <span class="token operator">**</span>kwargs<span class="token punctuation">)</span>
File <span class="token string">"/home/jack/Repos/design-patterns/test_ddt.py"</span><span class="token punctuation">,</span> line <span class="token number">15</span><span class="token punctuation">,</span> <span class="token keyword">in</span> test_with_ddt_data
self<span class="token punctuation">.</span>assertGreater<span class="token punctuation">(</span>x<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span>
AssertionError<span class="token punctuation">:</span> <span class="token operator">-</span><span class="token number">5</span> <span class="token keyword">not</span> greater than <span class="token number">0</span></code></pre>
<p>This means that <strong>all of the inputs were tested</strong>, and two of them failed. Now you know why DDT is so cool!</p>
<p>It takes less than 2 minutes to read the <a href="https://ddt.readthedocs.io/en/latest/example.html">documentation</a>, and the examples are great!</p>
<p>The main reason why I like DDT is that it’s very easy to use: just decorate a test class with the <code>@ddt</code> decorator, and every test case you want with one of the decorators provided by this module. Here are the decorators available:</p>
<ul>
<li><code>@data</code>: contains as many arguments as the values you want to feed to the test. This values can be numbers, strings, tuples, etc. In the case of tuples, a cool feature is that you can <code>@unpack</code> them.</li>
<li><code>@file_data</code>: loads the test data from a JSON or YAML file.</li>
<li><code>@idata</code>: generates a new data sample from a generator function you defined somewhere in the code. (At this moment this decorator is not mentioned in the documentation).</li>
</ul>
<p>Here is an example with a generator function and <code>@idata</code>:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> unittest
<span class="token keyword">from</span> ddt <span class="token keyword">import</span> ddt<span class="token punctuation">,</span> idata
<span class="token keyword">def</span> <span class="token function">number_generator</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">for</span> x <span class="token keyword">in</span> <span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">5</span><span class="token punctuation">]</span><span class="token punctuation">:</span>
<span class="token keyword">yield</span> x
<span class="token decorator annotation punctuation">@ddt</span>
<span class="token keyword">class</span> <span class="token class-name">TestDDTGenerator</span><span class="token punctuation">(</span>unittest<span class="token punctuation">.</span>TestCase<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@idata</span><span class="token punctuation">(</span>number_generator<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">test_with_ddt_idata</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> x<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>assertGreater<span class="token punctuation">(</span>x<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span></code></pre>
<p>And here an example where the data is stored in an external file (JSON):</p>
<p><em>mydatafile.json</em></p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">]</span></code></pre>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> unittest
<span class="token keyword">from</span> ddt <span class="token keyword">import</span> ddt<span class="token punctuation">,</span> file_data
<span class="token decorator annotation punctuation">@ddt</span>
<span class="token keyword">class</span> <span class="token class-name">TestDDTDataFile</span><span class="token punctuation">(</span>unittest<span class="token punctuation">.</span>TestCase<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@file_data</span><span class="token punctuation">(</span><span class="token string">'mydatafile.json'</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">test_with_ddt_file_data</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> x<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>assertGreater<span class="token punctuation">(</span>x<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span></code></pre>
<p>Finally, an example where the data in unpacked:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> unittest
<span class="token keyword">from</span> ddt <span class="token keyword">import</span> ddt<span class="token punctuation">,</span> data<span class="token punctuation">,</span> unpack
<span class="token decorator annotation punctuation">@ddt</span>
<span class="token keyword">class</span> <span class="token class-name">TestDDTDataUnpack</span><span class="token punctuation">(</span>unittest<span class="token punctuation">.</span>TestCase<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@data</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token string">'hello'</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token string">'answer'</span><span class="token punctuation">,</span> <span class="token number">42</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@unpack</span>
<span class="token keyword">def</span> <span class="token function">test_with_ddt_data_unpack</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> some_string<span class="token punctuation">,</span> some_integer<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>assertIsInstance<span class="token punctuation">(</span>some_string<span class="token punctuation">,</span> <span class="token builtin">str</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>assertIsInstance<span class="token punctuation">(</span>some_integer<span class="token punctuation">,</span> <span class="token builtin">int</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-alternatives-to-ddt" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/multiply-your-python-unit-test-cases-with-ddt/#h-alternatives-to-ddt"><span aria-hidden="true">#</span></a> Alternatives to DDT</h2>
<p>The idea of test generators is not new, and there are at least two modules with similar capabilities: <a href="https://github.com/box/genty/blob/master/README.rst">genty</a> and <a href="https://pypi.python.org/pypi/unittest-data-provider/1.0.0">data-provider</a>. I opted for DDT because it seems better documented and more pythonic, but genty looks pretty good too. In particular, the <code>@genty_repeat</code> decorator might be a nice feature that is not available in DDT(even if one could probably obtain the same functionality by using the <a href="https://pypi.python.org/pypi/retrying">retrying</a> module).</p>
https://www.giacomodebidda.com/posts/first-steps-with-postgresql/First steps with PostgreSQL2017-03-13T00:00:00Z2017-03-13T00:00:00Z
<p>In a Django project, PostgreSQL is probably the most popular choice when it comes to deploy a database for a production environment. In this article I’ll go through the necessary steps to set it up on Ubuntu, along with a list of some basic commands to create databases and tables, as well as manage <em>roles</em> (i.e. users).</p>
<p>Here I will create a new role called <code>test_user</code> and a new database called <code>test_db</code>. You can pick different names if you want, but try to avoid mixing lowercase/uppercase. This is because if you create a user with a mix of lowercase and uppercase characters (e.g. <code>test_User</code>) you will need to type the double quotation marks every time.</p>
<h2 id="h-dependencies" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-dependencies"><span aria-hidden="true">#</span></a> Dependencies</h2>
<p>To satisfy the dependencies of the operative system, open a terminal and type:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">sudo</span> <span class="token function">apt-get</span> <span class="token function">install</span> postgresql postgresql-contrib libpq-dev python-dev</code></pre>
<p>For the python dependencies, I’d suggest to create a virtual environment with <a href="https://virtualenv.pypa.io/en/stable/">virtualenv</a>, or even better with <a href="https://www.giacomodebidda.com/posts/virtual-environments-with-virtualenvwrapper/">virtualenvwrapper</a>, and install the <code>psycopg2</code> driver:</p>
<pre class="language-shell"><code class="language-shell">pip <span class="token function">install</span> psycopg2</code></pre>
<h2 id="h-the-postgres-shell" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-the-postgres-shell"><span aria-hidden="true">#</span></a> The postgres shell</h2>
<p><code>psql</code> is the interactive terminal for working with PostgreSQL. You can launch it with <code>sudo -i -u postgres</code> and then <code>psql</code>.</p>
<p>Here are some useful commands when using the <code>psql</code> shell:</p>
<ul>
<li><strong>\du</strong> list all roles (namely the users) and their privileges;</li>
<li><strong>\l</strong> list all databases, their owners and access privileges;</li>
<li><strong>\c [DB NAME]</strong> connect to the <strong>[DB NAME]</strong> database with the user currently logged in;</li>
<li><strong>\d</strong> list all the tables of the database you are currently connected to;</li>
<li><strong>\d [TABLE NAME]</strong> show the schema of the table <strong>[TABLE NAME]</strong>, of the database you are currently connected to;</li>
<li><strong>\h</strong> help on SQL commands;</li>
<li><strong>?</strong> help on psql commands;</li>
<li><strong>\conninfo</strong> show some information about the current database connection (db name and user name);</li>
<li><strong>\q</strong> exit the psql shell.</li>
</ul>
<h2 id="h-create-a-new-user-aka-role" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-create-a-new-user-aka-role"><span aria-hidden="true">#</span></a> Create a new user (aka role)</h2>
<p>PostgreSQL comes with a default user called <code>postgres</code>, which is the root user. Let’s create a new user.</p>
<p>As user <code>postgres</code>, exit the <code>psql</code> shell and type:</p>
<pre class="language-shell"><code class="language-shell">createuser <span class="token parameter variable">--interactive</span></code></pre>
<p>Choose a username (e.g. <code>test_user</code>) and decide wheter this user should be a superuser, should be allowed to create new databases and/or new roles.</p>
<pre class="language-shell"><code class="language-shell">Enter name of role to add: test_user
Shall the new role be a superuser? <span class="token punctuation">(</span>y/n<span class="token punctuation">)</span> n
Shall the new role be allowed to create databases? <span class="token punctuation">(</span>y/n<span class="token punctuation">)</span> y
Shall the new role be allowed to create <span class="token function">more</span> new roles? <span class="token punctuation">(</span>y/n<span class="token punctuation">)</span> y</code></pre>
<p>If you want to check that the user was created correctly, go back to the <code>psql</code> shell and type <code>\du</code>.</p>
<h2 id="h-assign-a-password-to-your-new-user" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-assign-a-password-to-your-new-user"><span aria-hidden="true">#</span></a> Assign a password to your new user</h2>
<p>In the <code>psql</code> shell, type:</p>
<pre class="language-sql"><code class="language-sql"><span class="token keyword">ALTER</span> <span class="token keyword">USER</span> test_user <span class="token keyword">WITH</span> PASSWORD <span class="token string">'test_password'</span><span class="token punctuation">;</span></code></pre>
<p>Don’t forget the semi-colon and <a href="https://lerner.co.il/2013/11/30/quoting-postgresql/">avoid double quotation marks</a>.</p>
<h2 id="h-create-a-database" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-create-a-database"><span aria-hidden="true">#</span></a> Create a database</h2>
<p>In the <code>psql</code> shell, as user <code>postgres</code>, type:</p>
<pre class="language-sql"><code class="language-sql"><span class="token keyword">CREATE</span> <span class="token keyword">DATABASE</span> test_db<span class="token punctuation">;</span></code></pre>
<p>This command creates a new database called <code>test_db</code>. At this moment, only the user <code>postgres</code> can perform operations on this database.</p>
<h2 id="h-create-a-table" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-create-a-table"><span aria-hidden="true">#</span></a> Create a table</h2>
<pre class="language-sql"><code class="language-sql"><span class="token keyword">CREATE</span> <span class="token keyword">TABLE</span> items<span class="token punctuation">(</span>
item_id <span class="token keyword">serial</span> <span class="token keyword">PRIMARY</span> <span class="token keyword">KEY</span><span class="token punctuation">,</span>
item_description <span class="token keyword">text</span> <span class="token operator">NOT</span> <span class="token boolean">NULL</span><span class="token punctuation">,</span>
item_added <span class="token keyword">timestamp</span> <span class="token keyword">DEFAULT</span> <span class="token boolean">NULL</span>
<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="h-assign-privileges-to-testuser" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-assign-privileges-to-testuser"><span aria-hidden="true">#</span></a> Assign privileges to <code>test_user</code></h2>
<p>You need to allow your new user to modify the content of the <code>test_db</code> database. In order to do so you will need to grant him privileges on the database itself, and on the tables available in the database.</p>
<pre class="language-sql"><code class="language-sql"><span class="token keyword">GRANT</span> <span class="token keyword">ALL</span> <span class="token keyword">PRIVILEGES</span> <span class="token keyword">ON</span> <span class="token keyword">DATABASE</span> test_db <span class="token keyword">TO</span> test_user<span class="token punctuation">;</span></code></pre>
<p>The main reason to grant privileges on the database is to allow or revoke the connection to the database, but in order to allow for changes in the content of the database itself, the user <code>test_user</code> needs the privileges on all the tables he is allowed to modify. So, if you want to allow <code>test_user</code> to edit the contents of the table <code>items</code>, connect to the database with <code>\c test_db</code> and assign the privileges with:</p>
<pre class="language-sql"><code class="language-sql"><span class="token keyword">GRANT</span> <span class="token keyword">ALL</span> <span class="token keyword">PRIVILEGES</span> <span class="token keyword">ON</span> <span class="token keyword">TABLE</span> items <span class="token keyword">TO</span> test_user<span class="token punctuation">;</span></code></pre>
<p>Instead of typing the two aforementioned commands, you can achieve the same result with a single command:</p>
<pre class="language-sql"><code class="language-sql"><span class="token keyword">GRANT</span> <span class="token keyword">ALL</span> <span class="token keyword">ON</span> items <span class="token keyword">TO</span> test_user<span class="token punctuation">;</span></code></pre>
<p>Ok, now <code>test_user</code> can connect to <code>test_db</code> and change its content. If he is the only user allowed to work on this database, it make sense to make him the owner:</p>
<pre class="language-sql"><code class="language-sql"><span class="token keyword">ALTER</span> <span class="token keyword">DATABASE</span> test_db OWNER <span class="token keyword">TO</span> test_user<span class="token punctuation">;</span></code></pre>
<h2 id="h-connect-to-testdb-with-testuser" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-connect-to-testdb-with-testuser"><span aria-hidden="true">#</span></a> Connect to <code>test_db</code> with <code>test_user</code></h2>
<p>You are still connected as user <code>postgres</code>. Exit the <code>psql</code> shell with <code>\q</code> and log in with <code>test_user</code> (you will need to type the password).</p>
<pre class="language-shell"><code class="language-shell">psql <span class="token parameter variable">-h</span> localhost <span class="token parameter variable">-U</span> test_user <span class="token parameter variable">-d</span> test_db</code></pre>
<p>Here is what you should see if you type <code>\conninfo</code>:</p>
<pre class="language-shell"><code class="language-shell"><span class="token assign-left variable">test_db</span><span class="token operator">=</span><span class="token operator">></span> <span class="token punctuation">\</span>conninfo
You are connected to database <span class="token string">"test_db"</span> as user <span class="token string">"test_user"</span> on <span class="token function">host</span> <span class="token string">"localhost"</span> at port <span class="token string">"5432"</span><span class="token builtin class-name">.</span>
SSL connection <span class="token punctuation">(</span>protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: <span class="token number">256</span>, compression: off<span class="token punctuation">)</span></code></pre>
<h2 id="h-cleanup" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-cleanup"><span aria-hidden="true">#</span></a> Cleanup</h2>
<p>Probably you don’t want to keep the user, the database and the table we have just created, so let’s remove them. Keep in mind that you cannot drop an open database, nor you can drop it if you are not the database owner or a superuser. So, exit the <code>psql</code> shell and re-log into it as user <code>postgres</code> (you just have to type <code>psql</code>).</p>
<pre class="language-sql"><code class="language-sql"><span class="token keyword">DROP</span> <span class="token keyword">TABLE</span> items<span class="token punctuation">;</span>
<span class="token keyword">DROP</span> <span class="token keyword">DATABASE</span> test_db<span class="token punctuation">;</span>
<span class="token keyword">DROP</span> <span class="token keyword">USER</span> test_user<span class="token punctuation">;</span></code></pre>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/first-steps-with-postgresql/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<p>Here are some additional resources:</p>
<ul>
<li><a href="https://www.postgresguide.com/utilities/psql/">psql</a></li>
<li><a href="https://www.digitalocean.com/community/tutorials/how-to-secure-postgresql-on-an-ubuntu-vps">How to secure PostgreSQL on a Ubuntu VPS</a></li>
<li><a href="https://www.digitalocean.com/community/tutorials/how-to-back-up-restore-and-migrate-postgresql-databases-with-barman-on-centos-7">backup and restore postgreSQL databases with Barman</a></li>
</ul>
https://www.giacomodebidda.com/posts/factory-method-and-abstract-factory-in-python/Factory Method and Abstract Factory in Python2017-03-13T00:00:00Z2017-03-13T00:00:00Z
<p>Factory Method and Abstract Factory are <strong>creational</strong> design patterns and allow you to create objects without manually invoking a constructor. These patterns are closely related and share many similarities, that’s why I had a hard time in understanding the difference between the two.</p>
<p>A <a href="https://stackoverflow.com/questions/4209791/design-patterns-abstract-factory-vs-factory-method/4210168">concise answer on Stack Overflow</a> pointed me in the right direction, suggesting me to focus on the intent of these patterns. So, let’s see what problem Factory Method and Abstract Factory try to solve.</p>
<h2 id="h-factory-method" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/factory-method-and-abstract-factory-in-python/#h-factory-method"><span aria-hidden="true">#</span></a> Factory Method</h2>
<p>In Factory Method the client knows what she wants, but for some reason she can’t create the object directly. The reasons vary case-by-case: maybe she wants to use a common interface instead of manually instantiating the class she requires, or maybe she would need to pass a huge set of parameters to the constructor. Most of the time the client wants a single object, and this pattern relieves her of the responsability of creating this object directly.</p>
<p>Let’s see a very simple example:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">_Car</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token keyword">class</span> <span class="token class-name">_Bike</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token keyword">def</span> <span class="token function">factory_method</span><span class="token punctuation">(</span>product_type<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> product_type <span class="token operator">==</span> <span class="token string">'car'</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> _Car<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">elif</span> product_type <span class="token operator">==</span> <span class="token string">'bike'</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> _Bike<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> ValueError<span class="token punctuation">(</span><span class="token string">'Cannot make: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>product_type<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">for</span> product_type <span class="token keyword">in</span> <span class="token punctuation">(</span><span class="token string">'car'</span><span class="token punctuation">,</span> <span class="token string">'bike'</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
product <span class="token operator">=</span> factory_method<span class="token punctuation">(</span>product_type<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">(</span>product<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>Here the client knows that in the end she wants a bike or a car, but since these two classes are private she should not use them directly. Instead she will call <code>factory_method</code>, that will instantiate such classes for her. Here factory_method is just a function, and acts as a “virtual” constructor of either <code>_Car</code> or <code>_Bike</code>.</p>
<h2 id="h-abstract-factory" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/factory-method-and-abstract-factory-in-python/#h-abstract-factory"><span aria-hidden="true">#</span></a> Abstract Factory</h2>
<p>In Abstract Factory the client might not known what she wants, and how many objects she wants. This pattern provides an interface for creating families of related objects without the client having to specify the classes of the objects being created. In fact, the emphasys on families of related objects is the hallmark of the abstract factory pattern.</p>
<p>Let’s start from the client code, namely the main function.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> random
<span class="token keyword">import</span> inspect
<span class="token keyword">from</span> abc <span class="token keyword">import</span> ABC<span class="token punctuation">,</span> abstractmethod
<span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
triangles <span class="token operator">=</span> give_me_some_polygons<span class="token punctuation">(</span>TriangleFactory<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{} triangles'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>triangles<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> triangle <span class="token keyword">in</span> triangles<span class="token punctuation">:</span>
print_polygon<span class="token punctuation">(</span>triangle<span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>The function <code>give_me_some_polygons</code> is the <strong>interface</strong> between the client and the factory (in this example I wanted to be flexible and pass multiple abstract factories to this function, but this is just a detail). <code>give_me_some_polygons</code> calls the factory’s <code>make_polygon</code> method a random number of times, and returns a list of products to the client.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">give_me_some_polygons</span><span class="token punctuation">(</span>factories<span class="token punctuation">,</span> color<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Interface between the client and a Factory class.
Parameters
----------
factories : list, or abc.ABCMeta
list of factory classes, or a factory class
color : str
color to pass to the manufacturing method of the factory class.
Returns
-------
products : list
a list of objects manufactured by the Factory classes specified
"""</span>
<span class="token keyword">if</span> <span class="token keyword">not</span> <span class="token builtin">hasattr</span><span class="token punctuation">(</span>factories<span class="token punctuation">,</span> <span class="token string">'__len__'</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
factories <span class="token operator">=</span> <span class="token punctuation">[</span>factories<span class="token punctuation">]</span>
products <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> factory <span class="token keyword">in</span> factories<span class="token punctuation">:</span>
num <span class="token operator">=</span> random<span class="token punctuation">.</span>randint<span class="token punctuation">(</span><span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>num<span class="token punctuation">)</span><span class="token punctuation">:</span>
product <span class="token operator">=</span> factory<span class="token punctuation">.</span>make_polygon<span class="token punctuation">(</span>color<span class="token punctuation">)</span>
products<span class="token punctuation">.</span>append<span class="token punctuation">(</span>product<span class="token punctuation">)</span>
<span class="token keyword">return</span> products</code></pre>
<p>Let’s jump to <code>PolygonFactory</code>, the abstract factory at the top of the class hierarchy of factories. I decided to use the module <code>abc</code> and make <code>PolygonFactory</code> inherit from <code>ABC</code> so it’s clear that it’s an abstract class and cannot be instantiated. Since it’s a factory, it will manufacture some products. The list of products available for this class is a characteristic of the class itself, that’s why I used the <code>@classmethod</code> decorator.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">PolygonFactory</span><span class="token punctuation">(</span>ABC<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Basic abstract Factory class for making polygons (products).
This class has to be sublassed by a factory class that MUST implement
the "products" method.
A factory class can create many different polygon objects (products) without
exposing the instantiation logic to the client. Infact, since all methods of
this class are abstract, this class can't be instantiated at all! Also, each
subclass of PolygonFactory should implement the "products" method and keep
it abstract, so even that subclass can't be instatiated.
"""</span>
<span class="token decorator annotation punctuation">@classmethod</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">products</span><span class="token punctuation">(</span>cls<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Products that the factory can manufacture. Implement in subclass."""</span>
<span class="token keyword">pass</span>
<span class="token decorator annotation punctuation">@classmethod</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">make_polygon</span><span class="token punctuation">(</span>cls<span class="token punctuation">,</span> color<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Instantiate a random polygon from all the ones that are available.
This method creates an instance of a product randomly chosen from all
products that the factory class can manufacture. The 'color' property of
the manufactured object is reassigned here. Then the object is returned.
Parameters
----------
color : str
color to assign to the manufactured object. It replaces the color
assigned by the factory class.
Returns
-------
polygon : an instance of a class in cls.products()
polygon is the product manufactured by the factory class. It's one
of the products that the factory class can make.
"""</span>
product_name <span class="token operator">=</span> random<span class="token punctuation">.</span>choice<span class="token punctuation">(</span>cls<span class="token punctuation">.</span>products<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
this_module <span class="token operator">=</span> <span class="token builtin">__import__</span><span class="token punctuation">(</span>__name__<span class="token punctuation">)</span>
polygon_class <span class="token operator">=</span> <span class="token builtin">getattr</span><span class="token punctuation">(</span>this_module<span class="token punctuation">,</span> product_name<span class="token punctuation">)</span>
polygon <span class="token operator">=</span> polygon_class<span class="token punctuation">(</span>factory_name<span class="token operator">=</span>cls<span class="token punctuation">.</span>__name__<span class="token punctuation">)</span>
<span class="token keyword">if</span> color <span class="token keyword">is</span> <span class="token keyword">not</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
polygon<span class="token punctuation">.</span>color <span class="token operator">=</span> color
<span class="token keyword">return</span> polygon
<span class="token decorator annotation punctuation">@classmethod</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">color</span><span class="token punctuation">(</span>cls<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'black'</span></code></pre>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">TriangleFactory</span><span class="token punctuation">(</span>PolygonFactory<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Abstract Factory class for making triangles."""</span>
<span class="token decorator annotation punctuation">@classmethod</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">products</span><span class="token punctuation">(</span>cls<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token builtin">tuple</span><span class="token punctuation">(</span>
<span class="token punctuation">[</span><span class="token string">'_TriangleEquilateral'</span><span class="token punctuation">,</span> <span class="token string">'_TriangleIsosceles'</span><span class="token punctuation">,</span> <span class="token string">'_TriangleScalene'</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">QuadrilateralFactory</span><span class="token punctuation">(</span>PolygonFactory<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Abstract Factory class for making quadrilaterals."""</span>
<span class="token decorator annotation punctuation">@classmethod</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">products</span><span class="token punctuation">(</span>cls<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token builtin">tuple</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'_Square'</span><span class="token punctuation">,</span> <span class="token string">'_Rectangle'</span><span class="token punctuation">,</span> <span class="token string">'_ConvexQuadrilateral'</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<p>The only thing that a subclass of <code>PolygonFactory</code> has to do is to override the <code>products</code> method.<br />
It’s very easy to extend the suite of products a factory can manufacture. For example, <code>TriangleFactory</code> contains a list of triangles that it can create. If you want to create a new type of triangle, you just have to create a new triangle class (e.g. <code>_TriangleRectangle</code>) and add it to the list of triangles in the products method.<br />
Also, this design makes exchanging product families easy, because the specific class of the factory object appears only once in the application. The client is loosely coupled with the products, and if she needs a different family of products (e.g. quadrilaterals instead of triangles) she just needs to pass a different abstract factory to the <code>give_me_some_polygons</code> interface.</p>
<pre class="language-python"><code class="language-python">quadrilaterals <span class="token operator">=</span> give_me_some_polygons<span class="token punctuation">(</span>QuadrilateralFactory<span class="token punctuation">,</span> color<span class="token operator">=</span><span class="token string">'blue'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{} quadrilaterals'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>quadrilaterals<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> quadrilateral <span class="token keyword">in</span> quadrilaterals<span class="token punctuation">:</span>
print_polygon<span class="token punctuation">(</span>quadrilateral<span class="token punctuation">)</span></code></pre>
<p>The only portion of the code that knows what class to instantiate is the <code>make_polygon</code> method. It creates an instance of a product (e.g. <code>_Square</code>), randomly chosen from all the products that the factory class can manufacture, and returns it to the caller, which is the <code>give_me_some_polygons</code> function.</p>
<p>There are only a couple of things still missing for the code to run: a function to print the polygons, and, obviously, all the product classes.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">print_polygon</span><span class="token punctuation">(</span>polygon<span class="token punctuation">,</span> show_repr<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">,</span> show_hierarchy<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">(</span>polygon<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> show_repr<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token builtin">repr</span><span class="token punctuation">(</span>polygon<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> show_hierarchy<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>inspect<span class="token punctuation">.</span>getmro<span class="token punctuation">(</span>polygon<span class="token punctuation">.</span>__class__<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\n'</span><span class="token punctuation">)</span></code></pre>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">_Polygon</span><span class="token punctuation">(</span>ABC<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Basic abstract class for polygons.
This class is private because the client should not try to instantiate it.
The instantiation process should be carried out by a Factory class.
A _Polygon subclass MUST override ALL _Polygon's abstract methods, otherwise
a TypeError will be raised as soon as we try to instantiate that subclass.
"""</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> factory_name<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_color <span class="token operator">=</span> <span class="token string">'black'</span>
self<span class="token punctuation">.</span>_manufactured <span class="token operator">=</span> factory_name
<span class="token keyword">def</span> <span class="token function">__str__</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'{} {} manufactured by {} (perimeter: {}; area: {})'</span>\
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>color<span class="token punctuation">,</span> self<span class="token punctuation">.</span>__class__<span class="token punctuation">.</span>__name__<span class="token punctuation">,</span> self<span class="token punctuation">.</span>manufactured<span class="token punctuation">,</span>
self<span class="token punctuation">.</span>perimeter<span class="token punctuation">,</span> self<span class="token punctuation">.</span>area<span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">family</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">perimeter</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">area</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">color</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>_color
<span class="token decorator annotation punctuation">@color<span class="token punctuation">.</span>setter</span>
<span class="token keyword">def</span> <span class="token function">color</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> new_color<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_color <span class="token operator">=</span> new_color
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">manufactured</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>_manufactured
<span class="token decorator annotation punctuation">@manufactured<span class="token punctuation">.</span>setter</span>
<span class="token keyword">def</span> <span class="token function">manufactured</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> factory_name<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_manufactured <span class="token operator">=</span> factory_name
<span class="token keyword">class</span> <span class="token class-name">_Triangle</span><span class="token punctuation">(</span>_Polygon<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Basic concrete class for triangles."""</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">family</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'Triangles'</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">perimeter</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'a+b+c'</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">area</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'base*height/2'</span>
<span class="token keyword">class</span> <span class="token class-name">_TriangleEquilateral</span><span class="token punctuation">(</span>_Triangle<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">perimeter</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'3a'</span>
<span class="token keyword">class</span> <span class="token class-name">_TriangleIsosceles</span><span class="token punctuation">(</span>_Triangle<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">perimeter</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'2a+b'</span>
<span class="token keyword">class</span> <span class="token class-name">_TriangleScalene</span><span class="token punctuation">(</span>_Triangle<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token keyword">class</span> <span class="token class-name">_Quadrilateral</span><span class="token punctuation">(</span>_Polygon<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Basic concrete class for quadrilaterals."""</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">family</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'Quadrilaterals'</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">perimeter</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'a+b+c+d'</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">area</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'Bretschneider\'s formula'</span>
<span class="token keyword">class</span> <span class="token class-name">_Square</span><span class="token punctuation">(</span>_Quadrilateral<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">perimeter</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'4a'</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">area</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'a*a'</span>
<span class="token keyword">class</span> <span class="token class-name">_Rectangle</span><span class="token punctuation">(</span>_Quadrilateral<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">perimeter</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'2a+2b'</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">area</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'base*height'</span>
<span class="token keyword">class</span> <span class="token class-name">_ConvexQuadrilateral</span><span class="token punctuation">(</span>_Quadrilateral<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span></code></pre>
<p>You need the code? Grab it <a href="https://github.com/jackdbd/design-patterns">here</a>!</p>
https://www.giacomodebidda.com/posts/strategy-pattern-in-python/Strategy pattern in Python2017-01-28T00:00:00Z2017-01-28T00:00:00Z
<p><em>Strategy</em> (also known as <em>Policy</em>) is a behavioral design pattern that enables an algorithm’s behavior to be selected at runtime.</p>
<p>All implemented behaviors are either classes, methods, or functions, and they are usually called <em>strategies</em>. The portion of code that decides which strategy to adopt is called <em>context</em>.</p>
<p>Strategy follows two important principles:</p>
<ul>
<li>Open/closed principle: software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification.</li>
<li>Inversion of Control principle: custom-written portions of a program (e.g. a method in a subclass) receive the flow of control from a generic framework (e.g. a base class).</li>
</ul>
<p>Following these two principles is extremely useful when you want to design a common interface (the <em>Closed</em> part of the Open/Closed principle), but allow for changes in the implementation details (the <em>Open</em> part of the Open/Closed principle). Every time you want to program a new implementation, you pass it to the common interface without altering anything in the interface code, and you plug the client code to the interface. This way the client code is <em>loosely coupled</em>, namely it is coupled only with an abstraction (i.e. the common interface), not with the concrete implementations (i.e. the various strategies).</p>
<h2 id="h-how-to-implement-a-strategy-pattern-in-python%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/strategy-pattern-in-python/#h-how-to-implement-a-strategy-pattern-in-python%3F"><span aria-hidden="true">#</span></a> How to implement a Strategy pattern in Python?</h2>
<p>In programming languages like Java you can implement the Strategy pattern by creating a common (abstract) interface and subclassing it with a new class for each strategy. You can do the same in Python, like it’s done <a href="https://python-3-patterns-idioms-test.readthedocs.io/en/latest/FunctionObjects.html#strategy-choosing-the-algorithm-at-runtime">here</a>. However, you can also use a leaner approach: create a single <code>Strategy</code> class and replace a method of that class, at runtime, with a different function based on a given context.</p>
<p>Enough talking, let’s see some code!</p>
<h2 id="h-the-strategy-class" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/strategy-pattern-in-python/#h-the-strategy-class"><span aria-hidden="true">#</span></a> The <code>Strategy</code> class</h2>
<p>This class is the interface that the client code will use. It represents the <em>Closed</em> part of the Open/Closed principle, so it should not be modified.</p>
<p>If the client code does not provide a function <code>func</code>, <code>Strategy</code> will use the default algorithm, namely <code>execute</code> in this example.</p>
<p>In the <code>__init__</code> method we are taking advantage of the fact that Python supports <strong>higher order functions</strong>, namely we can pass a function as an argument to another function, and that Python functions are <strong>first class objects</strong>, so they can be assigned to variables, or stored in data structures (e.g. a dict).</p>
<p>If the client provides a function <code>func</code>, this will be passed to the <code>__init__</code> method and assigned to the <code>execute</code> method. This means that the <code>execute</code> method will be redefined when the <code>Strategy</code> class is <em>instantiated</em>.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">Strategy</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> func<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> func <span class="token keyword">is</span> <span class="token keyword">not</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>execute <span class="token operator">=</span> func
self<span class="token punctuation">.</span>name <span class="token operator">=</span> <span class="token string">'{}_{}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>__class__<span class="token punctuation">.</span>__name__<span class="token punctuation">,</span> func<span class="token punctuation">.</span>__name__<span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>name <span class="token operator">=</span> <span class="token string">'{}_default'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>__class__<span class="token punctuation">.</span>__name__<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">execute</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Default method'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}\n'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>That’s cool, but there is a problem: <code>func</code> is <strong>just a function</strong>, it contains no reference to the instance it is bound to (like a Python method defined by using the <code>@staticmethod</code> decorator). Within the redefined <code>execute</code> you cannot access other methods or attributes of the instance.</p>
<p>However with some Python magic and the help of the <code>types</code> module you can convert a normal function into a bound method, namely a function that contains a reference to the instance it is bound to.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> types
<span class="token keyword">class</span> <span class="token class-name">Strategy</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> func<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> func <span class="token keyword">is</span> <span class="token keyword">not</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
<span class="token comment"># take a function, bind it to this instance, and replace the default bound method 'execute' with this new bound method.</span>
self<span class="token punctuation">.</span>execute <span class="token operator">=</span> types<span class="token punctuation">.</span>MethodType<span class="token punctuation">(</span>func<span class="token punctuation">,</span> self<span class="token punctuation">)</span>
self<span class="token punctuation">.</span>name <span class="token operator">=</span> <span class="token string">'{}_{}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>__class__<span class="token punctuation">.</span>__name__<span class="token punctuation">,</span> func<span class="token punctuation">.</span>__name__<span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>name <span class="token operator">=</span> <span class="token string">'{}_default'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>__class__<span class="token punctuation">.</span>__name__<span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">execute</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Default method'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}\n'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<p>Now it’s time to implement some <em>strategies</em>.</p>
<h2 id="h-implement-the-strategies" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/strategy-pattern-in-python/#h-implement-the-strategies"><span aria-hidden="true">#</span></a> Implement the <em>strategies</em></h2>
<p>Let’s define a couple of <em>replacement strategies</em> for the default method <code>execute</code>. Don’t mind the <code>self</code> parameter, for now these ones are just regular functions. I decided to use <code>self</code> because I know that these functions, passed to the <code>Strategy</code>’s <code>__init__</code> method, will be bound to an instance of <code>Strategy</code> (this is done when the line <code>types.MethodType(func, self)</code> is executed).</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">execute_replacement1</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Replacement1 method'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}\n'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">execute_replacement2</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Replacement2 method'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}\n'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>name<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-select-a-strategy-at-runtime" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/strategy-pattern-in-python/#h-select-a-strategy-at-runtime"><span aria-hidden="true">#</span></a> Select a strategy at runtime</h2>
<p>In this simple example the part of the program which decides which strategy to use, namely the <em>context</em>, is the <code>main</code> function. As you can see, the three instances of <code>Strategy</code> are calling the same method.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
s0 <span class="token operator">=</span> Strategy<span class="token punctuation">(</span><span class="token punctuation">)</span>
s0<span class="token punctuation">.</span>execute<span class="token punctuation">(</span><span class="token punctuation">)</span>
s1 <span class="token operator">=</span> Strategy<span class="token punctuation">(</span>execute_replacement1<span class="token punctuation">)</span>
s1<span class="token punctuation">.</span>execute<span class="token punctuation">(</span><span class="token punctuation">)</span>
s2 <span class="token operator">=</span> Strategy<span class="token punctuation">(</span>execute_replacement2<span class="token punctuation">)</span>
s2<span class="token punctuation">.</span>execute<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>This is the output:</p>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> Default method
<span class="token operator">>></span><span class="token operator">></span> Strategy_default
<span class="token operator">>></span><span class="token operator">></span> Replacement1 method
<span class="token operator">>></span><span class="token operator">></span> Strategy_execute_replacement1
<span class="token operator">>></span><span class="token operator">></span> Replacement2 method
<span class="token operator">>></span><span class="token operator">></span> Strategy_execute_replacement2</code></pre>
<h2 id="h-an-even-simpler-strategy" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/strategy-pattern-in-python/#h-an-even-simpler-strategy"><span aria-hidden="true">#</span></a> An even simpler Strategy</h2>
<p>Sometimes you don’t even have to create a class, and you can achieve a similar result by assigning a function to an object, and then later calling that object. For example, this is also a Strategy:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">add</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> a <span class="token operator">+</span> b
<span class="token keyword">def</span> <span class="token function">subtract</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> a <span class="token operator">-</span> b
solve <span class="token operator">=</span> add
solve<span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span>
solve <span class="token operator">=</span> subtract
solve<span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span></code></pre>
<h2 id="h-strategy-vs-template-method" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/strategy-pattern-in-python/#h-strategy-vs-template-method"><span aria-hidden="true">#</span></a> Strategy VS Template Method</h2>
<p>Strategy and <a href="https://www.giacomodebidda.com/posts/template-method-pattern-in-python/">Template Method</a> are very similar and follow the same principles. The main difference is that in Template Method an implementation is chosen at <em>compile time</em> by <em>inheritance</em>, while in Strategy is chosen at <em>runtime</em> by <em>containment</em>. See also <a href="https://stackoverflow.com/a/669366">here</a> to understand the difference between these two behavioral design patterns.</p>
<p>You need the code? Grab it <a href="https://github.com/jackdbd/design-patterns">here</a>!</p>
https://www.giacomodebidda.com/posts/template-method-pattern-in-python/Template Method pattern in Python2017-01-27T00:00:00Z2017-01-27T00:00:00Z
<p>Behavioral design patterns are a family of patterns which focus on algorithms and the assignments of responsibilities between objects. They help in changing the behavior of the objects and reduce the coupling between them. In this article we will see a Python implementation of the <em>Template Method</em> pattern. Next time we will see the <a href="https://www.giacomodebidda.com/posts/strategy-pattern-in-python/"><em>Strategy</em> pattern</a>.</p>
<p>Template Method defines an algorithm’s skeleton in a base class, and lets subclasses redefine certain steps of the algorithm. The skeleton stays the same for all subclasses, and it defines which methods to call and when to call them. Some methods in the base class are just <em>placeholders</em> (or <em>hooks</em>), and they have to be overridden in each subclass.</p>
<p>A typical implementation of this pattern consists in an abstract class for the base class, and one or more concrete subclasses. The abstract class has the following methods:</p>
<ol>
<li>one <em>template method</em> that calls one or more methods. Subclasses should not override this particular method, which defines the skeleton of the algorithm. This method gives the name to this pattern, so in this Python implementation I called it <code>template_method</code>.</li>
<li>one or more abstract methods, which represent the customizable part of the algorithm. The abstract base class defines these methods, but they are just placeholders and have to be overriden by each subclass. Sometimes they are named “primitive operations”, “hooks” or “placeholders”. I called these methods <code>do_step_1</code> and <code>do_step_2</code>.</li>
<li>zero or more methods which are common in each subclass. These methods are implemented in the base class and should not be overridden in any subclass. In Java we could make these methods <code>final</code>. In Python we can place two underscore before the method name and “protect” them from being overridden by using <a href="https://docs.python.org/2/reference/expressions.html#atom-identifiers">private name mangling</a>. The method can still be overriden by a subclass, but with no effect, because the name mangling replaces the original method name with the class name where the method is originally defined, plus the original method name itself. In the code below, <code>__do_absolutely_this</code> is one of such methods.</li>
<li>one or more methods that have a default implementation in the base class, but that can be overridden by some sublasses. Here is the <code>do_something</code> method.</li>
</ol>
<p>Private name mangling for the <code>__do_absolutely_this</code> method. The original class name is prepended to the method name. That’s why we see <code>Algorithm__do_absolutely_this</code> even when we look at the list of attributes of class <code>AlgorithmA</code>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303221/private_name_mangling_nmkcrn.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303221/private_name_mangling_nmkcrn.png</a></p>
<h2 id="h-the-skeleton" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/template-method-pattern-in-python/#h-the-skeleton"><span aria-hidden="true">#</span></a> The Skeleton</h2>
<p>Here is the abstract base class which defines the <code>template_method</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> sys
<span class="token keyword">from</span> abc <span class="token keyword">import</span> ABC<span class="token punctuation">,</span> abstractmethod
<span class="token keyword">class</span> <span class="token class-name">Algorithm</span><span class="token punctuation">(</span>ABC<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">template_method</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Skeleton of operations to perform. DON'T override me.
The Template Method defines a skeleton of an algorithm in an operation,
and defers some steps to subclasses.
"""</span>
self<span class="token punctuation">.</span>__do_absolutely_this<span class="token punctuation">(</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>do_step_1<span class="token punctuation">(</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>do_step_2<span class="token punctuation">(</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>do_something<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">__do_absolutely_this</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Protected operation. DON'T override me."""</span>
this_method_name <span class="token operator">=</span> sys<span class="token punctuation">.</span>_getframe<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>f_code<span class="token punctuation">.</span>co_name
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}.{}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>__class__<span class="token punctuation">.</span>__name__<span class="token punctuation">,</span> this_method_name<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">do_step_1</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Primitive operation. You HAVE TO override me, I'm a placeholder."""</span>
<span class="token keyword">pass</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">do_step_2</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Primitive operation. You HAVE TO override me, I'm a placeholder."""</span>
<span class="token keyword">pass</span>
<span class="token keyword">def</span> <span class="token function">do_something</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Hook. You CAN override me, I'm NOT a placeholder."""</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'do something'</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-a-quick-note-about-the-abc-module" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/template-method-pattern-in-python/#h-a-quick-note-about-the-abc-module"><span aria-hidden="true">#</span></a> A quick note about the <code>ABC</code> module.</h2>
<p>Sometimes in Python you create “abstract” methods this way:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">some_method_in_base_class</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> NotImplementedError<span class="token punctuation">(</span><span class="token string">'Implement me in subclass'</span><span class="token punctuation">)</span></code></pre>
<p>However, you can still create instances of a subclass if you don’t override such method. You will simply get a <code>NotImplementedError</code> exception when you try to call that method on an instance of the subclass.</p>
<p>I like to use <code>ABC</code> because <strong>you cannot even instantiate</strong> a subclass of <code>Algorithm</code> if you don’t override all abstract methods.</p>
<h2 id="h-the-customizable-parts" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/template-method-pattern-in-python/#h-the-customizable-parts"><span aria-hidden="true">#</span></a> The Customizable Parts</h2>
<p>Each concrete subclass <strong>have to</strong> override <code>do_step_1</code> and <code>do_step_2</code>, because they are decorated with the <code>@abstractmethod</code> decorator from the <code>ABC</code> module. Each sublclass <strong>can</strong> override <code>do_something</code>, or it will use the default implementation provided by the base class.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">AlgorithmA</span><span class="token punctuation">(</span>Algorithm<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">do_step_1</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'do step 1 for Algorithm A'</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">do_step_2</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'do step 2 for Algorithm A'</span><span class="token punctuation">)</span>
<span class="token keyword">class</span> <span class="token class-name">AlgorithmB</span><span class="token punctuation">(</span>Algorithm<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">do_step_1</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'do step 1 for Algorithm B'</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">do_step_2</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'do step 2 for Algorithm B'</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">do_something</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'do something else'</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-the-hollywood-principle" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/template-method-pattern-in-python/#h-the-hollywood-principle"><span aria-hidden="true">#</span></a> The Hollywood Principle</h2>
<p>Here comes the interesting part…</p>
<p>Most of the time a subclass calls the methods of its parent, so the flow of control goes from the subclass to its parent. However, in <code>template_method</code> the flow goes from the base class to a subclass, a principle called <em>Inversion of Control (IoC)</em>.</p>
<p>IoC, also called Hollywood Principle - <em>Don’t call us, we’ll call you</em> - decouples the execution of a task from its implementation. The client code doesn’t call directly the methods responsible for the implementation (<code>do_step_1</code> and <code>do_step_2</code>), but calls a method provided by the base class (<code>template_method</code>), which calls the implementation methods itself.</p>
<blockquote>
<p>Capture the abstraction in an interface, and bury the implementation details in its subclasses.</p>
</blockquote>
<p>This situation is quite common in frameworks. Each framework designs a common interface that implements the invariant pieces of a system’s architecture, and defines placeholders for all customizable parts. Most of the time it’s the framework itself that calls the methods supplied by the user.</p>
<h2 id="h-the-client-code" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/template-method-pattern-in-python/#h-the-client-code"><span aria-hidden="true">#</span></a> The client code</h2>
<p>Finally, here is the client code…</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Algorithm A'</span><span class="token punctuation">)</span>
a <span class="token operator">=</span> AlgorithmA<span class="token punctuation">(</span><span class="token punctuation">)</span>
a<span class="token punctuation">.</span>template_method<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\nAlgorithm B'</span><span class="token punctuation">)</span>
b <span class="token operator">=</span> AlgorithmB<span class="token punctuation">(</span><span class="token punctuation">)</span>
b<span class="token punctuation">.</span>template_method<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>and the output:</p>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> Algorithm A
<span class="token operator">>></span><span class="token operator">></span> AlgorithmA<span class="token punctuation">.</span>__do_absolutely_this
<span class="token operator">>></span><span class="token operator">></span> do step <span class="token number">1</span> <span class="token keyword">for</span> Algorithm A
<span class="token operator">>></span><span class="token operator">></span> do step <span class="token number">2</span> <span class="token keyword">for</span> Algorithm A
<span class="token operator">>></span><span class="token operator">></span> do something
<span class="token operator">>></span><span class="token operator">></span> Algorithm B
<span class="token operator">>></span><span class="token operator">></span> AlgorithmB<span class="token punctuation">.</span>__do_absolutely_this
<span class="token operator">>></span><span class="token operator">></span> do step <span class="token number">1</span> <span class="token keyword">for</span> Algorithm B
<span class="token operator">>></span><span class="token operator">></span> step <span class="token number">2</span> <span class="token keyword">for</span> Algorithm B
<span class="token operator">>></span><span class="token operator">></span> something <span class="token keyword">else</span></code></pre>
<p>You need the code? Grab it <a href="https://github.com/jackdbd/design-patterns">here</a>!</p>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/template-method-pattern-in-python/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<p>Here are some additional resources you might find useful:</p>
<ul>
<li><a href="https://sourcemaking.com/design_patterns/template_method">Template Method on Sourcemaking.com</a></li>
<li><a href="https://martinfowler.com/bliki/InversionOfControl.html">Martin Fowler’s article on Inversion of Control</a></li>
</ul>
https://www.giacomodebidda.com/posts/bridge-pattern-in-python/Bridge pattern in Python2017-01-21T00:00:00Z2017-01-21T00:00:00Z
<p>I struggled quite a bit with the Bridge pattern. The idea itself is rather simple, decouple an interface from its implementation, but I couldn’t think about a simple, yet “real life” example of this pattern.</p>
<h2 id="h-a-bridge-between-two-class-hierarchies" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/bridge-pattern-in-python/#h-a-bridge-between-two-class-hierarchies"><span aria-hidden="true">#</span></a> A Bridge between two class hierarchies</h2>
<p>The purpose of the Bridge pattern is to split a concept into two independent class hierarchies. These two class hierarchies are usually called <em>Interface</em> (or <em>Handle</em>, or <em>Abstraction</em>) and <em>Implementation</em> (or <em>Body</em>).</p>
<p>A <a href="https://en.wikipedia.org/wiki/Bridge_pattern#Java">classic example</a> of Bridge is used in the definition of shapes in an UI environment: one class hierarchy is responsible to define shapes, the other one to draw them on the screen.</p>
<p>Bridge achieves the <a href="https://en.wikipedia.org/wiki/Separation_of_concerns">separation of concerns</a> between orthogonal class hierarchies via <em>composition</em>: the interface object encapsulates an instance of an Implementation class. The Interface class is used directly by the client, but the actual work is done in the Implementation class. The client interacts with the interface object, and doesn’t have to deal with the details of the different implementations. It’s the interface object that delegates all requests to the implementation object it encapsulates.</p>
<blockquote>
<p>Decouple an abstraction from its implementation so that the two can vary independently.</p>
</blockquote>
<h2 id="h-a-simple-yet-real-world-use-case" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/bridge-pattern-in-python/#h-a-simple-yet-real-world-use-case"><span aria-hidden="true">#</span></a> A simple, yet real world use-case</h2>
<p>As I said at the beginning, it wasn’t easy for me to find a simple real-life scenario where I would use this pattern. I didn’t want to reuse the shape-draw example, and the first attempts I made where either too trivial or they were missing the point.</p>
<p>Then I though about A/B testing.</p>
<p>Let’s say that you want to build a news website and you want to show different content for different users. You want to give paid users full access to articles, without any ads. At the same time, you want to give free users some excerpts from the articles, with some ads on the page. Finally, you want to show a <a href="https://en.wikipedia.org/wiki/Call_to_action_(marketing)"><em>call to action</em></a> to all free users, so you can hopefully convert them to paid users.</p>
<p>You know what to draw in the UI, and where to draw it, but you still can’t decide on what to put in the call to action. Here is where A/B testing comes in: you can create two different implementations of the UI components, and by changing only the call to action you can decide which one is more effective.<br />
In a real life scenario the call to action could be a web component of some sort, maybe an element with a slightly different style, color, font size, etc. In this example it’s just a different sentence for the two implementations.</p>
<p>It might be useful to summarize what the components in this toy example represent:</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption></caption>
<thead class="">
<tr class=""><th scope="col" class="">Client</th><th scope="col" class="">Interface</th><th scope="col" class="">Implementation</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">the user</td><td class="">the UI of the website</td><td class="">what it's drawn in each UI component</td></tr>
</tbody>
</table>
</div>
</div>
<p>The Interface is the UI of website. There are free users and paid users, so there are two interfaces to build: one with ads, excerpts and a call to action (free version) and another one with full articles (paid version).</p>
<p>Ok, let’s see some code!</p>
<p>First, let’s define an abstract class for the website. This is the <em>abstraction of the Interface</em> (or <em>abstraction of the Abstraction</em>).</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">from</span> abc <span class="token keyword">import</span> ABC<span class="token punctuation">,</span> abstractmethod
<span class="token comment"># Abstract Interface (aka Handle) used by the client</span>
<span class="token keyword">class</span> <span class="token class-name">Website</span><span class="token punctuation">(</span>ABC<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> implementation<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token comment"># encapsulate an instance of a concrete implementation class</span>
self<span class="token punctuation">.</span>_implementation <span class="token operator">=</span> implementation
<span class="token keyword">def</span> <span class="token function">__str__</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'Interface: {}; Implementation: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>
self<span class="token punctuation">.</span>__class__<span class="token punctuation">.</span>__name__<span class="token punctuation">,</span> self<span class="token punctuation">.</span>_implementation<span class="token punctuation">.</span>__class__<span class="token punctuation">.</span>__name__<span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">show_page</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span></code></pre>
<p>Here is the free version of the website. It’s a concrete Interface (or <em>refined Abstraction</em>).</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Concrete Interface 1</span>
<span class="token keyword">class</span> <span class="token class-name">FreeWebsite</span><span class="token punctuation">(</span>Website<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">show_page</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
ads <span class="token operator">=</span> self<span class="token punctuation">.</span>_implementation<span class="token punctuation">.</span>get_ads<span class="token punctuation">(</span><span class="token punctuation">)</span>
text <span class="token operator">=</span> self<span class="token punctuation">.</span>_implementation<span class="token punctuation">.</span>get_excerpt<span class="token punctuation">(</span><span class="token punctuation">)</span>
call_to_action <span class="token operator">=</span> self<span class="token punctuation">.</span>_implementation<span class="token punctuation">.</span>get_call_to_action<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>ads<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>text<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>call_to_action<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">''</span><span class="token punctuation">)</span></code></pre>
<p>And here is the paid version of the website.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Concrete Interface 2</span>
<span class="token keyword">class</span> <span class="token class-name">PaidWebsite</span><span class="token punctuation">(</span>Website<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">show_page</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
text <span class="token operator">=</span> self<span class="token punctuation">.</span>_implementation<span class="token punctuation">.</span>get_article<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>text<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">''</span><span class="token punctuation">)</span></code></pre>
<p>Now it’s useful to have a look at the Client code. Each interface object requires an instance of an Implementation class, but apart from that, the client code interacts only with the interface, not with the implementation.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Client</span>
<span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
a_free <span class="token operator">=</span> FreeWebsite<span class="token punctuation">(</span>ImplementationA<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>a_free<span class="token punctuation">)</span>
a_free<span class="token punctuation">.</span>show_page<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment"># the client interacts only with the interface</span>
b_free <span class="token operator">=</span> FreeWebsite<span class="token punctuation">(</span>ImplementationB<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>b_free<span class="token punctuation">)</span>
b_free<span class="token punctuation">.</span>show_page<span class="token punctuation">(</span><span class="token punctuation">)</span>
a_paid <span class="token operator">=</span> PaidWebsite<span class="token punctuation">(</span>ImplementationA<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>a_paid<span class="token punctuation">)</span>
a_paid<span class="token punctuation">.</span>show_page<span class="token punctuation">(</span><span class="token punctuation">)</span>
b_paid <span class="token operator">=</span> PaidWebsite<span class="token punctuation">(</span>ImplementationB<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>b_paid<span class="token punctuation">)</span>
b_paid<span class="token punctuation">.</span>show_page<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>Finally, here is the Implementation class hierarchy. First, the abstract class.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Abstract Implementation (aka Body) decoupled from the client</span>
<span class="token keyword">class</span> <span class="token class-name">Implementation</span><span class="token punctuation">(</span>ABC<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">get_excerpt</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'excerpt from the article'</span>
<span class="token keyword">def</span> <span class="token function">get_article</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'full article'</span>
<span class="token keyword">def</span> <span class="token function">get_ads</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'some ads'</span>
<span class="token decorator annotation punctuation">@abstractmethod</span>
<span class="token keyword">def</span> <span class="token function">get_call_to_action</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span></code></pre>
<p>Second, the concrete implementations.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Concrete Implementation 1</span>
<span class="token keyword">class</span> <span class="token class-name">ImplementationA</span><span class="token punctuation">(</span>Implementation<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">get_call_to_action</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'Pay 10 $ a month to remove ads'</span></code></pre>
<pre class="language-python"><code class="language-python"><span class="token comment"># Concrete Implementation 2</span>
<span class="token keyword">class</span> <span class="token class-name">ImplementationB</span><span class="token punctuation">(</span>Implementation<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">get_call_to_action</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token string">'Remove ads with just 10 $ a month'</span></code></pre>
<p>Here is the output of the client code (main function) in this example:</p>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> Interface<span class="token punctuation">:</span> FreeWebsite<span class="token punctuation">;</span> Implementation<span class="token punctuation">:</span> ImplementationA
<span class="token operator">>></span><span class="token operator">></span> some ads
<span class="token operator">>></span><span class="token operator">></span> excerpt <span class="token keyword">from</span> the article
<span class="token operator">>></span><span class="token operator">></span> Pay <span class="token number">10</span> $ a month to remove ads <span class="token comment"># <-- call to action A</span></code></pre>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> Interface<span class="token punctuation">:</span> FreeWebsite<span class="token punctuation">;</span> Implementation<span class="token punctuation">:</span> ImplementationB
<span class="token operator">>></span><span class="token operator">></span> some ads
<span class="token operator">>></span><span class="token operator">></span> excerpt <span class="token keyword">from</span> the article
<span class="token operator">>></span><span class="token operator">></span> Remove ads <span class="token keyword">with</span> just <span class="token number">10</span> $ a month <span class="token comment"># <-- call to action B</span></code></pre>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> Interface<span class="token punctuation">:</span> PaidWebsite<span class="token punctuation">;</span> Implementation<span class="token punctuation">:</span> ImplementationA
<span class="token operator">>></span><span class="token operator">></span> full article</code></pre>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> Interface<span class="token punctuation">:</span> PaidWebsite<span class="token punctuation">;</span> Implementation<span class="token punctuation">:</span> ImplementationB
<span class="token operator">>></span><span class="token operator">></span> full article</code></pre>
<p>There is no call to action in the paid version, so there is no difference between ImplementationA and ImplementationB. This is a toy example after all. In the real world <code>ImplementationA</code> could return a random article, while <code>ImplementationB</code> could track the user preferences and guess an article he/she might be interested in. Also, in a real world scenario you could perform A/B testing of a website by choosing a random implementation instead of declaring it explicitly. Something along these lines:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> random
random_implementation <span class="token operator">=</span> random<span class="token punctuation">.</span>choice<span class="token punctuation">(</span><span class="token punctuation">[</span>ImplementationA<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> ImplementationB<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/bridge-pattern-in-python/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<p>These articles really helped me to understand the Bridge pattern:</p>
<ul>
<li><a href="https://sourcemaking.com/design_patterns/bridge">Bridge</a></li>
<li><a href="https://simpleprogrammer.com/2015/06/08/design-patterns-simplified-the-bridge-pattern/">Design Patterns Simplified: The Bridge Pattern</a></li>
</ul>
https://www.giacomodebidda.com/posts/visualize-gml-files-with-d3/Visualize GML files with D32017-01-12T00:00:00Z2017-01-12T00:00:00Z
<p>I have recently switched from D3 v3.0 to D3 v4.0, and I have already encountered some important changes. For example, the scaling functions are quite different, and the <a href="https://github.com/d3/d3-3.x-api-reference/blob/master/Layouts.md">old layouts available in the older API</a> have been moved and renamed. In fact, D3 v4.0 is no longer a single library, but many small libraries (about 30) that are designed to work together.</p>
<p>For a weekend project I wanted to create a visualization of a simple graph with nodes and edges. In D3 v3.0 this can be done with <code>d3.layout.force</code>. In D3 v4.0 this is a job for <code>d3.forceSimulation</code>.</p>
<h2 id="h-looking-for-some-data" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/visualize-gml-files-with-d3/#h-looking-for-some-data"><span aria-hidden="true">#</span></a> Looking for some data</h2>
<p>I started looking for JSON files that contained the necessary information to represent a small social network graph. To my surprise, I didn’t find any. Instead, I found some small network data sets as GML files <a href="https://networkdata.ics.uci.edu/index.php">here</a> and <a href="https://www-personal.umich.edu/~mejn/netdata/">here</a>. I had never heard of the GML file format before, but since I liked these data sets I decided to proceed. At the end of the article you can see a visualization of a small social network of dolphins (62 nodes, 159 edges). The original file (<code>dolphins.gml</code>) was found on <a href="https://networkdata.ics.uci.edu/data.php?id=6">this page</a> of the University of California Irvine.</p>
<h2 id="h-gml-graph-modelling-language" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/visualize-gml-files-with-d3/#h-gml-graph-modelling-language"><span aria-hidden="true">#</span></a> GML (Graph Modelling Language)</h2>
<p>The <a href="https://networkx.github.io/documentation/networkx-1.10/reference/readwrite.gml.html">Graph Modelling Language</a>, not to be confused with the <a href="https://en.wikipedia.org/wiki/Geography_Markup_Language">Geography Markup Language</a>, is a portable file format for graphs.</p>
<p>A GML file looks like this:</p>
<pre class="language-shell"><code class="language-shell"><span class="token comment"># dolphins.gml</span>
Creator <span class="token string">"Mark Newman on Wed Jul 26 15:04:20 2006"</span>
graph
<span class="token punctuation">[</span>
directed <span class="token number">0</span>
<span class="token function">node</span>
<span class="token punctuation">[</span>
<span class="token function">id</span> <span class="token number">0</span>
label <span class="token string">"Beak"</span>
<span class="token punctuation">]</span>
<span class="token function">node</span>
<span class="token punctuation">[</span>
<span class="token function">id</span> <span class="token number">1</span>
label <span class="token string">"Beescratch"</span>
<span class="token punctuation">]</span>
<span class="token comment"># more nodes here...</span>
edge
<span class="token punctuation">[</span>
<span class="token builtin class-name">source</span> <span class="token number">8</span>
target <span class="token number">3</span>
<span class="token punctuation">]</span>
edge
<span class="token punctuation">[</span>
<span class="token builtin class-name">source</span> <span class="token number">9</span>
target <span class="token number">5</span>
<span class="token punctuation">]</span>
<span class="token comment"># more edges here...</span>
<span class="token punctuation">]</span></code></pre>
<p>It doesn’t look to different from a JSON file, but the problem is that D3 cannot load it directly. After a brief Google search I found out that I could use a Python library to load the GML file and convert it into JSON.</p>
<h2 id="h-networkx" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/visualize-gml-files-with-d3/#h-networkx"><span aria-hidden="true">#</span></a> NetworkX</h2>
<p><a href="https://networkx.readthedocs.io/en/networkx-1.11/index.html">NetworkX</a> is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. It’s distributed with a BSD license and was developed by Aric Hagberg, Dan Schulz and Pieter Swart.</p>
<p>In NetworkX one can create graphs for undirected and directed networks, add and remove nodes and edges in different ways, visualize the network and export network data to draw it with an external tool.</p>
<p>In order to visualize the network data you can use the <code>networkx.draw</code> function and save the image as a PNG with <code>matplotlib</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> networkx <span class="token keyword">as</span> nx
<span class="token keyword">import</span> matplotlib<span class="token punctuation">.</span>pylab <span class="token keyword">as</span> plt
g <span class="token operator">=</span> nx<span class="token punctuation">.</span>read_gml<span class="token punctuation">(</span><span class="token string">'dolphins.gml'</span><span class="token punctuation">)</span>
nx<span class="token punctuation">.</span>draw<span class="token punctuation">(</span>g<span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>savefig<span class="token punctuation">(</span><span class="token string">'dolphins.png'</span><span class="token punctuation">)</span></code></pre>
<p>Apparently there must be some randomness when the figure gets generated, because I ran the code twice and I got 2 different PNG images.</p>
<p>Here is the first one:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303249/dolphins_1_i4rute.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303249/dolphins_1_i4rute.png</a></p>
<p>and the second one:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303249/dolphins_2_osadpo.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303249/dolphins_2_osadpo.png</a></p>
<p>The network data can also be exported and visualized in a program like GraphViz or Gephi, or, like I will do here, in D3.</p>
<p>I didn’t play with NetworkX too much, but from what I’ve seen <a href="https://networkx.readthedocs.io/en/networkx-1.11/tutorial/">the documentation is great</a>. There is also a nice presentation <a href="https://www.cl.cam.ac.uk/~cm542/teaching/2010/stna-pdfs/stna-lecture8.pdf">here</a>.</p>
<p>Here is the snippet of code to convert a GML file into a JSON file:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> networkx <span class="token keyword">as</span> nx
<span class="token keyword">import</span> simplejson <span class="token keyword">as</span> json
<span class="token keyword">from</span> networkx<span class="token punctuation">.</span>readwrite <span class="token keyword">import</span> json_graph
<span class="token comment"># parse the gml file and build the graph object</span>
g <span class="token operator">=</span> nx<span class="token punctuation">.</span>read_gml<span class="token punctuation">(</span><span class="token string">'dolphins.gml'</span><span class="token punctuation">)</span>
<span class="token comment"># create a dictionary in a node-link format that is suitable for JSON serialization</span>
d <span class="token operator">=</span> json_graph<span class="token punctuation">.</span>node_link_data<span class="token punctuation">(</span>g<span class="token punctuation">)</span>
<span class="token keyword">with</span> <span class="token builtin">open</span><span class="token punctuation">(</span><span class="token string">'dolphins.json'</span><span class="token punctuation">,</span> <span class="token string">'w'</span><span class="token punctuation">)</span> <span class="token keyword">as</span> fp<span class="token punctuation">:</span>
json<span class="token punctuation">.</span>dump<span class="token punctuation">(</span>d<span class="token punctuation">,</span> fp<span class="token punctuation">)</span></code></pre>
<p>Note that the GML file I used with the latest NetworkX version (1.11) caused the Exception <code>networkx.exception.NetworkXError: cannot tokenize u'graph' at (2, 1)</code>. I downgraded NetworkX to version 1.9.1 as suggestes in <a href="https://www.bountysource.com/issues/27097685-problem-reading-gml-file">this thread</a> and it worked fine. Another approach would have been to <a href="https://stackoverflow.com/questions/32895291/unexpected-error-reading-gml-graph/37819717#37819717">format the GML file in a different way</a>.</p>
<h2 id="h-the-d3-visualization" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/visualize-gml-files-with-d3/#h-the-d3-visualization"><span aria-hidden="true">#</span></a> The D3 visualization</h2>
<p>Finally, with the network data available as <code>dolphins.json</code>, we can use <code>d3.json</code> to read it. Drag a node or hover on it to know the name of the dolphins in this network!</p>
<figure class="dolphins-graph"></figure>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/visualize-gml-files-with-d3/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<p>The file <code>dolphins.gml</code> contains an undirected social network of frequent associations between 62 dolphins in a community living off Doubtful Sound,<br />
New Zealand, as compiled by Lusseau et al. (2003).</p>
<p><em>D. Lusseau, K. Schneider, O. J. Boisseau, P. Haase, E. Slooten, and S. M. Dawson, The bottlenose dolphin community of Doubtful Sound features a large proportion of long-lasting associations, Behavioral Ecology and Sociobiology 54, 396-405 (2003).</em></p>
<p>Additional information on the network can be found in:</p>
<ul>
<li>
<p><em>D. Lusseau, The emergent properties of a dolphin social network, Proc. R. Soc. London B (suppl.) 270, S186-S188 (2003).</em></p>
</li>
<li>
<p><em>D. Lusseau, <a href="https://arxiv.org/abs/q-bio.PE/0607048">Evidence for social role in a dolphin social network</a>, Preprint q-bio/0607048</em></p>
</li>
</ul>
https://www.giacomodebidda.com/posts/interesting-things-other-people-did-in-2016/Interesting things other people did in 20162016-12-29T00:00:00Z2016-12-29T00:00:00Z
<p>Some time ago I followed a data science course taught by Jeff Leek on Coursera. The course was so good and thorough that eventually they decided to split it into several courses and create a <a href="https://www.coursera.org/specializations/jhu-data-science">data science specialization</a> out of it.<br />
During the course Jeff Leek mentioned that toward the end of the year he likes to compile a <a href="https://simplystatistics.org/posts/2016-12-20-noncomprehensive-list-of-awesome/">list of things other people did during the year</a>.</p>
<p>I think it’s a nice idea, so I’ll make a list of the things I stumbled upon during the year and that I found interesting.</p>
<p>Here it is:</p>
<ul>
<li>Nathan Yau posted a scary but entertaining visualization on <a href="https://flowingdata.com/2016/01/19/how-you-will-die/">how you will die</a>. The analysis and data preparation was performed in R, while for the visualization he used D3.js.</li>
<li>Omar Wagih created <a href="https://guessthecorrelation.com/">Guess the Correlation</a>, a hilarious game where you try to guess how correlated the two variables in a scatter plot are.</li>
<li>Toby Walsh published a <a href="https://arxiv.org/pdf/1602.06462v1.pdf">short paper on AI and singularity</a>. Quite a different opinion than the one expressed by Ray Kurzweil in <a href="https://en.wikipedia.org/wiki/The_Singularity_Is_Near">The Singularity is Near</a>.</li>
<li>Clayton d’Arnault wrote a really interesting article about information overload, one of the biggest problem of our hyperconnected society. The title couldn’t be more appropriate: <a href="https://digitalculturist.com/drowning-in-a-sea-of-information-563a3160efbb#.2x8iotwh4">Drowning in a Sea of Information</a>. I really like the paragraph about <a href="https://jessicaabel.com/idea-debt/">idea debt</a>.</li>
<li>Someone created <a href="https://geon.github.io/programming/2016/03/03/dsxyliea">dsxyliea</a>, a web page that gives you an idea about how people with dyslexia see text.</li>
<li>Someone created an <a href="https://rawgit.com/osnr/horrifying-pdf-experiments/master/breakout.pdf">Atari Breakout in a pdf</a>. You have to view the PDF in Chrome to be able to play it. The author created this game to show that PDF has its own JavaScript scripting (which uses a completely different standard library from the Javascript available in a browser).</li>
<li>Two twin teenagers built a fusion reactor in their bedroom and posted an <a href="https://www.reddit.com/r/IAmA/comments/4tgsaz/iama_i_built_a_fusion_reactor_in_my_bedroom_ama/">AMA on Reddit</a>.</li>
<li>These guys created a really cool project about <a href="https://monterail.com/blog/2016/monterale_breweree_how_we_merge_passion_of_brewing_beer_with_iot/">Beer and IoT</a>.</li>
<li>Charles Scalfani wrote an article on <a href="https://medium.com/@cscalfani/why-experts-make-bad-teachers-ccaed2df029b#.rqtpbtiqf">why experts make bad teachers</a>. Really, really interesting point of view.</li>
<li>Rody Zakovich created a funny Sankey diagram on the <a href="https://public.tableau.com/views/SouthParkSeasonOneWordsAnalysis/TheWordsofSouthParkSeason1?:embed=y&:display_count=yes&:showVizHome=no">most common swear words in South Park season 1</a>.</li>
<li>Brandon Liu of <a href="https://pureinformation.net/">pureinformation.net</a> created a beautiful map of New York where you can explore the <a href="https://pureinformation.net/building-age-nyc/#12/40.7457/-73.8841">age of the buildings</a>. I love maps like this one. I want to build something similar in 2017.</li>
<li>Jose Aguinaga wrote a hilarious article about the state of the Javascript ecosystem, now that a new framework/transpiler/shiny thing seems to pop up every single day. <a href="https://hackernoon.com/how-it-feels-to-learn-javascript-in-2016-d3a717dd577f#.iflg3l8s5">How it feels to learn Javascript in 2016</a>.</li>
<li>The University of Michigan created <a href="https://www.vizhealth.org/gallery/">Visualizing Health</a>, a website which hosts a collection of visualizations on healthcare topics. All visualizations are distributed via a Creative Commons license, which allows anybody – academics, healthcare organizations, even for-profit businesses — to adapt them for their own objectives.</li>
<li>Christopher Möller put together a nice list of resources on <a href="https://github.com/wbkd/awesome-interactive-journalism?utm_source=hackernewsletter&utm_medium=email&utm_term=fav">interactive journalism</a>. I proptly starred this repository on my GitHub account.</li>
<li>Deep Learning was the buzzword of 2016 (and probably it will be of 2017 too). This article about <a href="https://getdango.com/emoji-and-deep-learning/">Emoji & Deep Learning</a> was a bit different from the usual neural network tutorial :-)</li>
</ul>
<p><em>Note: I was really tempted to steal some stuff from Jeff’s list, there were really some good links…</em></p>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/interesting-things-other-people-did-in-2016/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<p>Jeff Leek’s <a href="https://simplystatistics.org/posts/2016-12-20-noncomprehensive-list-of-awesome/">“A non-comprehensive list of awesome things other people did in 2016”</a>.</p>
https://www.giacomodebidda.com/posts/virtual-environments-with-virtualenvwrapper/Virtual environments with virtualenvwrapper2016-12-11T00:00:00Z2016-12-11T00:00:00Z
<p><code>virtualenvwrapper</code> is a set of shell functions built on top of the <code>virtualenv</code> python module, and make it even easier to create and manage python virtual environments.</p>
<p>The <a href="https://virtualenvwrapper.readthedocs.io/en/latest/install.html">documentation</a> for this project is quite good, but here I wanted to write a reminder (mostly for me) about the configuration of this tool on a Linux server.</p>
<h2 id="h-configuration" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/virtual-environments-with-virtualenvwrapper/#h-configuration"><span aria-hidden="true">#</span></a> Configuration</h2>
<p>You can install <code>virtualenwrapper</code> with <code>pip</code>:</p>
<pre class="language-shell"><code class="language-shell">pip <span class="token function">install</span> virtualenvwrapper</code></pre>
<p>Then you will need to create a (hidden) folder in your <code>home</code> directory and call it <code>.virtualenvs</code>.</p>
<p>Lastly, you will need to configure your terminal to execute <code>virtualenvwrapper</code> commands. This is done by adding 2 lines of code to a bash configuration file.</p>
<p>Here’s the **catch:</p>
<p>If you are working on your computer you have to add these 2 lines to your <code>~/.bashrc</code> file.</p>
<pre class="language-shell"><code class="language-shell"><span class="token comment"># ~/.bashrc (your local machine)</span>
<span class="token builtin class-name">export</span> <span class="token assign-left variable">WORKON_HOME</span><span class="token operator">=</span><span class="token environment constant">$HOME</span>/.virtualenvs
<span class="token builtin class-name">source</span> /usr/local/bin/virtualenvwrapper.sh</code></pre>
<p>If you are working on a remote server via SSH you have to add the very same 2 lines, but in a different file, the <code>~/.bash_profile</code> file.</p>
<pre class="language-shell"><code class="language-shell"><span class="token comment"># ~/.bash_profile (your linux server)</span>
<span class="token builtin class-name">export</span> <span class="token assign-left variable">WORKON_HOME</span><span class="token operator">=</span><span class="token environment constant">$HOME</span>/.virtualenvs
<span class="token builtin class-name">source</span> /usr/local/bin/virtualenvwrapper.sh</code></pre>
<p>Thanks to <a href="https://askubuntu.com/a/121075">this answer on askubuntu</a> I found out that this difference is due to the different access modality:</p>
<ul>
<li>on your local machine you are accessing the console in interactive, non-login mode, and the <code>~/.bashrc</code> file will be sourced;</li>
<li>on a remote server (e.g. a DigitalOcean droplet) via SSH you are accessing the console in interactive, login mode, and the <code>~/.bash_profile</code> file will be sourced.</li>
</ul>
<p>If you want to know more about these bash files, see <a href="https://stackoverflow.com/questions/415403/whats-the-difference-between-bashrc-bash-profile-and-environment">here</a>.</p>
<h2 id="h-hooks" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/virtual-environments-with-virtualenvwrapper/#h-hooks"><span aria-hidden="true">#</span></a> Hooks</h2>
<p>If you are already using <code>virtualenv</code> you will probably know the <code>source bin/activate</code> command. This is a <em>hook</em> that sets an environment variable called <code>VIRTUAL_ENV</code>, another one called <code>PYTHON_HOME</code> and a few others. The command <code>deactivate</code> is another hook that unset the same environment variables previously set.</p>
<p><code>virtualenvwrapper</code> defines some additional hooks, like <code>postactivate</code> and <code>predeactivate</code>. You can use these hooks to set additional environment variables, set aliases, ect. See <a href="https://gist.github.com/manuganji/9069466">here</a> for some examples that you might find useful.</p>
<h2 id="h-most-useful-commands" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/virtual-environments-with-virtualenvwrapper/#h-most-useful-commands"><span aria-hidden="true">#</span></a> Most useful commands</h2>
<p>Here are the most common <code>virtualenwrapper</code> comands:</p>
<pre class="language-shell"><code class="language-shell">mkvirtualenv YOUR_VIRTUALENV <span class="token comment"># create virtual environment (and activate it)</span>
mkvirtualenv YOUR_VIRTUALENV <span class="token parameter variable">--python</span><span class="token operator">=</span>python3.5 <span class="token comment"># create virtual enviroment and specifiy the python version</span>
workon YOUR_VIRTUALENV <span class="token comment"># activate virtual environment</span>
rmvirtualenv YOUR_VIRTUALENV <span class="token comment"># remove virtual environment</span>
deactivate <span class="token comment"># deactivate current active virtual environment</span></code></pre>
https://www.giacomodebidda.com/posts/squashing-git-commits/Squashing Git commits2016-12-04T00:00:00Z2016-12-04T00:00:00Z
<p>In Git you can revise your commit history before pushing your changes to a remote repository. This comes in handy whenever you want to group certain changes together, edit commit messages, or change the order of commits.</p>
<h2 id="h-the-golder-rule" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/squashing-git-commits/#h-the-golder-rule"><span aria-hidden="true">#</span></a> The golder rule</h2>
<p>Squashing commits is an operation which rewrites the history of a git branch, so it should be performed <strong>only on a local branch</strong>. In other words, <strong>never squash commits on a public branch</strong>.</p>
<h2 id="h-why-is-it-useful%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/squashing-git-commits/#h-why-is-it-useful%3F"><span aria-hidden="true">#</span></a> Why is it useful?</h2>
<p>When using a version control system like git, it’s often considered a good practice to commit early and often. The problem with this approach is that you might end up with a history full of tiny little commits that aren’t really that meaningful by themselves. You can solve this issue by squashing all commits related to a single topic into a single commit, and write a good commit message which explains what you implemented with your changes. If you do so, you’ll end up with a leaner and more meaningful branch history.</p>
<blockquote>
<p>Squashing commits is a great way to group certain changes together before sharing them with others.</p>
</blockquote>
<p>Let’s imagine that you are working on a particular feature and as soon as you have some meaningful changes (e.g. a function) you would like to commit them. If you have already decided that you will squash some commits later on, you could commit some code without having to worry about writing a good commit message. For example you could write a message like <code>WIP on new feature</code>, because you know that this message will never make it to the branch history (WIP = Work In Progress). After several “WIP commits” you can squash them together into a single one, and write a good commit message for it.</p>
<h2 id="h-how-is-it-done%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/squashing-git-commits/#h-how-is-it-done%3F"><span aria-hidden="true">#</span></a> How is it done?</h2>
<p>There isn’t a <code>git squash</code> command. Squashing commits is an operation which can be performed in several ways by using other commands. I’m aware of 3 ways to do it:</p>
<ol>
<li>reset</li>
<li>rebase</li>
<li>merge</li>
</ol>
<h2 id="h-1-reset" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/squashing-git-commits/#h-1-reset"><span aria-hidden="true">#</span></a> 1. Reset</h2>
<p>This is the easiest approach. I first heard about this method <a href="https://stackoverflow.com/a/7275658">here</a>. You just have to start with a clean working tree and run <code>git reset --soft HEAD~X</code>, where <code>X</code> is the number of commits to squash together.</p>
<p>Let’s make a simple example to see how it works.<br />
After some specific commit (<code>Initial commit</code> in the image below) you wrote some changes and you committed 3 times. When you wrote those 3 commit messages you just wanted to save your work, maybe because you had to checkout a different branch and fix a bug, or maybe because you don’t like to use <code>git stash</code>.</p>
<p>Let’s say that with the 3rd “WIP commit” you have concluded your work on the new feature and you would like to wrap your changes up and start working on a different topic. This is what you should see with <code>gitk</code>:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_4_vbpscq.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_4_vbpscq.png</a></p>
<p>Make sure that you have a clean working tree and run:</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> reset <span class="token parameter variable">--soft</span> HEAD~3</code></pre>
<p>This is the current situation: the changes are staged, but not yet committed.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_2_ism7ym.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_2_ism7ym.png</a></p>
<p>Commit these changes, and this time write a meaningful commit message.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_3_pygx89.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_3_pygx89.png</a></p>
<p>Notice that now that you have moved the HEAD, all changes done in the 3 “WIP commits” are no longer in the branch history, because you have just rewritten it. That’s the reason why you should never squash commits on a branch where other developers are working.</p>
<h2 id="h-2-rebase" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/squashing-git-commits/#h-2-rebase"><span aria-hidden="true">#</span></a> 2. Rebase</h2>
<p>The rebase approach is a bit more complex but offers a much greater flexibility. You are no longer limited to squash all of your changes into a single commit, but you can squash X commits together into any number of commits between 1 and X-1. You can also change the order of commits!<br />
The command you are looking for is <code>git rebase -i HEAD~X</code>.</p>
<p>Once again, let’s pretend that you have 3 commits and you would like to squash them into one.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_4_vbpscq.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_4_vbpscq.png</a></p>
<p>Use git rebase to carry on the squashing procedure.</p>
<pre class="language-shell"><code class="language-shell"><span class="token function">git</span> rebase <span class="token parameter variable">-i</span> HEAD~3</code></pre>
<p>Now the default editor will display a message like the following one:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_5_bjmj1p.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_5_bjmj1p.png</a></p>
<p>Since you want a single commit to appear in the branch history, you have to use one commit and squash the remaining two.</p>
<p>In this image <strong>the last commit is the third one from the top</strong>, but you don’t want to keep that. You want to keep the first commit (the oldest one), and then squash the second one and the third one together into it. I must admit that I find it a bit counterintuitive and it took me a while to get used to it. If this sounds a bit weird to you, don’t worry, because if we mess up we can still revert to the original situation with <code>git rebase --abort</code>.</p>
<p>In this specific example I wrote a meaningless message for the first commit (WIP 1), so I will <code>reword</code> it. If the commit message is fine, you can simply <code>pick</code> the commit. The other 2 commits will be squashed with <code>squash</code>.</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_6_xcii5l.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_6_xcii5l.png</a></p>
<p>When you accept the changes (<code>Ctrl + X</code> in Nano, <code>Esc</code> then <code>:wq</code> in Vim) you will see a recap like this one:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_7_ajppex.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_7_ajppex.png</a></p>
<p>You can also edit this recap, which will appear in the commit <em>description</em>. Save your changes for this recap message. In <code>gitk</code> you should see something like this:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_3_pygx89.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_3_pygx89.png</a></p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_9_fjoalj.png">https://res.cloudinary.com/jackdbd/image/upload/v1599303211/squash_9_fjoalj.png</a></p>
<h2 id="h-3-merge" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/squashing-git-commits/#h-3-merge"><span aria-hidden="true">#</span></a> 3. Merge</h2>
<p>I must admit I’ve never tried this approach. It looks overly complicated and uses a git reset, so it’s probably the riskiest option of the three. Anyway, this is where I found it:<br />
<a href="https://stackoverflow.com/a/5190323">squash-merge</a>.</p>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/squashing-git-commits/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<ul>
<li><a href="https://jamescooke.info/git-to-squash-or-not-to-squash.html">To squash or not to squash</a></li>
<li><a href="https://ariejan.net/2011/07/05/git-squash-your-latests-commits-into-one/">Squashing with rebase, 1</a></li>
<li><a href="https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History">Squashing with rebase, 2</a>.</li>
</ul>
https://www.giacomodebidda.com/posts/better-git-commits/Better Git commits2016-12-04T00:00:00Z2016-12-04T00:00:00Z
<p>Knowing what changes to commit and when to commit them is a valuable skill that comes in handy either if you are working in a team or by yourself. In this brief article I’ll try to describe some common best practices.</p>
<h2 id="h-atomic-commits" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/better-git-commits/#h-atomic-commits"><span aria-hidden="true">#</span></a> Atomic commits</h2>
<p>First of all, a commit should be <a href="https://www.freshconsulting.com/insights/blog/atomic-commits/">atomic</a>, namely it should be about one task or one fix. If you have changes which are completely unrelated, you should have different commits for each one of them.</p>
<h2 id="h-squashing%3F-yes-please" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/better-git-commits/#h-squashing%3F-yes-please"><span aria-hidden="true">#</span></a> Squashing? Yes please!</h2>
<p>It’s good to commit early and often, but you probably don’t want to end up with a zillion tiny commits on a public branch. It’s very hard to read the history of that branch and go back to a specific snapshot where you (or another developer in you team) introduced the changes. You can instead commit many times on a feature branch, rewrite your local history by <a href="https://giacomodebidda.com/posts/squashing-git-commits/">squashing several commits together</a> into a reasonable number of commits, and only then push your squashed commits to a public branch.<br />
There are also <a href="https://stackoverflow.com/questions/6543913/git-commit-best-practices/6544580#6544580">some other strategies</a>, but I have to admit that I’m still not very comfortable with them.</p>
<h2 id="h-the-commit-message" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/better-git-commits/#h-the-commit-message"><span aria-hidden="true">#</span></a> The commit message</h2>
<p>Writing a good commit message is an important skill to have, even when working in a small project. Remember that <em>the person who is most likely to read that commit messages is the developer who wrote it</em>, so it makes perfect sense to stop for a while and think carefully about what to write when committing some changes. It certainly takes some time to craft a good explanation of the changes introduced with a particular commit, but I think it’s time well spent.</p>
<p>It’s tempting to use the shortcut <code>git commit -m</code>, but I try to avoid it. It makes me feel bad every time I can’t think about a way to express my changes into 50 characters or less.</p>
<p>A commit message should start with a brief <strong>summary of 50 characters or less</strong>. This short text serves the purpose of explaining <em>what</em> these changes are about. Use the imperative mode (e.g. <em>Fix issue #1234</em>), not a past tense (e.g. <em>Fixed issue #1234</em>). This is consistent with the commit messages generated by commands like <code>git merge</code>. Also, don’t forget to use uppercase for the first letter in the summary.</p>
<p>After the summary you should leave one blank line and write a more detailed <strong>description</strong> about our changes. It’s a good idea to wrap this text at <strong>72 characters</strong> per line. The length of this text depends on the commit itself: if you are committing only some small changes you might skip it altogether, while if you need to explain thoroughly the reasoning behind a given design implementation of a class or a function it might cover several lines of text.<br />
If the description is really long, you can divide it into several paragraphs and leave a blank line after each one. If you have some bullet points, you should keep the indentation consistent.</p>
<blockquote>
<p>A good commit message should explain <em>what</em> you did (in the summary) and <em>why</em> you did it (in the description).</p>
</blockquote>
<p>Here is an example:</p>
<p><a href="https://res.cloudinary.com/jackdbd/image/upload/v1599301996/commit_1_ah0dkn.png">https://res.cloudinary.com/jackdbd/image/upload/v1599301996/commit_1_ah0dkn.png</a></p>
<h2 id="h-why-72-characters%3F-and-how-to-do-it%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/better-git-commits/#h-why-72-characters%3F-and-how-to-do-it%3F"><span aria-hidden="true">#</span></a> Why 72 characters? And how to do it?</h2>
<p>The reason behind 72 characters is that git commands like <code>git log</code> don’t do any special wrapping, so if you enter them in a 80 column terminal, and if you need to reserve 4 columns for left indentation and 4 for right indentation, you are left with 72 columns.</p>
<p>A really handy way to have syntax highlighting and automatic text wrapping is writing commit messages using Vim. You can set Vim as the default editor for git either by opening a terminal and entering <code>git config --global core.editor "vim"</code>, or by locating our .gitconfig file (it’s in the <code>home</code> folder in Linux) and adding an entry in the <code>[core]</code> section (or create that section if is missing).</p>
<pre class="language-shell"><code class="language-shell"><span class="token comment"># .gitconfig</span>
<span class="token punctuation">[</span>core<span class="token punctuation">]</span>
editor <span class="token operator">=</span> <span class="token function">vim</span></code></pre>
<p>Then you need to configure Vim. Locate the <code>.vimrc</code> file in the <code>home</code> folder (or create one if it’s missing) and copy/paste the content of the example file you can find <a href="https://vim.fandom.com/wiki/Example_vimrc">here</a> to have 50 character limit on the first line, text wrap to 72 characters for the commit message description, syntax highlighting and line numbers.<br />
If you are curious why you can’t find these settings in the text you just pasted in the <code>.vimrc</code> file, it’s because Vim understands automatically that we are writing a gitcommit file, and uses a specific setting with them. We can check these settings by typing <code>:set</code> in command mode while we are about to commit our changes. We will see some entries like <code>filetype=gitcommit</code> and <code>textwidth=72</code>.</p>
<h2 id="h-references" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/better-git-commits/#h-references"><span aria-hidden="true">#</span></a> References</h2>
<p>Here are some additional resources on writing great commit messages:</p>
<ul>
<li><a href="https://thoughtbot.com/blog/5-useful-tips-for-a-better-commit-message">5 useful tips for a better commit message</a></li>
<li><a href="https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html">A note about git commit messages</a></li>
<li><a href="https://sethrobertson.github.io/GitBestPractices/">Git best practices</a></li>
<li><a href="https://cbea.ms/git-commit/">Git commit</a></li>
</ul>
https://www.giacomodebidda.com/posts/facade-pattern-in-python/Façade pattern in Python2016-11-26T21:12:03Z2016-11-26T21:12:03Z
<p>Let’s continue our journey through the most used design patterns by implementing a <em>Façade</em> pattern in Python.</p>
<h2 id="h-fa%C3%A7ade-is-a-structural-design-pattern" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/facade-pattern-in-python/#h-fa%C3%A7ade-is-a-structural-design-pattern"><span aria-hidden="true">#</span></a> Façade is a structural design pattern</h2>
<p><em>Façade</em> can be used to define a simpler, leaner, higher-level, more consistent interface to expose to a <em>client</em> a specific subset of functionalities provided by one or more subsystems. Tipically these lower-level subsystems are called <em>complex parts</em>. All complex parts controlled by the Façade are often parts of smaller subsystems that are <em>associated</em> one to another.</p>
<blockquote>
<p>Façade builds a <em>convenient</em> interface which saves a client the hassle of<br />
dealing with <em>complex parts</em>.</p>
</blockquote>
<p>Note that the client can still have direct access to these functionalities: the Façade does not - and should not - prevent the client from accessing the complex parts.</p>
<blockquote>
<p>Subsystem implementation gains <em>flexibility</em>, client gains <em>simplicity</em>.</p>
</blockquote>
<p>The <a href="https://topepo.github.io/caret/index.html">caret package in R</a> is a great example of a Façade, because it wraps a collection of APIs into a single well-designed API. The R programming language contains a huge number of packages for implementing almost all statistical models ever created. Unfortunately, more often then not, these packages have their own specific syntax, so when training/testing the model, one must know the syntax for the model being used. Caret implements a set of functions that provide a uniform interface when creating predictive models (e.g. the functions <code>train</code> and <code>predict</code>), but if you want you can still use the original syntax to train/test a specific model.</p>
<h2 id="h-fa%C3%A7ade-pattern-in-python" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/facade-pattern-in-python/#h-fa%C3%A7ade-pattern-in-python"><span aria-hidden="true">#</span></a> Façade Pattern in Python</h2>
<p>To illustrate a Façade pattern I will use a car as an example: you would like to have access to a set of functionalities when using a car (e.g. drive, park, etc), but you probably don’t want to deal with all the complex parts a car is composed of.</p>
<p>In this example I decided to implement the complex parts as private classes. Since this is python, we can still access these classes without any issue. I just make them private to suggest that the client should call the Façade, not the complex parts directly.</p>
<p>The Complex parts.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">_IgnitionSystem</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@staticmethod</span>
<span class="token keyword">def</span> <span class="token function">produce_spark</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token boolean">True</span>
<span class="token keyword">class</span> <span class="token class-name">_Engine</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>revs_per_minute <span class="token operator">=</span> <span class="token number">0</span>
<span class="token keyword">def</span> <span class="token function">turnon</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>revs_per_minute <span class="token operator">=</span> <span class="token number">2000</span>
<span class="token keyword">def</span> <span class="token function">turnoff</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>revs_per_minute <span class="token operator">=</span> <span class="token number">0</span>
<span class="token keyword">class</span> <span class="token class-name">_FuelTank</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> level<span class="token operator">=</span><span class="token number">30</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_level <span class="token operator">=</span> level
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">level</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>_level
<span class="token decorator annotation punctuation">@level<span class="token punctuation">.</span>setter</span>
<span class="token keyword">def</span> <span class="token function">level</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> level<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_level <span class="token operator">=</span> level
<span class="token keyword">class</span> <span class="token class-name">_DashBoardLight</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> is_on<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_is_on <span class="token operator">=</span> is_on
<span class="token keyword">def</span> <span class="token function">__str__</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>__class__<span class="token punctuation">.</span>__name__
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">is_on</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> self<span class="token punctuation">.</span>_is_on
<span class="token decorator annotation punctuation">@is_on<span class="token punctuation">.</span>setter</span>
<span class="token keyword">def</span> <span class="token function">is_on</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> status<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>_is_on <span class="token operator">=</span> status
<span class="token keyword">def</span> <span class="token function">status_check</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> self<span class="token punctuation">.</span>_is_on<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}: ON'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{}: OFF'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">class</span> <span class="token class-name">_HandBrakeLight</span><span class="token punctuation">(</span>_DashBoardLight<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token keyword">class</span> <span class="token class-name">_FogLampLight</span><span class="token punctuation">(</span>_DashBoardLight<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">pass</span>
<span class="token keyword">class</span> <span class="token class-name">_Dashboard</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>lights <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">'handbreak'</span><span class="token punctuation">:</span> _HandBrakeLight<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">'fog'</span><span class="token punctuation">:</span> _FogLampLight<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">}</span>
<span class="token keyword">def</span> <span class="token function">show</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">for</span> light <span class="token keyword">in</span> self<span class="token punctuation">.</span>lights<span class="token punctuation">.</span>values<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
light<span class="token punctuation">.</span>status_check<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>The Façade.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">Car</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>ignition_system <span class="token operator">=</span> _IgnitionSystem<span class="token punctuation">(</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>engine <span class="token operator">=</span> _Engine<span class="token punctuation">(</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>fuel_tank <span class="token operator">=</span> _FuelTank<span class="token punctuation">(</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>dashboard <span class="token operator">=</span> _Dashboard<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@property</span>
<span class="token keyword">def</span> <span class="token function">km_per_litre</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token number">17.0</span>
<span class="token keyword">def</span> <span class="token function">consume_fuel</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> km<span class="token punctuation">)</span><span class="token punctuation">:</span>
litres <span class="token operator">=</span> <span class="token builtin">min</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>fuel_tank<span class="token punctuation">.</span>level<span class="token punctuation">,</span> km <span class="token operator">/</span> self<span class="token punctuation">.</span>km_per_litre<span class="token punctuation">)</span>
self<span class="token punctuation">.</span>fuel_tank<span class="token punctuation">.</span>level <span class="token operator">-=</span> litres
<span class="token keyword">def</span> <span class="token function">start</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\nStarting...'</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>dashboard<span class="token punctuation">.</span>show<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> self<span class="token punctuation">.</span>ignition_system<span class="token punctuation">.</span>produce_spark<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>engine<span class="token punctuation">.</span>turnon<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Can\'t start. Faulty ignition system'</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">has_enough_fuel</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> km<span class="token punctuation">,</span> km_per_litre<span class="token punctuation">)</span><span class="token punctuation">:</span>
litres_needed <span class="token operator">=</span> km <span class="token operator">/</span> km_per_litre
<span class="token keyword">if</span> self<span class="token punctuation">.</span>fuel_tank<span class="token punctuation">.</span>level <span class="token operator">></span> litres_needed<span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token boolean">True</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token boolean">False</span>
<span class="token keyword">def</span> <span class="token function">drive</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> km<span class="token operator">=</span><span class="token number">100</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\n'</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> self<span class="token punctuation">.</span>engine<span class="token punctuation">.</span>revs_per_minute <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">:</span>
<span class="token keyword">while</span> self<span class="token punctuation">.</span>has_enough_fuel<span class="token punctuation">(</span>km<span class="token punctuation">,</span> self<span class="token punctuation">.</span>km_per_litre<span class="token punctuation">)</span><span class="token punctuation">:</span>
self<span class="token punctuation">.</span>consume_fuel<span class="token punctuation">(</span>km<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Drove {}km'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>km<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'{:.2f}l of fuel still left'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>fuel_tank<span class="token punctuation">.</span>level<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Can\'t drive. The Engine is turned off!'</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">park</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\nParking...'</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>dashboard<span class="token punctuation">.</span>lights<span class="token punctuation">[</span><span class="token string">'handbreak'</span><span class="token punctuation">]</span><span class="token punctuation">.</span>is_on <span class="token operator">=</span> <span class="token boolean">True</span>
self<span class="token punctuation">.</span>dashboard<span class="token punctuation">.</span>show<span class="token punctuation">(</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>engine<span class="token punctuation">.</span>turnoff<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">switch_fog_lights</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> status<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\nSwitching {} fog lights...'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>status<span class="token punctuation">)</span><span class="token punctuation">)</span>
boolean <span class="token operator">=</span> <span class="token boolean">True</span> <span class="token keyword">if</span> status <span class="token operator">==</span> <span class="token string">'ON'</span> <span class="token keyword">else</span> <span class="token boolean">False</span>
self<span class="token punctuation">.</span>dashboard<span class="token punctuation">.</span>lights<span class="token punctuation">[</span><span class="token string">'fog'</span><span class="token punctuation">]</span><span class="token punctuation">.</span>is_on <span class="token operator">=</span> boolean
self<span class="token punctuation">.</span>dashboard<span class="token punctuation">.</span>show<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">fill_up_tank</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\nFuel tank filled up!'</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>fuel_tank<span class="token punctuation">.</span>level <span class="token operator">=</span> <span class="token number">100</span></code></pre>
<p>The Client here is simply the main function.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
car <span class="token operator">=</span> Car<span class="token punctuation">(</span><span class="token punctuation">)</span>
car<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>
car<span class="token punctuation">.</span>drive<span class="token punctuation">(</span><span class="token punctuation">)</span>
car<span class="token punctuation">.</span>switch_fog_lights<span class="token punctuation">(</span><span class="token string">'ON'</span><span class="token punctuation">)</span>
car<span class="token punctuation">.</span>switch_fog_lights<span class="token punctuation">(</span><span class="token string">'OFF'</span><span class="token punctuation">)</span>
car<span class="token punctuation">.</span>park<span class="token punctuation">(</span><span class="token punctuation">)</span>
car<span class="token punctuation">.</span>fill_up_tank<span class="token punctuation">(</span><span class="token punctuation">)</span>
car<span class="token punctuation">.</span>drive<span class="token punctuation">(</span><span class="token punctuation">)</span>
car<span class="token punctuation">.</span>start<span class="token punctuation">(</span><span class="token punctuation">)</span>
car<span class="token punctuation">.</span>drive<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
main<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>You need the code? Grab it <a href="https://github.com/jackdbd/design-patterns">here</a>!</p>
https://www.giacomodebidda.com/posts/adapter-pattern-in-python/Adapter pattern in Python2016-11-26T09:12:03Z2016-11-26T09:12:03Z
<p>Some weeks ago I decided to start studying design patterns and implementing them in Python. <em>Design patterns</em> and <em>Head first in design patterns</em> are constantly cited for being really good books. I added them to my reading list some time ago, but I still haven’t managed to read them so far. Nonetheless, I’ve read several blog posts, articles on Wikipedia and answers on Stack Overflow and started implementing some of these patterns.</p>
<p>Here we are going to see the <em>Adapter</em> pattern.</p>
<h2 id="h-adapter-is-a-structural-design-pattern" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/adapter-pattern-in-python/#h-adapter-is-a-structural-design-pattern"><span aria-hidden="true">#</span></a> Adapter is a structural design pattern</h2>
<p>Structural design patterns are concerned with how classes and objects are <em>composed</em> to form larger structures. They help to use classes or methods which may not be usable directly, or they can ease the design by identifying a simple way to build relationships between entities.</p>
<p><em>Adapter</em> allows a <em>Client</em> to access otherwise not directly accessible functionalities of a <em>Supplier</em> . Adapter makes things work after they are designed: it produces an interface for a single object or class, and <em>adapts</em> such class in a way that a <em>Client</em> can use it.</p>
<blockquote>
<p>You have got <em>this</em>, and you <strong>need</strong> <em>that</em>.</p>
</blockquote>
<h2 id="h-how-do-we-implement-the-adapter-pattern%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/adapter-pattern-in-python/#h-how-do-we-implement-the-adapter-pattern%3F"><span aria-hidden="true">#</span></a> How do we implement the Adapter Pattern?</h2>
<p>There are two ways of implementing the Adapter pattern:</p>
<ol>
<li>Object Adapter</li>
<li>Class Adapter</li>
</ol>
<p>The object adapter uses <strong>encapsulation</strong>, while the class adapter uses <strong>multiple inheritance</strong> (Python supports both encapsulation and multiple inheritance).</p>
<p>Let’s imagine that you have a smartphone (client) and you want to charge it. In order to charge a mobile phone you need a direct current (DC) and an input voltage of a few volts (from 3.7V to 5.2V I suppose), so you can’t simply plug it directly into a wall socket (supplier), which provides an alternate current (AC) and outputs either 230V (in Europe) or 120V (in the US). Namely, without a phone charger (the Adapter) you <strong>can’t</strong> charge your phone.</p>
<p><em>In the following code I implemented both a European Socket class and and an American Socket class because they will be useful later on when explaining the Class Adapter approach. For now you can ignore the USSocket class.</em></p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Client</span>
<span class="token keyword">class</span> <span class="token class-name">Smartphone</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
max_input_voltage <span class="token operator">=</span> <span class="token number">5</span>
<span class="token decorator annotation punctuation">@classmethod</span>
<span class="token keyword">def</span> <span class="token function">outcome</span><span class="token punctuation">(</span>cls<span class="token punctuation">,</span> input_voltage<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> input_voltage <span class="token operator">></span> cls<span class="token punctuation">.</span>max_input_voltage<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"Input voltage: {}V -- BURNING!!!"</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>input_voltage<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"Input voltage: {}V -- Charging..."</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>input_voltage<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">charge</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> input_voltage<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Charge the phone with the given input voltage."""</span>
self<span class="token punctuation">.</span>outcome<span class="token punctuation">(</span>input_voltage<span class="token punctuation">)</span>
<span class="token comment"># Supplier</span>
<span class="token keyword">class</span> <span class="token class-name">Socket</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
output_voltage <span class="token operator">=</span> <span class="token boolean">None</span>
<span class="token keyword">class</span> <span class="token class-name">EUSocket</span><span class="token punctuation">(</span>Socket<span class="token punctuation">)</span><span class="token punctuation">:</span>
output_voltage <span class="token operator">=</span> <span class="token number">230</span>
<span class="token keyword">class</span> <span class="token class-name">USSocket</span><span class="token punctuation">(</span>Socket<span class="token punctuation">)</span><span class="token punctuation">:</span>
output_voltage <span class="token operator">=</span> <span class="token number">120</span></code></pre>
<p>This is the current scenario:</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption></caption>
<thead class="">
<tr class=""><th scope="col" class="">Client</th><th scope="col" class="">Supplier</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">Smartphone</td><td class="">EUSocket</td></tr>
</tbody>
</table>
</div>
</div>
<p>If you take a Smartphone instance and you call <code>charge</code> with <code>EUSocket.output_voltage</code> as argument, you will fail at charging your phone.</p>
<pre class="language-python"><code class="language-python">smartphone <span class="token operator">=</span> Smartphone<span class="token punctuation">(</span><span class="token punctuation">)</span>
smartphone<span class="token punctuation">.</span>charge<span class="token punctuation">(</span>EUSocket<span class="token punctuation">.</span>output_voltage<span class="token punctuation">)</span>
<span class="token operator">>></span><span class="token operator">></span> Input voltage<span class="token punctuation">:</span> 230V <span class="token operator">-</span><span class="token operator">-</span> BURNING!!!</code></pre>
<h2 id="h-1-object-adapter" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/adapter-pattern-in-python/#h-1-object-adapter"><span aria-hidden="true">#</span></a> 1. Object Adapter</h2>
<p>Obvioulsy, you need a phone charger to charge your smarthone. You can think of this phone charger as a completely independent <em>entity</em> from the smartphone and the wall socket. This new entity encapsulates client and supplier, and allows you to call the <code>charge</code> method without changing anything, neither in the Smartphone class, nor the Socket class. The phone charger converts an alternate current, high voltage power supply, into a direct current, low voltage power supply that can be used to charge the smartphone.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">EUAdapter</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""EUAdapter encapsulates client (Smartphone) and supplier (EUSocket)."""</span>
input_voltage <span class="token operator">=</span> EUSocket<span class="token punctuation">.</span>output_voltage
output_voltage <span class="token operator">=</span> Smartphone<span class="token punctuation">.</span>max_input_voltage</code></pre>
<p>The EUAdapter class is a Supplier to the Smartphone class, and at the same time it’s a Client to the EUSocket class.</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption></caption>
<thead class="">
<tr class=""><th scope="col" class="">Client</th><th scope="col" class="">Supplier</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">Smartphone</td><td class="">EUAdapter</td></tr><tr class=""><td class="">EUAdapter</td><td class="">EUSocket</td></tr>
</tbody>
</table>
</div>
</div>
<p>If you now take a Smartphone instance and call <code>charge</code> with <code>EUAdapter.output_voltage</code> as argument, you can finally charge your phone.</p>
<pre class="language-python"><code class="language-python">smartphone<span class="token punctuation">.</span>charge<span class="token punctuation">(</span>EUAdapter<span class="token punctuation">.</span>output_voltage<span class="token punctuation">)</span>
<span class="token operator">>></span><span class="token operator">></span> Input voltage<span class="token punctuation">:</span> 5V <span class="token operator">-</span><span class="token operator">-</span> Charging<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></code></pre>
<h2 id="h-2-class-adapter" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/adapter-pattern-in-python/#h-2-class-adapter"><span aria-hidden="true">#</span></a> 2. Class Adapter</h2>
<p>You can also think that the combination smartphone + phone charger <em>defines</em> a unique system which can directly use the wall socket.</p>
<p>You started with a <code>Smartphone</code> and a <code>Socket</code>, and now you want to define a system which inherits methods and attributes both from <code>Smartphone</code> and <code>Socket</code>. You have to use <em>multiple inheritance</em>.</p>
<p>With this approach you don’t create a new entity between the client and the supplier, but you redefine the client in a way that it can directly work with the supplier. You don’t have a <code>Smartphone</code> any longer, you have a new entity which is the combination of a <code>Smartphone</code> and a <code>Socket</code>.</p>
<p>Since you are getting the <code>output_voltage</code> from a <code>Socket</code>, you have to define a method <code>transform_voltage</code> to convert a high voltage AC to a low voltage DC. Then you need to override the <code>charge</code> method inherited from <code>Smartphone</code> and call <code>transform_voltage</code> before calling the <code>outcome</code> method.</p>
<p>I decided to have two subclasses of Socket to make this example a bit closer to the real world. When you take a Smartphone and a Socket, you define a system which will work for that Smartphone and <strong>that specific type of Socket</strong> (e.g. USSocket), but will not work with the same Smartphone and a different type of Socket (EUSocket).</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">CannotTransformVoltage</span><span class="token punctuation">(</span>Exception<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""Exception raised by the SmartphoneAdapter.
This exception represents the fact that an adapter can not provide the
right voltage to the Smartphone if the voltage of the Socket is wrong."""</span>
<span class="token keyword">pass</span>
<span class="token keyword">class</span> <span class="token class-name">SmartphoneAdapter</span><span class="token punctuation">(</span>Smartphone<span class="token punctuation">,</span> Socket<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token decorator annotation punctuation">@classmethod</span>
<span class="token keyword">def</span> <span class="token function">transform_voltage</span><span class="token punctuation">(</span>cls<span class="token punctuation">,</span> input_voltage<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> input_voltage <span class="token operator">==</span> cls<span class="token punctuation">.</span>output_voltage<span class="token punctuation">:</span>
<span class="token keyword">return</span> cls<span class="token punctuation">.</span>max_input_voltage
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">raise</span> CannotTransformVoltage<span class="token punctuation">(</span>
<span class="token string">"Can\'t transform {0}-{1}V. This adapter transforms {2}-{1}V."</span>
<span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>input_voltage<span class="token punctuation">,</span> cls<span class="token punctuation">.</span>max_input_voltage<span class="token punctuation">,</span>
cls<span class="token punctuation">.</span>output_voltage<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@classmethod</span>
<span class="token keyword">def</span> <span class="token function">charge</span><span class="token punctuation">(</span>cls<span class="token punctuation">,</span> input_voltage<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">try</span><span class="token punctuation">:</span>
voltage <span class="token operator">=</span> cls<span class="token punctuation">.</span>transform_voltage<span class="token punctuation">(</span>input_voltage<span class="token punctuation">)</span>
cls<span class="token punctuation">.</span>outcome<span class="token punctuation">(</span>voltage<span class="token punctuation">)</span>
<span class="token keyword">except</span> CannotTransformVoltage <span class="token keyword">as</span> e<span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>e<span class="token punctuation">)</span>
<span class="token keyword">class</span> <span class="token class-name">SmartphoneEUAdapter</span><span class="token punctuation">(</span>SmartphoneAdapter<span class="token punctuation">,</span> EUSocket<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""System (smartphone + adapter) for a European Socket.
Note: SmartphoneAdapter already inherited from Smartphone and Socket, but by re-inheriting from EUSocket we redefine all the stuff inherited from Socket.
"""</span> <span class="token keyword">pass</span>
<span class="token keyword">class</span> <span class="token class-name">SmartphoneUSAdapter</span><span class="token punctuation">(</span>SmartphoneAdapter<span class="token punctuation">,</span> USSocket<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token triple-quoted-string string">"""System (smartphone + adapter) for an American Socket."""</span>
<span class="token keyword">pass</span></code></pre>
<p>Here are the two classes you are dealing with:</p>
<div class="table-container">
<div class="table-scroll">
<table class="">
<caption></caption>
<thead class="">
<tr class=""><th scope="col" class="">Client</th><th scope="col" class="">Supplier</th></tr>
</thead>
<tbody class="">
<tr class=""><td class="">SmartphoneEUAdapter</td><td class="">EUSocket</td></tr>
</tbody>
</table>
</div>
</div>
<p>If you now take a <code>SmartphoneEUAdapter</code> instance and call <code>charge</code> with <code>EUSocket.output_voltage</code> as argument, you can see that you can charge your phone. However, if you take the same instance and call <code>charge</code> with <code>USSocket.output_voltage</code> as argument, you get a <code>CannotTransformVoltage</code> exception. In the latter case, you are using the wrong Adapter for a particular Supplier.</p>
<pre class="language-python"><code class="language-python">smarthone_with_eu_adapter <span class="token operator">=</span> SmartphoneEUAdapter<span class="token punctuation">(</span><span class="token punctuation">)</span>
smarthone_with_eu_adapter<span class="token punctuation">.</span>charge<span class="token punctuation">(</span>EUSocket<span class="token punctuation">.</span>output_voltage<span class="token punctuation">)</span>
<span class="token operator">>></span><span class="token operator">></span> Input voltage<span class="token punctuation">:</span> 5V <span class="token operator">-</span><span class="token operator">-</span> Charging<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span>
smarthone_with_eu_adapter<span class="token punctuation">.</span>charge<span class="token punctuation">(</span>USSocket<span class="token punctuation">.</span>output_voltage<span class="token punctuation">)</span>
<span class="token operator">>></span><span class="token operator">></span> Can't transform <span class="token number">120</span><span class="token operator">-</span>5V<span class="token punctuation">.</span> This adapter transforms <span class="token number">230</span><span class="token operator">-</span>5V<span class="token punctuation">.</span></code></pre>
<h2 id="h-object-adapter-or-class-adapter%3F" tabindex="-1"><a class="header-anchor" href="https://www.giacomodebidda.com/posts/adapter-pattern-in-python/#h-object-adapter-or-class-adapter%3F"><span aria-hidden="true">#</span></a> Object adapter or Class Adapter?</h2>
<p>There are two <a href="https://stackoverflow.com/questions/5467005/adapter-pattern-class-adapter-vs-object-adapter">strong reasons</a> to prefer the Object Adapter over the Class Adapter:</p>
<ul>
<li>loose coupling</li>
<li>multiple inheritance is tricky</li>
</ul>
<p>With the Object Adapter you have <a href="https://en.wikipedia.org/wiki/Loose_coupling">loose coupling</a>, so the Client is not required to know anything about the Supplier. The Smartphone doesn’t care where it gets its 5 volts. As long as it gets them, it will charge.</p>
<p>With the Class Adapter you lose this property, because you have a new entity which is <em>defined</em> by the Client and the Supplier, and it works only for this specific type of Client and specific type of Supplier (e.g. SmartphoneEUAdapter doesn’t work with a USSocket). This means that you have created an interface which allows you to use the Client and the Supplier, but where Client and Supplier are <em>strongly coupled</em>. Since you usually want to design interfaces to <em>uncouple</em> things, this is not a desired property.</p>
<p>Another reason why I decided to define two subclasses of Socket is to show that multiple inheritance can be tricky. As we can see in the code above, <code>SmartphoneAdapter</code> already contains all attributes and methods from <code>Smartphone</code> and <code>Socket</code>. However, since what you really want to use are the subclasses of <code>Socket</code>, namely <code>EUSocket</code> and <code>USSocket</code>, you need to re-inherit when you subclass <code>SmartphoneAdapter</code>. You can use a different strategy and create <code>SmartphoneEUAdapter</code> by directly inheriting from <code>Smartphone</code> and <code>EUSocket</code>, but then you would need to do the same for <code>SmartphoneUSAdapter</code>, which needs to inherit from <code>Smartphone</code> and <code>USSocket</code>. This will result in duplicate code, because you would need to write <code>transform_voltage</code> and <code>charge</code> twice.</p>
<p>You need the code? Grab it <a href="https://github.com/jackdbd/design-patterns">here</a>!</p>