
Interestingly, exactly two years after [migrating from Python-based Lektor to Python-based Nikola](/posts/switch-from-lektor-to-nikola)
with this blog, I'm migrating to JS-based Gatsby.
Since the page was becoming out of date
and after successfully building [my company's page](https://loreyventures.com) with Gatsby in just a few hours,
I thought why not try it with this page and a little more data.

## The good

The conversion of the pages was finished really quickly.
Also converting the existing static CSS-based design over to tailwind was fun and made things much more structured
while keeping the layout 95% the same.
Overall, Gatsby with its static site generation gives you the best of both worlds:
One the one side modern frontend capabilities with React, Tailwind, and other modules,
on the other side strong SEO through static site generation.
Finally, the deployment was done within minutes just by setting up Github Actions.

## The bad

After converting layouts and pages, I started with the blog.
Since Gatsby provides much of its functionality via an GraphQL-based data layer,
you need to integrate your existing data into it as well to leverage core functionality.
As long as you keep the data simple, that's quite convenient and quickly done,
but as soon as you have customized objects,
it becomes really cumbersome.
For example, Lektor supported Markdown frontmatters with the draft status.
To replicate this in Gatsby took me more time than necessary by adapting page generation and filtering blog entries.
An explanation much better than the one I can provide,
can be found at Jared Palmer's [Gatsby vs. NextJS](https://jaredpalmer.com/gatsby-vs-nextjs).

## The ugly

Maybe it is just me being not that experienced in the Javascript world,
but I feel the integration of data with GraphQL forces you to repeat filtering logic quickly in different components.
Also, that integration basically resulting in a shared data antipattern
as data is accessed from each component directly.
Which will result in having to change several components
if you change something in the data.
For small projects this is not an issue, but for bigger projects I don't see a quick fix.
As recommended in the article linked above, I'll look into NextJS next (pun not intended).


Switching from Nikola to Gatsby (static site generators)


This guide will show you how to install Django inside Docker with docker-compose
and without installing any dependencies on the host system.
Since there are a few quirks and I have to look it up myself every damn time,
here's a guide hopefully helping me and other the next time.

This guide will:

- help you install the newest version of django inside your docker container
- be independent of your host OS and python version, so you can always get the newest version inside docker
- provide a basic setup of django inside docker with docker-compose to build upon

# Basic requirements file

Start with a requirements.txt file containing only:

```
django
```

Save it as `requirements.txt`.

# Basic Dockerfile

Firstly, a Dockerfile is needed to set up and run our project.
We'll use a basic version you can expand later.
It uses Python 3.8, installs all dependencies from your `requirements.txt` file,
and then copies your project files into the image.

So copy this in your `Dockerfile`:

```Dockerfile
FROM python:3.8

WORKDIR /code

# copy and install requirements first
# -> speeds up build if requirements haven't changed
COPY requirements.txt /code/
RUN pip install -r requirements.txt

# copy rest of files
# (not needed since we also mount a volume,
# but you won't mount in production)
COPY . /code/
```

# Basic docker-compose

In `docker-compose.yml`:

```yaml
version: "3"

services:
  web:
    build: .
    volumes:
      - .:/code/
    command: bash
    tty: True
    ports:
      # external is the port you use on your host, i.e. localhost:8000
      # internal is the port django uses inside the container
      # format: external:internal
      - "8000:8000"
```

# Running it all

After having set up the above files, run the following steps:

1. Run `docker-compose build` to build the images.
2. Run `docker-compose up` to start the container, leave it running.
3. Run `docker-compose exec web pip freeze > requirements.txt` to pin the installed dependencies to their actual version. `requirements.txt` should now contain version numbers.
4. Run `docker-compose exec web django-admin startproject YOURNAME .` (mind the dot!) to start a new django project in the current directory.
5. Run `sudo chown -R $USER ./` to own the docker-generated files. Otherwise you'll get file permission problems when working with the generated files on your host (outside of docker).
6. Run `docker-compose exec web python ./manage.py runserver 0.0.0.0:8000`.

You should now be able to access a congratulations page at `localhost:8000`.

# Next steps

If you're interested in setting up Django for production,
feel free to check out my guide on
[how to set up Django deployment for production](http://karllorey.com/posts/django-production-docker-mod-wsgi/) for the same stack.

# Note: Changing ports

If you want to change the port to access your Django application,
you only have to change the external port, i.e. the `docker-compose.yml` file.
So to switch to port 80, change the line below `ports:` to

```yaml
ports:
  - "80:8000"
```

Then the container can be accessed with `localhost:80` or just `localhost` (as 80 is the default).


Install and run Django completely inside Docker


Using Google's [G Suite](https://gsuite.google.com/) for your personal domain allows you to use your domain with all of Google's tools.
It includes Gmail (without ads), Meet, Calendar, Drive with > 30 GB, Docs, etc.
What differentiates it from a regular Google account is that you can use your personal domain and thus your own email, e.g. mail@yourname.com.
After having used a G Suite of my company for a while, I wanted to have it for my personal email and domain, too.

# Benefits

Besides being able to use your personal email for everything and as a Google account, you get:

- event/calendar invites sent to your email will be added to your calendar automatically
- people can invite you to events/docs/drive with your private email
- your Android phone has all the Google tools built in and provides a seamless experience
- and so on, I mean, everybody has a Google account nowadays

# Costs/Pricing

If you G-Suite use it for your personal domain, you'll be needing one user only.
So the price is between 6 and 25 USD per month.
You'll be needing the basic version for 6 USD only, I think.

# Getting started

So I basically signed up for a free trial.
Google asked me for my main email, i.e. the one used for your Google account.
I choose mail@mydomain.com (say hello if you like :) ).

There are only a few steps to take and the experience is seamless:

- create a G Suite account
- set up your [MX records](https://en.wikipedia.org/wiki/MX_record) to point at Google's mailservers
  (Google provided a tutorial for my registrar automatically)
- wait a little, and you're ready to go
- notify contacts of your (new) email

# Set up

Personally, I didn't change much, but here's what I found useful:

- Gmail: set up a signature, change GMail's layout
- Calendar: set up default notifications for events 30 and 3 minutes ahead
  (allows me to get notified early during deep work and just before calls)
- Drive: Migrate folders/files to your new personal Google Drive.
  You can do this by sharing files and folders from your old account to your new account.

# Misc

Some random notes:

- somehow you automatically become a business customer and can't change that, which is weird but isn't a problem in my case
- site verification, a part of the setup process, took a long time. Just give it a few minutes for the DNS records to update.


G Suite for your personal domain and email


To show, edit, and work with a model inside the Django admin site,
you usually have to add each desired model manually by adding a line to `{your app}/admin.py`.
For example, to add your `Project` class to your Admin site,
you have to add the following to your `admin.py` file:

```python
from django.contrib import admin
from myproject.myapp.models import Project

admin.site.register(Project)
```

Over time, especially during development when you create new objects a lot,
this can become quite tedious.
So I cam up with a quick hack to add all models to your Django admin site automatically.
Instead of registering all classes manually,
we simply use inspection to add all classes of our models module.
This is what you need to add to the `admin.py` file inside your app:

```python
import inspect

from django.contrib import admin

from myproject.myapp import models

for name, obj in inspect.getmembers(models):
    if inspect.isclass(obj):
        admin.site.register(obj)
```

This code uses inspection to automatically add all classes inside your models.
Note that you cannot have any other classes inside models withough extending the above code.
I would recommend to use this during development only
and add all classes manually in production.


Add all models to Django admin automatically


I decided to migrate this website/blog from Lektor to Nikola.
Both are static site generators
implemented in Python
that allow you to generate a static, secure, and maintenance-free website/blog.
Here is my experience, why I did it, how to do it, and what to expect.
In short: While I liked Lektor for its simplicity,
it lacked several features I needed.
So far, Nikola has these features and many more
which is why I'm very happy to have switched.
Read the rest of the pros and cons here.

# Background: Static Site Generators

Both Lektor and Nikola are Static Site Generators (SSG) implemented in Python.
You basically create everything on your computer and the the generator generates HTML
which can be hosted on GitHub or almost everywhere without any hassle.
Besides, it's blazing fast, secure, and needs no server configuration/maintenance.
Find an overview at [Full Stack Python](https://www.fullstackpython.com/static-site-generator.html).

# Lektor vs. Nikola

When I created this website, I chose Lektor for it's simplicity.
Compared to Wordpress, which I had used before, it was easy to set up and maintain.

## Why not Lektor?

- to little default markup: the site did not rank for quite some time which might be due to litte semantic markup
- it especially misses several essential features to rank and make the site visible
  - no RSS feed by default which allows search engines to discover new posts quickly
  - no meta tags which would enable nice previews on social media (OpenGraph for Facebook, etc.)
- default directory structure: blog posts in Lektor all have an own folder containing a file named contents.lr.
  While this looks trivial and helps with structuring files belonging to a post, I simply did not like it too much.
- Lektor does not use regular virtualenv environments which just makes stuff a little trickier to understand
- the ecosystem is small, there are not many plugins, and development seems to have halted

## Why Nikola?

Again, this is highly opinionated.

- most-active repository compared to [Lektor](https://github.com/lektor/lektor), [Pelican](https://github.com/getpelican/pelican), [MkDocs](https://github.com/mkdocs/mkdocs/), [mynt](https://github.com/Anomareh/mynt), and other static site generators in Python
- seamless integration of blog posts and regular pages
- clear documentation
- lots(!) of features and configuration options
- RSS feeds, tags, categories, blog archives, etc.
- themes with semantic HTML

# Switching from Lektor to Nikola

I basically set up a clean Nikola project and then adapted my old content.
Here's what I did with a rough time estimation:

- Reading the docs and getting started (30m)
- Setting up a clean project within my existing repo (30m)
- Renaming articles and sites (15m)
- adding Nikola-specific markdown headers (5m per site/article)
- formatting code blocks to leverage syntax highlighting and fixing markdown errors (30m)
- integrating my existing theme into Nikola (2h)
- setting up deployment on Travis CI (30m)

# What I like so far

- a lot of functionality with sane defaults
- whole transition took me less than one day, no weird errors, no frustration
- clean structure that enables (even hierarchical) posts and pages without any issues
- super fast deployment setup with github (took me one minute)
- great default/base templates with mako or jinja as a templating engine that can be adapted really easily

# What I don't like so far

- The config file is huge and includes a lot of documentation, basically very similar to Django.
  While I like to configure stuff, it's hard to find settings and understand all implications.
- Lektor has an amazing concept to define own models, I haven't figured out how to do something similar in Nikola, e.g. for my [CV](/founder).
- There are not many templates available. I've also found my template to be quite messy and without clean syntax.
- There's a plugin system heavily relying on [doit](https://pydoit.org/) which makes site-creation very efficient but extending as a result rather difficult to understand. For example, I don't know how to insert a list of all subpages into the the current page which should be quite easy.

# Issues

If you're also getting started, here are the issues I ran into.
As I said, not many and not frustrating :)

## Overwriting themes

While there are a few basic and advanced themes you can build upon,
it took me quite some time to find out where these themes can be found.
This is necessary to understand which files there are and which files you need to overwrite to adapt a theme.
Basically most templates you can inherit from can be found at [nikola-themes](https://github.com/getnikola/nikola-themes)
but some basic templates are stored within [Nikola itself](https://github.com/getnikola/nikola/tree/c886ea38f7ee34f1fffb3edae7087694483a999d/nikola/data/themes).
If you've understood this, you basically just copy/paste the files you want to adapt inside your own theme
which works really nice.

## Deployment with existing gh-pages

Since I had previously set up Lektor deployment via GitHub Pages, I already had a `gh-pages` branch.
So when I tried to deploy via `nikola github_deploy`, I got the following error:

```text
To github.com:lorey/karllorey.com.git
 ! [rejected]        gh-pages -> gh-pages (fetch first)
error: failed to push some refs to 'git@github.com:lorey/karllorey.com.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
# .stacktrace
[2020-02-25T15:12:28Z] ERROR: github_deploy: Failed GitHub deployment -- command ['ghp-import', '-n', '-m', 'Nikola auto commit.\n\nSource commit: 5d74f162affaf80691de1d7c7b0d8728addea96a\nNikola version: 8.0.4', '-p', '-r', 'origin', '-b', 'gh-pages', 'output'] returned 1
```

Obviously, I forgot to pull the gh-pages branch first, so the new commit could be added on top.
After doing a `git pull origin gh-pages`, deploying worked like a charm.

## Deployment with Travis CI

To deploy with Travis CI, there's a great [post on the Nikola blog](https://getnikola.com/blog/automating-nikola-rebuilds-with-travis-ci.html).
What I did not understand at first was the distinction between the `master` and a `src` branch.
So the `src` branch is used for user pages that are deployed into the master branch, i.e. when you have a site pointing to you username, e.g. `lorey.github.io`.
If you do that, you have to use another branch to deploy to avoid an infinite loop (push to master > deploy to master > ...).
Since I use `gh-pages` as the branch containing all the generated HTML, I set up .travis.yml to deploy on pushes to master.
In the `conf.py` I still set:

```python
GITHUB_SOURCE_BRANCH = 'src'
```

Seems to work fine.

Another thing I encountered when setting up Travis CI:
Somehow the travis gem had issues when logging in so `travis login` and `travis enable` did not work.
Maybe because I use an old version or have 2FA enabled.
In the end I just ran `travis encrypt-file id_rsa --add` which worked fine and added a line to .travis.yml
storing the encrypted key.

Doing a `git branch master FETCH_HEAD` inside the .travis.yml failed:

```text
$ git branch master FETCH_HEAD
fatal: A branch named 'master' already exists.
The command "git branch master FETCH_HEAD" failed and exited with 128 during .
```


Switching from Lektor to Nikola (static site generators)


This article will show you how to run Django in production with docker and apache2/mod_wsgi.
While there are several guides on how to do it,
I found no simple enough tutorial on how to do it
since all of the existing solutions require a lot of configuration or a custom docker image.
The solution I found is way quicker and requires close to no configuration.

Guides I found:

- [Django with Apache and mod_wsgi](https://docs.djangoproject.com/en/3.0/howto/deployment/wsgi/modwsgi/) assumes you have installed everything already and requires configuration possibly unnecessary for a dockerized version
- [mod_wsgi](https://modwsgi.readthedocs.io/en/develop/) explains how to install mod_wsgi, but not how to use Django with it

So, let's do this.
If you just want to see the final Dockerfile, scroll to the end of the article :)

## Prerequisites

I assume you have some Dockerfile with Django application setup
and run Django via the command that spins up the development server:

```dockerfile
FROM python:3.6

WORKDIR /code/

# copy and install requirements first to leverage caching
COPY requirements.txt /code/
RUN pip install -r requirements.txt

# copy the actual code
COPY . /code/

CMD ./manage.py runserver 0.0.0.0:8000
```

## Step 1: Install Apache

We basically need a working Apache setup to run python code via mod_wsgi.
So in our Dockerfile, we need to install apache2 and apache2-dev.

```dockerfile
RUN apt-get install apache2 apache2-dev
```

## Step 2: install mod_wsgi

We want to run our project within a container.
The mod_wsgi documentation says the easiest and preferred using docker
is to use mod_wsgi-express as it does not require any configuration.
mod_wsgi-express can be used as a command after you have installed it via pip.
So we need to add:

```dockerfile
RUN pip install mod_wsgi
```

## Step 3: run Django inside Apache with mod_wsgi

Lastly, we simply have to run mod_wsgi-express
which will start an apache instance with our Django project.
This can be achieved via:

```text
mod_wsgi-express start-server /code/project_name/wsgi.py --user www-data --group www-data
```

The wsgi.py file is auto-generated by Django's `startproject` command and inside you app's folder,
i.e. `django-project/project_name`.
So, to run our project within Apache after we've set-up everything in the Dockerfile,
we have to add a `CMD` command at the end of the Dockerfile:

```dockerfile
CMD mod_wsgi-express start-server /code/connect_web/wsgi.py --user www-data --group www-data
```

The `--user` and `--group` parameters make sure Apache isn't run as root which results in errors.
If you want to find out, just run the command without them.

## Final Dockerfile: Installing and running Apache and mod_wsgi within Docker

Your final Dockerfile should now look like this:

```dockerfile
FROM python:3.6

# update packages
RUN apt-get -qq update
RUN apt-get install --yes apache2 apache2-dev
RUN pip install mod_wsgi

RUN mkdir /code
WORKDIR /code

COPY . /code/

CMD mod_wsgi-express start-server /code/project_name/wsgi.py --user www-data --group www-data
```

## Troubleshooting

If you have any problems, check the logs. Their locations are outputted when starting the container:

```text
web_1     | Server URL         : http://localhost:8000/
web_1     | Server Root        : /tmp/mod_wsgi-localhost:8000:0
web_1     | Server Conf        : /tmp/mod_wsgi-localhost:8000:0/httpd.conf
web_1     | Error Log File     : /tmp/mod_wsgi-localhost:8000:0/error_log (warn)
web_1     | Request Capacity   : 5 (1 process * 5 threads)
web_1     | Request Timeout    : 60 (seconds)
web_1     | Startup Timeout    : 15 (seconds)
web_1     | Queue Backlog      : 100 (connections)
web_1     | Queue Timeout      : 45 (seconds)
web_1     | Server Capacity    : 20 (event/worker), 20 (prefork)
web_1     | Server Backlog     : 500 (connections)
web_1     | Locale Setting     : en_US.UTF-8
```

If you want to monitor them, just `tail -f` them:

```text
tail -f /tmp/mod_wsgi-localhost:8000:0/error_log
```

### Permission problems

At first, I got an `Internal Server Error` when opening the page on my machine.
After checking the logs as described above, mod_wsgi seemed to have problems with file permissions on `.logs/debug.log`.
A simple `chown www-data /code/.logs/debug.log` from within the container
which makes www-data the owner of the file solved it for me.


Django in Production with mod_wsgi and Docker


Lost the private key for your EC2 instance and can't login via ssh anymore?
This tutorial will show you how to recover your EC2 instance by setting a new key pair to login.

What we'll do:

- mount the original instance's volume (a.k.a. it's filesystem) inside another temporary EC2 instance
- modify the keys allowed to login
- unmount the volume from the temporary instance and re-mount it in the original instance
- login with a new key to your original instance

In short, this replaces the key needed for ssh to connect with a new one of your choice.

## AWS Recovery Automation

There's an [Amazon recovery automation thing](https://aws.amazon.com/premiumsupport/knowledge-center/recover-access-lost-key-pair/) available that aims to recover your instance automatically,
it sadly did not work for me.
So here we go.

## Step 1: Find the instance

Go to the AWS EC2 console and find your (lost) instance.
Make sure you're in the right availability zone.
Note down the instance ID as well as the subnet.
Also note the instance's volume

## Step 2: Create a temporary instance

Launch an instance in the same availability zone.
Make sure to use the same subnet.
Create a new key pair with that instance
or use the key pair you'd like to use for your original instance from now on.

## Step 3: Attach the original volume to a temporary instance

Stop the original instance to be able to unmount the storage.
Note the volume ID under `Attachment information`.
Go to volumes and detach the volume with `Actions - Detach Volume`.
Attach the volume to the temporary instance with `Actions - Attach Volume`.
Choose one of the given options and note it down, e.g. `/dev/sdf`.
Connect to the temporary instance via SSH and mount the volume, e.g. to `/data` via the following command:

```text
mount /dev/sdf /data
```

The volume of the original instance is now mounted to `/data`.
This allows us to now modify the allowed keys.

## Step 4: Modify the allowed keys

We can now set the key of the current, temporary instance as an allowed key of the original instance.
The keys allowed to log in are stored in a file called `~/.ssh/authorized_keys` (background on [authorized_keys](https://www.ssh.com/ssh/authorized_keys/)).
Inside this file is just a line-by-line list of authorized keys.
Because of this, we can just append the file of our temporary instance (and thus our key from the temporary instance)
to the file of our original instance.

```text
cat ~/.ssh/authorized_keys >> /data/home/admin/.ssh/authorized_keys
```

Make sure to swap admin with the actual user you want to sign in as (check your ssh connection command if you're unsure).

### Check if everything went right

```text
cat /data/home/admin/.ssh/authorized_keys
```

should now contain the contents of:

```text
cat ~/.ssh/authorized_keys
```

## Step 4: Bring everything back in order

We now have set up another key for login.
All that's left is to unmount the volume from the temporary instance and mount it to the original instance.

First, we have to unmout the storage of the original instance inside the temporary instance by
`umount /dev/sdf` (make sure to use the right path here).
Afterwards, you stop the temporary instance via the AWS console under `Instances`.
You then attach the volume to the original instance inside the AWS console under `Volumes` via `Actions - Attach volume`.
Type in the original instance ID as well as `xvda` as the mount point.
Otherwise, you might get an error pointing out that there's no root volume when starting the instance.
You can now re-start the original instance and should be able to login with the new key.

Make sure to delete the temporary instance in case everything went well.

## Conclusion

So this guide showed you how to recover an AWS EC2 instance if you lose you private key.
We did this by using a temporary instance to swap or actually extend the authorized_keys file.
You should now be able to login to the original instance with your new key.

There's also a slightly different [guide by AWS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#replacing-lost-key-pair).


How to recover an AWS EC2 instance without the private key


I've come to like the code formatter [Black](https://black.readthedocs.io/en/stable/index.html) for Python.
It's opinionated, deterministic and thus very minimalistic.
And since I'm using it on more and more projects, I wanted to integrate it into my workflow.

So usually, you just install Black globally via `pip3 install black`.
But since I use Debian which still ships with Python 3.5
this yielded some smaller challenges as Black only runs under Python 3.6+.
This post is a small tutorial on how you can use Black on Debian and integrate it into the command line, PyCharm, and IdeaVim.

## Challenge 1: Python 3.6+ on Debian

Since Debian currently ships with Python 3.5, I needed to get Python 3.6 running somehow.
While installing [Anaconda](https://www.anaconda.com/distribution/) or [Python 3.6+ manually](https://www.python.org/downloads/)
are other solutions [might](https://community.hortonworks.com/idea/212478/independent-python-vs-anaconda-python.html) [work](https://unix.stackexchange.com/q/332641),
I decided to simply use Docker, as this is what I use for my regular development anyhow.
Simplest solution was to use an existing docker image,
namely [jbbarth's docker-black](https://github.com/jbbarth/docker-black),
which allows you to mount the current working directory into a (newly created) container, format the desired file, and throw the container away afterwards.
Sounds more complicated than it is, you just run one single command et voila.
This method has the additional benefit that it adheres to any [pyproject.toml](https://github.com/psf/black#pyprojecttoml) which can store configuration like line length.
There's also a more sleek image at [cytopia/docker-black](https://github.com/cytopia/docker-black) which might take less space.

So to run black irrespective of your local Python installation via `black main.py`,
you run

```text
docker run --rm -v $(pwd):/code jbbarth/black main.py
```

This creates a new container, mounts the current working directory into `/code` and formats `main.py`.
Afterwards, the `--rm` flag will delete the container as well as the created volume.
So far, so good, but as this command is quote long, I had to build an alias to invoke it quickly as a next step.

## Challenge 2: Black from command line

So to avoid the cumbersome docker syntax each time, you now want an alias to just run `black main.py` everywhere.
To do this, you have to map the black command to your black running inside docker.
You can do this by adding

```text
black() { docker run --rm -v $(pwd):/code jbbarth/black $*; }
```

to your bashrc or zshrc.
After opening a new terminal, you should now be able to invoke the black formatter inside docker by running `black ...`.

## Challenge 3: Black in PyCharm

To now integrate this setup into PyCharm,
you have to slightly adapt the [offical installation instructions](https://black.readthedocs.io/en/stable/editor_integration.html#pycharm-intellij-idea).
Go to `File -> Settings -> Tools -> External Tools`.
Click the + icon to add a new external tool with the following values:

- Name: `Black`
- Description: `Black code formatter`
- Program: `/usr/bin/docker`
- Arguments: `run --rm -v $FilePath$:/$FilePath$ jbbarth/black "$FilePath$"`
- Working directory: `$ProjectFileDir$`

Test it by running it with an opened python file via `Tools -> External Tools -> Black`.
After you made sure it works, re-open it again and untick `open console` to avoid a new console at every run.
You can basically mirror this guide to install a file watcher
that formats on every save (see [the docs](https://black.readthedocs.io/en/stable/editor_integration.html#pycharm-intellij-idea)).

Note: As you can see, this only mounts the current file and thus does not adhere to any config files.
The same applies for the next step as it builds upon this one.

## Challenge 4: Black in IdeaVim

Now for the bonus part: to trigger this setup quickly from within IdeaVim and format the current file with a single command,
equivalent to the [regular vim plugin](https://black.readthedocs.io/en/stable/editor_integration.html#vim),
we have to map the `:Black` command to our external command in PyCharm.
To do this, we edit our `.ideavimrc` file where all IdeaVim configuration is stored,
and add the following line:

```text
command Black action Tool_External Tools_Black
```

Now typing `:Black` will re-format the current file.

## Summary

This tutorial showed you one way to include Black into you daily development (esp. on Debian).
If you have any questions or feedback, hit me up on Twitter [@karllorey](https://twitter.com/karllorey)
or any of the other platforms listed below.

## Notes

### Deleting all docker containers of a specific image

If you want to remove all containers derived from a specific image, e.g. if you forgot to add the --rm flag:

```text
docker rm $(docker ps -a --filter ancestor=jbbarth/black -q)
```


How to set up Black with Debian, PyCharm, and IdeaVim


For work, I do a lot of data analysis to find the most promising young startups.
As a first step, you always have to import the desired data into a Pandas DataFrame
and do some preprocessing, for example by importing JSON data from some API. 
When doing this kind of pre-processing,
you usually have a lot of temporary columns in your DataFrame that get imported but need to be dropped later in the process.
To deal with these temporary columns,
I built a custom Context Manager that keeps track of all imported columns
and deletes them when you're done.
This way, your code stays lean and you don't have to remove temporary columns yourself.
In this short article, I will show how you can keep your pre-processing clean
and use a Python ContextManager to clean up temporary columns.

In this example I will use the actual code I use for importing data from the API of our CRM named Hubspot.
What I retrieve is a list of companies stored as a list of Python dictionaries.
To import a list of dictionaries in pandas you basically do:

```python
from pandas.io.json import json_normalize

df = json_normalize(data)
```

The json_normalize function generates a clean DataFrame based on the given `data` parameter and normalizes the hierarchy so you get clean column names.
This is especially useful for nested dictionaries.

## Ugly: Keeping imported columns

The problem with json_normalize is that you usually only want a subset of the imported columns,
mostly with different names or some kind of pre-processing, too.
So you might be tempted to do something like this:

```python
from pandas.io.json import json_normalize

df = json_normalize(data)

df['company_id'] = df['companyId']
df['location'] = df['properties.city.value']
df['name'] = df['properties.name.value']
df['domain'] = df['properties.website.value']
//..apply(), .as_type(int), whatever...
```

This works, but keeps all the imported columns inplace and might take a lot of storage.
So what can you do?

## Ugly: Dropping columns manually

So after importing, you want to get rid of all temporary columns from the import.
To do this, you have to either select the columns you want or drop all columns you don't want.
In both cases, you have to somehow keep track of the temporary columns or the ones you want to keep.
To deal with this, one solution would be to prefix temporary columns and delete them afterwards:

```python
from pandas.io.json import json_normalize

df = json_normalize(data)

// make temporary columns
df.columns = ['temp_' + c for c in df.columns]

// pre-processing, basic calculations, etc.
df['company_id'] = df['temp_companyId']
df['location'] = df['temp_properties.city.value']
df['name'] = df['temp_properties.name.value']
df['domain'] = df['temp_properties.website.value']
//..apply(), .as_type(int), whatever...
```

Afterwards, you would then select all desired columns or drop all undesired columns.

```python
df.drop([c for c in df.columns if c.startswith('temp_')], axis=1, inplace=True)
// or
df = df[[c for c in df.columns if not c.startswith('temp_')]]
```

While this works, it feels bloated and inefficient.
You have to prefix all the value names in the code which results in bloated column names.
You also have to keep track of column names you want in the end
or the used prefix in different places.
Just imagine you have to change the prefix `temp_` one day or make the code work with a different prefix.

## Clean and easy: using a Context Manager

After having used the above methods for some time, it struck me that [Python Context Managers](https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/) might be a cleaner solution.
You might know them from their most popular application `with open() as file:`.
If not, please take a few minutes to read more about them.
To make things short: They basically ensure that something, usually a cleanup, is executed in each exit scenario,
whether it is a usual exit like a return or an exception.
I thought I might use this to build a clean solution that keeps track and gets rid of temporary columns.
So I built a Context Manager that deals with temporary columns when importing JSON data so I don't have to.
You can basically use it like this:

```python
with DataFrameFromDict(companies) as df:
    // imported dict now in df, same result as json_normalize
    df['company_id'] = df['companyId']
    df['location'] = df['properties.city.value']
    df['name'] = df['properties.name.value']
    df['domain'] = df['properties.website.value']
// after context exits, df contains company_id, location, name, and domain
// but no more temporary columns
print(df)
```

The benefit: You don't have to keep track anymore and the context manager handles the deletion of all temporary columns.

## How it works

You can just copy and paste the following snippet to get going, I'll explain how it works below:

```python
class DataFrameFromDict(object):
    """
    Temporarily imports data frame columns and deletes them afterwards.
    """

    def __init__(self, data):
        self.df = json_normalize(data)
        self.columns = list(self.df.columns.values)

    def __enter__(self):
        return self.df

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.df.drop([c for c in self.columns], axis=1, inplace=True)
```

When opening the context, `__init__` and `__enter__` get called.
They create the DataFrame and remember all imported and thus temporary column names.
When the context is exited, `__exit__` makes sure to drop all previously created columns
and leaves only the newly created columns behind.

Hope this helps you to create a clean pre-processing pipeline.
Let me know what you think.
You can find the [code on GitHub](https://gist.github.com/lorey/2b57b4ebfec4d45221e15a49060f80d2).

Further reading:

- [json_normalize](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.io.json.json_normalize.html)
- [Python Context Managers](https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/)


Keeping Pandas DataFrames clean when importing JSON (with Context Managers)


I have long been thinking about writing a summary of my year
to reflect on things and to be able to see how I saw the world in a few years.
Also, if you haven't heard of me in 2018, this is the one article you should probably read.

As you probably know, I founded a venture capital fund, First Momentum, at the age of 27 in late 2017—at a point when I had not even finished university.
Gladly, I graduated last year while getting the fund running, so this year was all about getting traction.
Most notably this year, we managed to do our first closing and could since do our first investments.
Personally, I started to lead the investing team at First Momentum, currently consisting of five people.

So here are my reflections on this year, my learnings, and my goals for 2019.

## Overview of my 2018

To keep things short, I will use a list to give you an overview of my 2018:

- After founding the fund in 2017, we got registered with BaFin at the beginning of 2018, meaning we passed at regulatory checks. A huge milestone.
- So it was all about fundraising as quick as possible afterwards. Making call, having meetings, presenting at events.
- In June, we were able to to our First Closing. For funds, this is the kick-off and enabled us to actually invest for the first time.
- Quickly afterwards, we announced our first investment in August.
- After the first closing, I switched from fundraising over to investing and started to lead the investing team.
- We managed to grow our investment funnel radically until the end of the year and will announce several investments soon.

## My Challanges

To start the actual review of 2018, I will begin with my main challenges this year.

### Callenge 1: less tech, more management

First and foremost, the thing that was really hard this year was the switch from tech to management.
Since I've mostly had CTO or programmer roles before, I always dealt with a lot of tech.
This year, when building the fund I mostly did fundraising, consisting of calls and meetings.
Later on, when looking for our first investments, I had to build a team, manage it, and still do meetings and calls.
While I like that very much, too, my passion is to build technology.
And most importantly, coding just feels like my personal ikigai.
It doesn't bother me to be coding and thus working 16 hours a day.
I don't get tired, I don't get bored, and I feel extremely satisfied in the evening.
Even after 16 hours of coding, I have to push myself to stop and go sleeping to be able to retain my run the next day.
With other work, especially management, it just does not feel the same.
You don't single-task one thing, you're multitasking >10 things a day.
A while I was able to maintain a solid workload without much coding this year,
I just miss the feeling of complete focus and being in the zone for extended periods of time.

### Challenge 2: letting fires burn

The main challenge at First Momentum has been the constant struggle to do the right thing.
There are always more things you should be fixing than you can actually fix.
Reid Hoffman's Masters of Scale podcast even has [an own episode](https://www.reidhoffman.org/why-the-best-entrepreneurs-let-fires-burn/) on it.
It can be compared with several fires burning around you at once.
While you natural reaction would be to try to extinguish all fires,
your best reaction is to assess the fires and only deal with the ones, that will kill you within the next month,
so you still have time to think about your actual strategy.
In reality, there are so many distractions and seemingly urgent things that shoot at you every day.
Usually, your reaction would be to react and deal with everything (put out fires).
What you actually should be doing is assessing their importance and to say NO more often (letting fires burn).
In conclusion, the challenge often was to act instead of react and thus have and keep a clear strategy.
This takes practice, a lot of effort, and you have to remind yourself over and over again.

### Challenge 3: ageing, staying healthy

Most of people reading this will probably laugh.
But age has been an important aspect of my life this year.
Not only was it the first year I have actually noticed to age,
but it was also an the first year not being a student anymore which also makes me feel older.
On top of that, I'm the second-oldest at First Momentum and get reminded from time to time.
Furthermore, its quite a challenge to stay healthy when working more than 60 hours over extended periods of time.
To deal with age and health, I've started to do more sports, eat healthy, and sleep more.
Not that I did not do it before, I now do it with more purpose and not only just for fun when I feel like it.

### Challenge 4: staying social when stakes are high

Because of the workload this year, staying social has been quite a challenge.
I mean, its easy to be social during university:
daily classes, lots of parties, enough time to socialize.
But since the workload has risen this year,
I have to actually focus on keeping in touch with friends for the first time in my life.
It's just not as easy to have a beer or make a call after a long work-day.

## My Learnings

2018 has not only been about challenges of course.
When founding a startup, it's basically all about learning, fast!
So here are my top learnings this year:

### Learning 1: say no. let fires burn.

As a founder, your first priority is to act strategically.
To do this, you need to have time, be relaxed, and focus.
Like mentioned in my 2nd challenge this year, distractions are everywhere:
events with pitches or networking, people asking you for calls and meetings, startup competitions, you name it.
While not all of these activities are a waste of time in general, they can become one quickly.
The one main question to ask yourself is: what is the best way to spend these eight hours?
Is it really the conference?
Or rather a one-on-one with a key employee and six calls with excellent founders you already know?
So this year was all about learning to say no.
Saying no to pitches where our target audiences (founders or investors) are not present.
Saying no to calls and meetings without a clear purpose or benefit.
And saying no to startup competitions without huge monetary or marketing benefits.
This way, you will have a calendar with much more space to focus on the actually important things.

If you'd like to learn more:
There's a great article on [Why productive people have empty schedules](https://www.fastcompany.com/3009536/why-productive-people-have-empty-schedules) including Warren Buffet.
Tim Ferris also has two great episodes on saying no ([#282](https://tim.blog/2017/11/25/how-to-say-no/) and [#328](https://tim.blog/2018/07/19/essentialism/))

### Learning 2: focus on what's really important

So after having cleared your schedule by saying no more often and by letting fires burn the question remains:
What is the most important thing I should be working on?
I don't know.
And I'm often still not sure.
I guess nobody launching a startup is and can be sure because of the intrinsic uncertainty.
But what I learned in 2018 is that you can learn to focus on what's really important.
I basically started to take more time to re-think our status quo, week after week.
If you hear it, it may sound obvious, but things get ugly when you have a tight schedule, an unbearable workload, or deadlines to meet.
But still: nothing is more important than to reflect and to re-adjust your strategy.
For these strategy sessions, these are some of the questions I think about:

- what brings you one step closer to your vision?
- what is the task with the most leverage?
- which low-risk experiment could yield huge wins?
- which are the things that take most of your time? could they be more efficient? could they be replaced or even left out?
- what is the one hurdle that makes your life hard right now?

Beside these questions, here are some topics I learned to focus on more:

- continuous self-improvement: your priority is to grow
- thinking in long-term games: what can you do today to win in one year?
- startup culture: will keep your team together in tough times
- learning from others (podcasts, articles, mentors): others had your problems before and already solved them.

Naval Ravikant has great tweetstorms on stuff like this, for example [How to Get Rich (without getting lucky)](https://twitter.com/naval/status/1002103360646823936).
There's also a great [overview article on Medium](https://medium.com/@noahmadden/navalism-quotes-perceptions-by-naval-ravikant-a5fd60ac5788).
If you want to learn more about strategy and deriving tasks, I can highly recommend to look into OKRs and everything around them.
Also there's is a great episode on [Masters of Scale: How to kill your bad ideas](https://mastersofscale.com/#/mark-pincus-how-to-kill-your-bad-ideas-masters-of-scale-podcast/).
And my friend Feliks has written a great article on [strategy vs. execution](https://medium.com/@feliks/the-biggest-startup-strategy-myth-and-how-to-deal-with-it-455b034e93e8).

### Learning 3: growing your team is hard work

My company, First Momentum, grew from five founders to twelve employees this year.
While this might not seem a lot, it is hard work.
Firstly, you have to make sure your company culture stays on track.
While the founders may have a common sense of what the company stands for, new employees don't und you thus have to make sure you communicate these values well and often.
Secondly, the amount of work required to manage a team is underestimated.
You have to guide employees, explain things, and simply catch up with the emotionally from time to time.
Especially when new employees join the team, your deep work as a founder is reduced drastically.

### Learning 4: others had your problems before

It's easy to bury one's head in the sand when working hard and to forget to look left and right.
But in fact, you should do the opposite:
Many other founders had the same problems as you before.
So you can either try to come up with a solution yourself which takes a lot of time and effort.
Or you can simply approach others and get feedback.
The natural tendency is to come up with an own solution,
but the smart solution is to stand on the shoulders of giants and learn from others.

## My Achievements

- Made our first closing along with national press coverage
- Made our first investments
- formed a team of outstanding people. established culture and onboarding.
- competitive technological advantage. scrapers and reporting.
- released my first OSS library and re-launched my personal website
- founded my own holding company

## 2019 by the numbers

- sleep
- work
- challenges

## Miscellaneous

- neglecting my laptop.
- moving to the office full-time
- making extensive use of a notebook helps a lot.
- best hike of my life.
- pitching one billionaire
- closing a deal at the first call
- best talk this year at Let's Hack in Münster
- started to form an awesome team
- buying a new bike and doing more sports to relax
- part of jury with frank thelen.
- first year I'm not studying (part-time) anymore

Platforms, Services and Apps I'm using regularly

- Habits: tracking my habits (daily)
- Libra: tracking my weight (daily)
- Sleep for Android: tracking my sleep (daily)
- Instagram: keeping in touch with friends, getting inspired (daily)
- Twitter: getting news, networking (every other day)
- Linkedin: networking (weekly)
- Audible: listening to audiobooks (every other day)
- Netflix: watching TV (every other day)
- Spotify: listening to music (daily)

## The next year

Thank god, it's 2019.

---

## Appendix

To improve my reflection, I have used [YearCompass]().
It's a 20-page PDF with many questions about your past year that helps you to structure your self-reflection a lot.

This post was inspired by [Martin Thoma's new year blogs](https://martin-thoma.com/new-year-2019/)
and [Mathias Ockenfels' first year at Speedinvest X](https://medium.com/speedinvest/our-first-year-ef1558dc081e).
Thank you so much.
I later found out that [Tim Ferris is proposing to do a year-review instad of new year resolutions](https://tim.blog/2018/12/28/past-year-review/).


Founding a Venture Capital Fund as a Techie: My Year 2018


For some time I have been using a notebook now.
It's simple, analogue, and convenient nonetheless.
I first noticed people using one in university, when we heard talks from founders at our university club.
At that time, I bought one myself to try it out and have never looked back.
And I think every founder should use one.

In this article I would like to articulate my love for notebooks.
And I would like to help you get started using one.
First, by talking about my usage.
Secondly, by summarizing the benefits.
And lastly, by giving some recommendations on which notebook to choose and how to structure it.

## How I use my notebook

I basically carry my notebook with me all the time.
Anywhere.
As soon as something comes to my mind: I write it down.
For every meeting: I create a meeting note.
For every call: I write a note.
For every task that comes to my mind: I add a task to my ToDo note.
For every thought or self-reflection: I make a note.
As a result, if I need to recall something, I can just look it up.
No more searching across different places.
And since everything is chronological, you find things quickly.

## My top 5 reasons to use a notebook

So here are my top reasons to use a notebook in order of importance,
probably biased by my viewpoint from founder/investor perspective.

### 1. Compendium Character

The most important reasons for me is that all of my notes are in one place.
This has two significant advantages.
One being that you do not have to switch between tools and write everything down in one notebook.
The other being that you also don't have to search in different places.
While digital solutions offer this too, nothing keeps your notes together like one notebook.

### 2. Portable anywhere

Another important reason for an analogue solution and thus a notebook is that you can carry a notebook anywhere.
Sure, you can take your phone anywhere, too.
But what about meetings, dinners, long travels, vacations, or nature?
A notebook needs no power, no internet connection and is always available.
I think the major advantage over a phone or tablet is that nothing will distract you when using it.
And if everything else has an empty battery, your notebook is still there.

### 3. Unobtrusive usage

Building upon the last point, I would like to emphasize that analogue solutions still are more broadly accepted by society.
I found this to be especially useful for business meetings or lunches where the usage of a smartphone or tablet would be frowned upon.
On the contrary, writing in your notebook shows that you take the other person seriously and consider their advice.

### 4. One-handed sketching and note-taking

Another big advantage for me is that handwriting is the fastest way (known to me) to take one-haded notes or make sketches.
But why is this important?
While people might be able to type faster with a keyboard or even with a phone,
when I take notes I often include sketches, layouts, or even drawings.
The benefit of handwritten notes and a notebook is that you can do it one-handed,
for example while being on the phone.

### 5. Digitizable later

While many digital solutions to take notes exist,
to me, none have all of the aforementioned benefits.
So if you still want to use notes in a digital format later-on,
you can always digitize specific pages or the whole book if you like.

## My Recommendations

Finally, some recommandations on structuring and choosing your notebook.

### Structuring your notebook

To dive deeper into the topic, I can highly recommend the [bullet journal method](https://bulletjournal.com/pages/learn)
as a starting point to structure your thoughts.
First and foremost, simply make it a habit to take your notebook with you all the time.
The rest will follow automatically by using it.

### Choosing a notebook

If you're thinking about buying a notebook, I would recommend the following:
My go-to format would be something close to DinA5 as is very portable and not too bulky.
Keep in mind that you need to choose a smaller format if you want to carry it around in your pocket.
For the layout, I prefer a dotted layout over a squared one,
as it gives you equally good orientation but more freedom.
Thus, I think the most popular choice would be a [Moleskine](http://a.co/d/6W1PsBp).
Personally, I prefer the [Lechtturm1917](http://a.co/d/cTNSIV5) version though.
The design is solid and very similar to a Moleskine, but it offers two significant benefits:
Firstly, the pages are numbered, which is essential for me for later reference.
Secondly, it comes with two bookmarks instead of one.
Also make sure to use the same pen everytime, it makes writing so much simpler.

If you have any questions, additions, comments, thoughts, let me know!
Have fun with your notebook.

Further reading:

- [HN: How do you keep track of your creative thoughts?](https://news.ycombinator.com/item?id=18837345)
- If you are into digital solutions, I can highly recommend [notion.so](https://notion.so) and some [advanced Notion.so setups](https://www.youtube.com/watch?v=w_mh91IRLL8)


Lorey Ipsum