Configuration system for CLI (#6708)

* Rework how bin/qmk handles subcommands

* qmk config wip

* Code to show all configs

* Fully working `qmk config` command

* Mark some CLI arguments so they don't pollute the config file

* Fleshed out config support, nicer subcommand support

* sync with installable cli

* pyformat

* Add a test for subcommand_modules

* Documentation for the `qmk config` command

* split config_token on space so qmk config is more predictable

* Rework how subcommands are imported

* Document `arg_only`

* Document deleting from CLI

* Document how multiple operations work

* Add cli config to the doc index

* Add tests for the cli commands

* Make running the tests more reliable

* Be more selective about building all default keymaps

* Update new-keymap to fit the new subcommand style

* Add documentation about writing CLI scripts

* Document new-keyboard

* Update docs/cli_configuration.md

Co-Authored-By: noroadsleft <18669334+noroadsleft@users.noreply.github.com>

* Update docs/cli_development.md

Co-Authored-By: noroadsleft <18669334+noroadsleft@users.noreply.github.com>

* Update docs/cli_development.md

Co-Authored-By: noroadsleft <18669334+noroadsleft@users.noreply.github.com>

* Update docs/cli_development.md

Co-Authored-By: noroadsleft <18669334+noroadsleft@users.noreply.github.com>

* Address yan's comments.

* Apply suggestions from code review

suggestions from @noahfrederick

Co-Authored-By: Noah Frederick <code@noahfrederick.com>

* Apply suggestions from code review

Co-Authored-By: Noah Frederick <code@noahfrederick.com>

* Remove pip3 from the test runner

6 files changed, 408 insertions, 54 deletions

diff --git a/docs/_summary.md b/docs/_summary.md
index 4e87d8f1f..233b2cdaa 100644
--- a/docs/_summary.md
+++ b/docs/_summary.md
@@ -9,6 +9,7 @@
 * [QMK Basics](README.md)
   * [QMK Introduction](getting_started_introduction.md)
   * [QMK CLI](cli.md)
+  * [QMK CLI Config](cli_configuration.md)
   * [Contributing to QMK](contributing.md)
   * [How to Use Github](getting_started_github.md)
   * [Getting Help](getting_started_getting_help.md)
@@ -48,7 +49,7 @@
   * [Useful Functions](ref_functions.md)
   * [Configurator Support](reference_configurator_support.md)
   * [info.json Format](reference_info_json.md)
-  * [Python Development](python_development.md)
+  * [Python CLI Development](cli_development.md)
 
 * [Features](features.md)
   * [Basic Keycodes](keycodes_basic.md)
diff --git a/docs/cli.md b/docs/cli.md
index 4b8472b19..cb609e2a9 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -4,22 +4,70 @@ This page describes how to setup and use the QMK CLI.
 
 # Overview
 
-The QMK CLI makes building and working with QMK keyboards easier. We have provided a number of commands to help you work with QMK:
+The QMK CLI makes building and working with QMK keyboards easier. We have provided a number of commands to simplify and streamline tasks such as obtaining and compiling the QMK firmware, creating keymaps, and more.
 
-* `qmk compile`
-* `qmk doctor`
+* [Global CLI](#global-cli)
+* [Local CLI](#local-cli)
+* [CLI Commands](#cli-commands)
 
-# Setup
+# Requirements
 
-Simply add the `qmk_firmware/bin` directory to your `PATH`. You can run the `qmk` commands from any directory.
+The CLI requires Python 3.5 or greater. We try to keep the number of requirements small but you will also need to install the packages listed in [`requirements.txt`](https://github.com/qmk/qmk_firmware/blob/master/requirements.txt).
+
+# Global CLI
+
+QMK provides an installable CLI that can be used to setup your QMK build environment, work with QMK, and which makes working with multiple copies of `qmk_firmware` easier. We recommend installing and updating this periodically.
+
+## Install Using Homebrew (macOS, some Linux)
+
+If you have installed [Homebrew](https://brew.sh) you can tap and install QMK:
+
+```
+brew tap qmk/qmk
+brew install qmk
+export QMK_HOME='~/qmk_firmware' # Optional, set the location for `qmk_firmware`
+qmk setup  # This will clone `qmk/qmk_firmware` and optionally set up your build environment
+```
+
+## Install Using easy_install or pip
+
+If your system is not listed above you can install QMK manually. First ensure that you have python 3.5 (or later) installed and have installed pip. Then install QMK with this command:
+
+```
+pip3 install qmk
+export QMK_HOME='~/qmk_firmware' # Optional, set the location for `qmk_firmware`
+qmk setup  # This will clone `qmk/qmk_firmware` and optionally set up your build environment
+```
+
+## Packaging For Other Operating Systems
+
+We are looking for people to create and maintain a `qmk` package for more operating systems. If you would like to create a package for your OS please follow these guidelines:
+
+* Follow best practices for your OS when they conflict with these guidelines
+    * Documment why in a comment when you do deviate
+* Install using a virtualenv
+* Instruct the user to set the environment variable `QMK_HOME` to have the firmware source checked out somewhere other than `~/qmk_firmware`.
+
+# Local CLI
+
+If you do not want to use the global CLI there is a local CLI bundled with `qmk_firmware`. You can find it in `qmk_firmware/bin/qmk`. You can run the `qmk` command from any directory and it will always operate on that copy of `qmk_firmware`.
+
+**Example**:
 
 ```
-export PATH=$PATH:$HOME/qmk_firmware/bin
+$ ~/qmk_firmware/bin/qmk hello
+Ψ Hello, World!
 ```
 
-You may want to add this to your `.profile`, `.bash_profile`, `.zsh_profile`, or other shell startup scripts.
+## Local CLI Limitations
 
-# Commands
+There are some limitations to the local CLI compared to the global CLI:
+
+* The local CLI does not support `qmk setup` or `qmk clone`
+* The local CLI always operates on the same `qmk_firmware` tree, even if you have multiple repositories cloned.
+* The local CLI does not run in a virtualenv, so it's possible that dependencies will conflict
+
+# CLI Commands
 
 ## `qmk compile`
 
@@ -46,3 +94,53 @@ This command formats C code using clang-format. Run it with no arguments to form
 ```
 qmk cformat [file1] [file2] [...] [fileN]
 ```
+
+## `qmk config`
+
+This command lets you configure the behavior of QMK. For the full `qmk config` documentation see [CLI Configuration](cli_configuration.md).
+
+**Usage**:
+
+```
+qmk config [-ro] [config_token1] [config_token2] [...] [config_tokenN]
+```
+
+## `qmk doctor`
+
+This command examines your environment and alerts you to potential build or flash problems.
+
+**Usage**:
+
+```
+qmk doctor
+```
+
+## `qmk new-keymap`
+
+This command creates a new keymap based on a keyboard's existing default keymap.
+
+**Usage**:
+
+```
+qmk new-keymap [-kb KEYBOARD] [-km KEYMAP]
+```
+
+## `qmk pyformat`
+
+This command formats python code in `qmk_firmware`.
+
+**Usage**:
+
+```
+qmk pyformat
+```
+
+## `qmk pytest`
+
+This command runs the python test suite. If you make changes to python code you should ensure this runs successfully.
+
+**Usage**:
+
+```
+qmk pytest
+```
diff --git a/docs/cli_configuration.md b/docs/cli_configuration.md
new file mode 100644
index 000000000..ad9ff291c
--- /dev/null
+++ b/docs/cli_configuration.md
@@ -0,0 +1,121 @@
+# QMK CLI Configuration
+
+This document explains how `qmk config` works.
+
+# Introduction
+
+Configuration for QMK CLI is a key/value system. Each key consists of a subcommand and an argument name separated by a period. This allows for a straightforward and direct translation between config keys and the arguments they set.
+
+## Simple Example
+
+As an example let's look at the command `qmk compile --keyboard clueboard/66/rev4 --keymap default`.
+
+There are two command line arguments that could be read from configuration instead:
+
+* `compile.keyboard`
+* `compile.keymap`
+
+Let's set these now:
+
+```
+$ qmk config compile.keyboard=clueboard/66/rev4 compile.keymap=default
+compile.keyboard: None -> clueboard/66/rev4
+compile.keymap: None -> default
+Ψ Wrote configuration to '/Users/example/Library/Application Support/qmk/qmk.ini'
+```
+
+Now I can run `qmk compile` without specifying my keyboard and keymap each time.
+
+## Setting User Defaults
+
+Sometimes you want to share a setting between multiple commands. For example, multiple commands take the argument `--keyboard`. Rather than setting this value for every command you can set a user value which will be used by any command that takes that argument.
+
+Example:
+
+```
+$ qmk config user.keyboard=clueboard/66/rev4 user.keymap=default
+user.keyboard: None -> clueboard/66/rev4
+user.keymap: None -> default
+Ψ Wrote configuration to '/Users/example/Library/Application Support/qmk/qmk.ini'
+```
+
+# CLI Documentation (`qmk config`)
+
+The `qmk config` command is used to interact with the underlying configuration. When run with no argument it shows the current configuration. When arguments are supplied they are assumed to be configuration tokens, which are strings containing no spaces with the following form:
+
+    <subcommand|general|default>[.<key>][=<value>]
+
+## Setting Configuration Values
+
+You can set configuration values by putting an equal sign (=) into your config key. The key must always be the full `<section>.<key>` form.
+
+Example:
+
+```
+$ qmk config default.keymap=default
+default.keymap: None -> default
+Ψ Wrote configuration to '/Users/example/Library/Application Support/qmk/qmk.ini'
+```
+
+## Reading Configuration Values
+
+You can read configuration values for the entire configuration, a single key, or for an entire section. You can also specify multiple keys to display more than one value.
+
+### Entire Configuration Example
+
+    qmk config
+
+### Whole Section Example
+
+    qmk config compile
+
+### Single Key Example
+
+    qmk config compile.keyboard
+
+### Multiple Keys Example
+
+    qmk config user compile.keyboard compile.keymap
+
+## Deleting Configuration Values
+
+You can delete a configuration value by setting it to the special string `None`.
+
+Example:
+
+```
+$ qmk config default.keymap=None
+default.keymap: default -> None
+Ψ Wrote configuration to '/Users/example/Library/Application Support/qmk/qmk.ini'
+```
+
+## Multiple Operations
+
+You can combine multiple read and write operations into a single command. They will be executed and displayed in order:
+
+```
+$ qmk config compile default.keymap=default compile.keymap=None
+compile.keymap=skully
+compile.keyboard=clueboard/66_hotswap/gen1
+default.keymap: None -> default
+compile.keymap: skully -> None
+Ψ Wrote configuration to '/Users/example/Library/Application Support/qmk/qmk.ini'
+```
+
+# User Configuration Options
+
+| Key | Default Value | Description |
+|-----|---------------|-------------|
+| user.keyboard | None | The keyboard path (Example: `clueboard/66/rev4`) |
+| user.keymap | None | The keymap name (Example: `default`) |
+| user.name | None | The user's github username. |
+
+# All Configuration Options
+
+| Key | Default Value | Description |
+|-----|---------------|-------------|
+| compile.keyboard | None | The keyboard path (Example: `clueboard/66/rev4`) |
+| compile.keymap | None | The keymap name (Example: `default`) |
+| hello.name | None | The name to greet when run. |
+| new_keyboard.keyboard | None | The keyboard path (Example: `clueboard/66/rev4`) |
+| new_keyboard.keymap | None | The keymap name (Example: `default`) |
diff --git a/docs/cli_development.md b/docs/cli_development.md
new file mode 100644
index 000000000..f5c7ad139
--- /dev/null
+++ b/docs/cli_development.md
@@ -0,0 +1,175 @@
+# QMK CLI Development
+
+This document has useful information for developers wishing to write new `qmk` subcommands.
+
+# Overview
+
+The QMK CLI operates using the subcommand pattern made famous by git. The main `qmk` script is simply there to setup the environment and pick the correct entrypoint to run. Each subcommand is a self-contained module with an entrypoint (decorated by `@cli.subcommand()`) that performs some action and returns a shell returncode, or None.
+
+# Subcommands
+
+[MILC](https://github.com/clueboard/milc) is the CLI framework `qmk` uses to handle argument parsing, configuration, logging, and many other features. It lets you focus on writing your tool without wasting your time writing glue code.
+
+Subcommands in the local CLI are always found in `qmk_firmware/lib/python/qmk/cli`.
+
+Let's start by looking at an example subcommand. This is `lib/python/qmk/cli/hello.py`:
+
+```python
+"""QMK Python Hello World
+
+This is an example QMK CLI script.
+"""
+from milc import cli
+
+
+@cli.argument('-n', '--name', default='World', help='Name to greet.')
+@cli.subcommand('QMK Hello World.')
+def hello(cli):
+    """Log a friendly greeting.
+    """
+    cli.log.info('Hello, %s!', cli.config.hello.name)
+```
+
+First we import the `cli` object from `milc`. This is how we interact with the user and control the script's behavior. We use `@cli.argument()` to define a command line flag, `--name`. This also creates a configuration variable named `hello.name` (and the corresponding `user.name`) which the user can set so they don't have to specify the argument. The `cli.subcommand()` decorator designates this function as a subcommand. The name of the subcommand will be taken from the name of the function.
+
+Once inside our function we find a typical "Hello, World!" program. We use `cli.log` to access the underlying [Logger Object](https://docs.python.org/3.5/library/logging.html#logger-objects), whose behavior is user controllable. We also access the value for name supplied by the user as `cli.config.hello.name`. The value for `cli.config.hello.name` will be determined by looking at the `--name` argument supplied by the user, if not provided it will use the value in the `qmk.ini` config file, and if neither of those is provided it will fall back to the default supplied in the `cli.argument()` decorator.
+
+# User Interaction
+
+MILC and the QMK CLI have several nice tools for interacting with the user. Using these standard tools will allow you to colorize your text for easier interactions, and allow the user to control when and how that information is displayed and stored.
+
+## Printing Text
+
+There are two main methods for outputting text in a subcommand- `cli.log` and `cli.echo()`. They operate in similar ways but you should prefer to use `cli.log.info()` for most general purpose printing.
+
+You can use special tokens to colorize your text, to make it easier to understand the output of your program. See [Colorizing Text](#colorizing-text) below.
+
+Both of these methods support built-in string formatting using python's [printf style string format operations](https://docs.python.org/3.5/library/stdtypes.html#old-string-formatting). You can use tokens such as `%s` and `%d` within your text strings then pass the values as arguments. See our Hello, World program above for an example.
+
+You should never use the format operator (`%`) directly, always pass values as arguments.
+
+### Logging (`cli.log`)
+
+The `cli.log` object gives you access to a [Logger Object](https://docs.python.org/3.5/library/logging.html#logger-objects). We have configured our log output to show the user a nice emoji for each log level (or the log level name if their terminal does not support unicode.) This way the user can tell at a glance which messages are most important when something goes wrong.
+
+The default log level is `INFO`. If the user runs `qmk -v <subcommand>` the default log level will be set to `DEBUG`.
+
+| Function | Emoji |
+|----------|-------|
+| cli.log.critical | `{bg_red}{fg_white}¬_¬{style_reset_all}` |
+| cli.log.error | `{fg_red}☒{style_reset_all}` |
+| cli.log.warning | `{fg_yellow}⚠{style_reset_all}` |
+| cli.log.info | `{fg_blue}Ψ{style_reset_all}` |
+| cli.log.debug | `{fg_cyan}☐{style_reset_all}` |
+| cli.log.notset | `{style_reset_all}¯\\_(o_o)_/¯` |
+
+### Printing (`cli.echo`)
+
+Sometimes you simply need to print text outside of the log system. This is appropriate if you are outputting fixed data or writing out something that should never be logged. Most of the time you should prefer `cli.log.info()` over `cli.echo`.
+
+### Colorizing Text
+
+You can colorize the output of your text by including color tokens within text. Use color to highlight, not to convey information. Remember that the user can disable color, and your subcommand should still be usable if they do.
+
+You should generally avoid setting the background color, unless it's integral to what you are doing. Remember that users have a lot of preferences when it comes to their terminal color, so you should pick colors that work well against both black and white backgrounds.
+
+Colors prefixed with 'fg' will affect the foreground (text) color. Colors prefixed with 'bg' will affect the background color.
+
+| Color | Background | Extended Background | Foreground | Extended Foreground|
+|-------|------------|---------------------|------------|--------------------|
+| Black | {bg_black} | {bg_lightblack_ex} | {fg_black} | {fg_lightblack_ex} |
+| Blue | {bg_blue} | {bg_lightblue_ex} | {fg_blue} | {fg_lightblue_ex} |
+| Cyan | {bg_cyan} | {bg_lightcyan_ex} | {fg_cyan} | {fg_lightcyan_ex} |
+| Green | {bg_green} | {bg_lightgreen_ex} | {fg_green} | {fg_lightgreen_ex} |
+| Magenta | {bg_magenta} | {bg_lightmagenta_ex} | {fg_magenta} | {fg_lightmagenta_ex} |
+| Red | {bg_red} | {bg_lightred_ex} | {fg_red} | {fg_lightred_ex} |
+| White | {bg_white} | {bg_lightwhite_ex} | {fg_white} | {fg_lightwhite_ex} |
+| Yellow | {bg_yellow} | {bg_lightyellow_ex} | {fg_yellow} | {fg_lightyellow_ex} |
+
+There are also control sequences that can be used to change the behavior of
+ANSI output:
+
+| Control Sequences | Description |
+|-------------------|-------------|
+| {style_bright} | Make the text brighter |
+| {style_dim} | Make the text dimmer |
+| {style_normal} | Make the text normal (neither `{style_bright}` nor `{style_dim}`) |
+| {style_reset_all} | Reset all text attributes to default. (This is automatically added to the end of every string.) |
+| {bg_reset} | Reset the background color to the user's default |
+| {fg_reset} | Reset the foreground color to the user's default |
+
+# Arguments and Configuration
+
+QMK handles the details of argument parsing and configuration for you. When you add a new argument it is automatically incorporated into the config tree based on your subcommand's name and the long name of the argument. You can access this configuration in `cli.config`, using either attribute-style access (`cli.config.<subcommand>.<argument>`) or dictionary-style access (`cli.config['<subcommand>']['<argument>']`).
+
+Under the hood QMK uses [ConfigParser](https://docs.python.org/3/library/configparser.html) to store configurations. This gives us an easy and straightforward way to represent the configuration in a human-editable way. We have wrapped access to this configuration to provide some nicities that ConfigParser does not normally have.
+
+## Reading Configuration Values
+
+You can interact with `cli.config` in all the ways you'd normally expect. For example the `qmk compile` command gets the keyboard name from `cli.config.compile.keyboard`. It does not need to know whether that value came from the command line, an environment variable, or the configuration file.
+
+Iteration is also supported:
+
+```
+for section in cli.config:
+    for key in cli.config[section]:
+        cli.log.info('%s.%s: %s', section, key, cli.config[section][key])
+```
+
+## Setting Configuration Values
+
+You can set configuration values in the usual ways.
+
+Dictionary style:
+
+```
+cli.config['<section>']['<key>'] = <value>
+```
+
+Attribute style:
+
+```
+cli.config.<section>.<key> = <value>
+```
+
+## Deleting Configuration Values
+
+You can delete configuration values in the usual ways.
+
+Dictionary style:
+
+```
+del(cli.config['<section>']['<key>'])
+```
+
+Attribute style:
+
+```
+del(cli.config.<section>.<key>)
+```
+
+## Writing The Configuration File
+
+The configuration is not written out when it is changed. Most commands do not need to do this. We prefer to have the user change their configuration deliberitely using `qmk config`.
+
+You can use `cli.save_config()` to write out the configuration.
+
+## Excluding Arguments From Configuration
+
+Some arguments should not be propagated to the configuration file. These can be excluded by adding `arg_only=True` when creating the argument.
+
+Example:
+
+```
+@cli.argument('-o', '--output', arg_only=True, help='File to write to')
+@cli.argument('filename', arg_only=True, help='Configurator JSON file')
+@cli.subcommand('Create a keymap.c from a QMK Configurator export.')
+def json_keymap(cli):
+    pass
+```
+
+You will only be able to access these arguments using `cli.args`. For example:
+
+```
+cli.log.info('Reading from %s and writing to %s', cli.args.filename, cli.args.output)
+```
diff --git a/docs/python_development.md b/docs/python_development.md
deleted file mode 100644
index b976a7c0e..000000000
--- a/docs/python_development.md
+++ /dev/null
@@ -1,45 +0,0 @@
-# Python Development in QMK
-
-This document gives an overview of how QMK has structured its python code. You should read this before working on any of the python code.
-
-## Script directories
-
-There are two places scripts live in QMK: `qmk_firmware/bin` and `qmk_firmware/util`. You should use `bin` for any python scripts that utilize the `qmk` wrapper. Scripts that are standalone and not run very often live in `util`.
-
-We discourage putting anything into `bin` that does not utilize the `qmk` wrapper. If you think you have a good reason for doing so please talk to us about your use case.
-
-## Python Modules
-
-Most of the QMK python modules can be found in `qmk_firmware/lib/python`. This is the path that we append to `sys.path`.
-
-We have a module hierarchy under that path:
-
-* `qmk_firmware/lib/python`
-    * `milc.py` - The CLI library we use. Will be pulled out into its own module in the future.
-    * `qmk` - Code associated with QMK
-        * `cli` - Modules that will be imported for CLI commands.
-        * `errors.py` - Errors that can be raised within QMK apps
-        * `keymap.py` - Functions for working with keymaps
-
-## CLI Scripts
-
-We have a CLI wrapper that you should utilize for any user facing scripts. We think it's pretty easy to use and it gives you a lot of nice things for free.
-
-To use the wrapper simply place a module into `qmk_firmware/lib/python/qmk/cli`, and create a symlink to `bin/qmk` named after your module. Dashes in command names will be converted into dots so you can use hierarchy to manage commands.
-
-When `qmk` is run it checks to see how it was invoked. If it was invoked as `qmk` the module name is take from `sys.argv[1]`. If it was invoked as `qmk-<module-name>` then everything after the first dash is taken as the module name. Dashes and underscores are converted to dots, and then `qmk.cli` is prepended before the module is imported.
-
-The module uses `@cli.entrypoint()` and `@cli.argument()` decorators to define an entrypoint, which is where execution starts.
-
-## Example CLI Script
-
-We have provided a QMK Hello World script you can use as an example. To run it simply run `qmk hello` or `qmk-hello`. The source code is listed below.
-
-```
-from milc import cli
-
-@cli.argument('-n', '--name', default='World', help='Name to greet.')
-@cli.entrypoint('QMK Python Hello World.')
-def main(cli):
-    cli.echo('Hello, %s!', cli.config.general.name)
-```
diff --git a/docs/redirects.json b/docs/redirects.json
index 814518f40..651148c2c 100644
--- a/docs/redirects.json
+++ b/docs/redirects.json
@@ -43,6 +43,10 @@
             {
                 "from": "unicode.html",
                 "to": "feature_unicode.html"
+            },
+            {
+                "from": "python_development.html",
+                "to": "cli_development.html"
             }
         ]
 }