QMK CLI and JSON keymap support (#6176)

* Script to generate keymap.c from JSON file.

* Support for keymap.json

* Add a warning about the keymap.c getting overwritten.

* Fix keymap generating

* Install the python deps

* Flesh out more of the python environment

* Remove defunct json2keymap

* Style everything with yapf

* Polish up python support

* Hide json keymap.c into the .build dir

* Polish up qmk-compile-json

* Make milc work with positional arguments

* Fix a couple small things

* Fix some errors and make the CLI more understandable

* Make the qmk wrapper more robust

* Add basic QMK Doctor

* Clean up docstrings and flesh them out as needed

* remove unused compile_firmware() function

6 files changed, 455 insertions, 55 deletions

diff --git a/docs/_summary.md b/docs/_summary.md
index 8a40ccd7f..611c283ac 100644
--- a/docs/_summary.md
+++ b/docs/_summary.md
@@ -8,6 +8,7 @@
 
 * [QMK Basics](README.md)
   * [QMK Introduction](getting_started_introduction.md)
+  * [QMK CLI](cli.md)
   * [Contributing to QMK](contributing.md)
   * [How to Use Github](getting_started_github.md)
   * [Getting Help](getting_started_getting_help.md)
@@ -34,6 +35,8 @@
   * [Keyboard Guidelines](hardware_keyboard_guidelines.md)
   * [Config Options](config_options.md)
   * [Keycodes](keycodes.md)
+  * [Coding Conventions - C](coding_conventions_c.md)
+  * [Coding Conventions - Python](coding_conventions_python.md)
   * [Documentation Best Practices](documentation_best_practices.md)
   * [Documentation Templates](documentation_templates.md)
   * [Glossary](reference_glossary.md)
@@ -41,6 +44,7 @@
   * [Useful Functions](ref_functions.md)
   * [Configurator Support](reference_configurator_support.md)
   * [info.json Format](reference_info_json.md)
+  * [Python Development](python_development.md)
 
 * [Features](features.md)
   * [Basic Keycodes](keycodes_basic.md)
diff --git a/docs/cli.md b/docs/cli.md
new file mode 100644
index 000000000..0365f2c9c
--- /dev/null
+++ b/docs/cli.md
@@ -0,0 +1,31 @@
+# QMK CLI
+
+This page describes how to setup and use the QMK CLI.
+
+# Overview
+
+The QMK CLI makes building and working with QMK keyboards easier. We have provided a number of commands to help you work with QMK:
+
+* `qmk compile-json`
+
+# Setup
+
+Simply add the `qmk_firmware/bin` directory to your `PATH`. You can run the `qmk` commands from any directory.
+
+```
+export PATH=$PATH:$HOME/qmk_firmware/bin
+```
+
+You may want to add this to your `.profile`, `.bash_profile`, `.zsh_profile`, or other shell startup scripts.
+
+# Commands
+
+## `qmk compile-json`
+
+This command allows you to compile JSON files you have downloaded from <https://config.qmk.fm>.
+
+**Usage**:
+
+```
+qmk compile-json mine.json
+```
diff --git a/docs/coding_conventions_c.md b/docs/coding_conventions_c.md
new file mode 100644
index 000000000..cbddedf8b
--- /dev/null
+++ b/docs/coding_conventions_c.md
@@ -0,0 +1,58 @@
+# Coding Conventions (C)
+
+Most of our style is pretty easy to pick up on, but right now it's not entirely consistent. You should match the style of the code surrounding your change, but if that code is inconsistent or unclear use the following guidelines:
+
+* We indent using four (4) spaces (soft tabs)
+* We use a modified One True Brace Style
+  * Opening Brace: At the end of the same line as the statement that opens the block
+  * Closing Brace: Lined up with the first character of the statement that opens the block
+  * Else If: Place the closing brace at the beginning of the line and the next opening brace at the end of the same line.
+  * Optional Braces: Always include optional braces.
+    * Good: if (condition) { return false; }
+    * Bad: if (condition) return false;
+* We encourage use of C style comments: `/* */`
+  * Think of them as a story describing the feature
+  * Use them liberally to explain why particular decisions were made.
+  * Do not write obvious comments
+  * If you not sure if a comment is obvious, go ahead and include it.
+* In general we don't wrap lines, they can be as long as needed. If you do choose to wrap lines please do not wrap any wider than 76 columns.
+* We use `#pragma once` at the start of header files rather than old-style include guards (`#ifndef THIS_FILE_H`, `#define THIS_FILE_H`, ..., `#endif`)
+* We accept both forms of preprocessor if's: `#ifdef DEFINED` and `#if defined(DEFINED)`
+  * If you are not sure which to prefer use the `#if defined(DEFINED)` form.
+  * Do not change existing code from one style to the other, except when moving to a multiple condition `#if`.
+  * Do not put whitespace between `#` and `if`.
+  * When deciding how (or if) to indent directives keep these points in mind:
+    * Readability is more important than consistency.
+    * Follow the file's existing style. If the file is mixed follow the style that makes sense for the section you are modifying.
+    * When choosing to indent you can follow the indention level of the surrounding C code, or preprocessor directives can have their own indent level. Choose the style that best communicates the intent of your code.
+
+Here is an example for easy reference:
+
+```c
+/* Enums for foo */
+enum foo_state {
+  FOO_BAR,
+  FOO_BAZ,
+};
+
+/* Returns a value */
+int foo(void) {
+  if (some_condition) {
+    return FOO_BAR;
+  } else {
+    return -1;
+  }
+}
+```
+
+# Auto-formatting with clang-format
+
+[Clang-format](https://clang.llvm.org/docs/ClangFormat.html) is part of LLVM and can automatically format your code for you, because ain't nobody got time to do it manually. We supply a configuration file for it that applies most of the coding conventions listed above. It will only change whitespace and newlines, so you will still have to remember to include optional braces yourself.
+
+Use the [full LLVM installer](http://llvm.org/builds/) to get clang-format on Windows, or use `sudo apt install clang-format` on Ubuntu.
+
+If you run it from the command-line, pass `-style=file` as an option and it will automatically find the .clang-format configuration file in the QMK root directory.
+
+If you use VSCode, the standard C/C++ plugin supports clang-format, alternatively there is a [separate extension](https://marketplace.visualstudio.com/items?itemName=LLVMExtensions.ClangFormat) for it.
+
+Some things (like LAYOUT macros) are destroyed by clang-format, so either don't run it on those files, or wrap the sensitive code in `// clang-format off` and `// clang-format on`.
diff --git a/docs/coding_conventions_python.md b/docs/coding_conventions_python.md
new file mode 100644
index 000000000..c7743050e
--- /dev/null
+++ b/docs/coding_conventions_python.md
@@ -0,0 +1,314 @@
+# Coding Conventions (Python)
+
+Most of our style follows PEP8 with some local modifications to make things less nit-picky. 
+
+* We target Python 3.5 for compatability with all supported platforms.
+* We indent using four (4) spaces (soft tabs)
+* We encourage liberal use of comments
+  * Think of them as a story describing the feature
+  * Use them liberally to explain why particular decisions were made.
+  * Do not write obvious comments
+  * If you not sure if a comment is obvious, go ahead and include it.
+* We require useful docstrings for all functions.
+* In general we don't wrap lines, they can be as long as needed. If you do choose to wrap lines please do not wrap any wider than 76 columns.
+* Some of our practices conflict with the wider python community to make our codebase more approachable to non-pythonistas.
+
+# YAPF
+
+You can use [yapf](https://github.com/google/yapf) to style your code. We provide a config in [setup.cfg](setup.cfg).
+
+# Imports
+
+We don't have a hard and fast rule for when to use `import ...` vs `from ... import ...`. Understandability and maintainability is our ultimate goal.
+
+Generally we prefer to import specific function and class names from a module to keep code shorter and easier to understand. Sometimes this results in a name that is ambiguous, and in such cases we prefer to import the module instead. You should avoid using the "as" keyword when importing, unless you are importing a compatability module.
+
+Imports should be one line per module. We group import statements together using the standard python rules- system, 3rd party, local.
+
+Do not use `from foo import *`. Supply a list of objects you want to import instead, or import the whole module.
+
+## Import Examples
+
+Good:
+
+```
+from qmk import effects
+
+effects.echo()
+```
+
+Bad:
+
+```
+from qmk.effects import echo
+
+echo()  # It's unclear where echo comes from
+```
+
+Good:
+
+```
+from qmk.keymap import compile_firmware
+
+compile_firmware()
+```
+
+OK, but the above is better:
+
+```
+import qmk.keymap
+
+qmk.keymap.compile_firmware()
+```
+
+# Statements
+
+One statement per line.
+
+Even when allowed (EG `if foo: bar`) we do not combine 2 statements onto a single line.
+
+# Naming
+
+`module_name`, `package_name`, `ClassName`, `method_name`, `ExceptionName`, `function_name`, `GLOBAL_CONSTANT_NAME`, `global_var_name`, `instance_var_name`, `function_parameter_name`, `local_var_name`.
+
+Function names, variable names, and filenames should be descriptive; eschew abbreviation. In particular, do not use abbreviations that are ambiguous or unfamiliar to readers outside your project, and do not abbreviate by deleting letters within a word.
+
+Always use a .py filename extension. Never use dashes.
+
+## Names to Avoid
+
+* single character names except for counters or iterators. You may use "e" as an exception identifier in try/except statements.
+* dashes (-) in any package/module name
+* __double_leading_and_trailing_underscore__ names (reserved by Python)
+
+# Docstrings
+
+To maintain consistency with our docstrings we've set out the following guidelines.
+
+* Use markdown formatting
+* Always use triple-dquote docstrings with at least one linebreak: `"""\n"""`
+* First line is a short (< 70 char) description of what the function does
+* If you need more in your docstring leave a blank line between the description and the rest.
+* Start indented lines at the same indent level as the opening triple-dquote
+* Document all function arguments using the format described below
+* If present, Args:, Returns:, and Raises: should be the last three things in the docstring, separated by a blank line each.
+
+## Simple docstring example
+
+```
+def my_awesome_function():
+    """Return the number of seconds since 1970 Jan 1 00:00 UTC.
+    """
+    return int(time.time())
+```
+
+## Complex docstring example
+
+```
+def my_awesome_function():
+    """Return the number of seconds since 1970 Jan 1 00:00 UTC.
+
+    This function always returns an integer number of seconds.
+    """
+    return int(time.time())
+```
+
+## Function arguments docstring example
+
+```
+def my_awesome_function(start=None, offset=0):
+    """Return the number of seconds since 1970 Jan 1 00:00 UTC.
+
+    This function always returns an integer number of seconds.
+
+
+    Args:
+        start
+            The time to start at instead of 1970 Jan 1 00:00 UTC
+
+        offset
+            Return an answer that has this number of seconds subtracted first
+
+    Returns:
+        An integer describing a number of seconds.
+
+    Raises:
+        ValueError
+            When `start` or `offset` are not positive numbers
+    """
+    if start < 0 or offset < 0:
+        raise ValueError('start and offset must be positive numbers.')
+
+    if not start:
+        start = time.time()
+
+    return int(start - offset)
+```
+
+# Exceptions
+
+Exceptions are used to handle exceptional situations. They should not be used for flow control. This is a break from the python norm of "ask for forgiveness." If you are catching an exception it should be to handle a situation that is unusual.
+
+If you use a catch-all exception for any reason you must log the exception and stacktrace using cli.log.
+
+Make your try/except blocks as short as possible. If you need a lot of try statements you may need to restructure your code.
+
+# Tuples
+
+When defining one-item tuples always include a trailing comma so that it is obvious you are using a tuple. Do not rely on implicit one-item tuple unpacking. Better still use a list which is unambiguous.
+
+This is particularly important when using the printf-style format strings that are commonly used.
+
+# Lists and Dictionaries
+
+We have configured YAPF to differentiate between sequence styles with a trailing comma. When a trailing comma is omitted YAPF will format the sequence as a single line. When a trailing comma is included YAPF will format the sequence with one item per line.
+
+You should generally prefer to keep short definition on a single line. Break out to multiple lines sooner rather than later to aid readability and maintainability.
+
+# Parentheses
+
+Avoid excessive parentheses, but do use parentheses to make code easier to understand. Do not use them in return statements unless you are explicitly returning a tuple, or it is part of a math expression.
+
+# Format Strings
+
+We generally prefer printf-style format strings. Example:
+
+```
+name = 'World'
+print('Hello, %s!' % (name,))
+```
+
+This style is used by the logging module, which we make use of extensively, and we have adopted it in other places for consistency. It is also more familiar to C programmers, who are a big part of our casual audience.
+
+Our included CLI module has support for using these without using the percent (%) operator. Look at `cli.echo()` and the various `cli.log` functions (EG, `cli.log.info()`) for more details.
+
+# Comprehensions & Generator Expressions
+
+We encourage the liberal use of comprehensions and generators, but do not let them get too complex. If you need complexity fall back to a for loop that is easier to understand.
+
+# Lambdas
+
+OK to use but probably should be avoided. With comprehensions and generators the need for lambdas is not as strong as it once was.
+
+# Conditional Expressions
+
+OK in variable assignment, but otherwise should be avoided.
+
+Conditional expressions are if statements that are in line with code. For example:
+
+```
+x = 1 if cond else 2
+```
+
+It's generally not a good idea to use these as function arguments, sequence items, etc. It's too easy to overlook.
+
+# Default Argument Values
+
+Encouraged, but values must be immutable objects.
+
+When specifying default values in argument lists always be careful to specify objects that can't be modified in place. If you use a mutable object the changes you make will persist between calls, which is usually not what you want. Even if that is what you intend to do it is confusing for others and will hinder understanding.
+
+Bad:
+
+```
+def my_func(foo={}):
+    pass
+```
+
+Good:
+
+```
+def my_func(foo=None):
+    if not foo:
+        foo = {}
+```
+
+# Properties
+
+Always use properties instead of getter and setter functions.
+
+```
+class Foo(object):
+    def __init__(self):
+        self._bar = None
+
+    @property
+    def bar(self):
+        return self._bar
+
+    @bar.setter
+    def bar(self, bar):
+        self._bar = bar
+```
+
+# True/False Evaluations
+
+You should generally prefer the implicit True/False evaluation in if statements, rather than checking equivalency.
+
+Bad:
+
+```
+if foo == True:
+    pass
+
+if bar == False:
+    pass
+```
+
+Good:
+
+```
+if foo:
+    pass
+
+if not bar:
+    pass
+```
+
+# Decorators
+
+Use when appropriate. Try to avoid too much magic unless it helps with understanding.
+
+# Threading and Multiprocessing
+
+Should be avoided. If you need this you will have to make a strong case before we merge your code.
+
+# Power Features
+
+Python is an extremely flexible language and gives you many fancy features such as custom metaclasses, access to bytecode, on-the-fly compilation, dynamic inheritance, object reparenting, import hacks, reflection, modification of system internals, etc.
+
+Don't use these.
+
+Performance is not a critical concern for us, and code understandability is. We want our codebase to be approachable by someone who only has a day or two to play with it. These features generally come with a cost to easy understanding, and we would prefer to have code that can be readily understood over faster or more compact code.
+
+Note that some standard library modules use these techniques and it is ok to make use of those modules. But please keep readability and understandability in mind when using them.
+
+# Type Annotated Code
+
+For now we are not using any type annotation system, and would prefer that code remain unannotated. We may revisit this in the future.
+
+# Function length
+
+Prefer small and focused functions.
+
+We recognize that long functions are sometimes appropriate, so no hard limit is placed on function length. If a function exceeds about 40 lines, think about whether it can be broken up without harming the structure of the program.
+
+Even if your long function works perfectly now, someone modifying it in a few months may add new behavior. This could result in bugs that are hard to find. Keeping your functions short and simple makes it easier for other people to read and modify your code.
+
+You could find long and complicated functions when working with some code. Do not be intimidated by modifying existing code: if working with such a function proves to be difficult, you find that errors are hard to debug, or you want to use a piece of it in several different contexts, consider breaking up the function into smaller and more manageable pieces.
+
+# FIXMEs
+
+It is OK to leave FIXMEs in code. Why? Encouraging people to at least document parts of code that need to be thought out more (or that are confusing) is better than leaving this code undocumented.
+
+All FIXMEs should be formatted like:
+
+```
+FIXME(username): Revisit this code when the frob feature is done.
+```
+
+...where username is your GitHub username.
+
+# Unit Tests
+
+These are good. We should have some one day.
diff --git a/docs/contributing.md b/docs/contributing.md
index 7d1a9691c..761bc9959 100644
--- a/docs/contributing.md
+++ b/docs/contributing.md
@@ -54,62 +54,10 @@ Never made an open source contribution before? Wondering how contributions work
 
 # Coding Conventions
 
-Most of our style is pretty easy to pick up on, but right now it's not entirely consistent. You should match the style of the code surrounding your change, but if that code is inconsistent or unclear use the following guidelines:
-
-* We indent using four (4) spaces (soft tabs)
-* We use a modified One True Brace Style
-  * Opening Brace: At the end of the same line as the statement that opens the block
-  * Closing Brace: Lined up with the first character of the statement that opens the block
-  * Else If: Place the closing brace at the beginning of the line and the next opening brace at the end of the same line.
-  * Optional Braces: Always include optional braces.
-    * Good: if (condition) { return false; }
-    * Bad: if (condition) return false;
-* We encourage use of C style comments: `/* */`
-  * Think of them as a story describing the feature
-  * Use them liberally to explain why particular decisions were made.
-  * Do not write obvious comments
-  * If you not sure if a comment is obvious, go ahead and include it.
-* In general we don't wrap lines, they can be as long as needed. If you do choose to wrap lines please do not wrap any wider than 76 columns.
-* We use `#pragma once` at the start of header files rather than old-style include guards (`#ifndef THIS_FILE_H`, `#define THIS_FILE_H`, ..., `#endif`)
-* We accept both forms of preprocessor if's: `#ifdef DEFINED` and `#if defined(DEFINED)`
-  * If you are not sure which to prefer use the `#if defined(DEFINED)` form.
-  * Do not change existing code from one style to the other, except when moving to a multiple condition `#if`.
-  * Do not put whitespace between `#` and `if`.
-  * When deciding how (or if) to indent directives keep these points in mind:
-    * Readability is more important than consistency.
-    * Follow the file's existing style. If the file is mixed follow the style that makes sense for the section you are modifying.
-    * When choosing to indent you can follow the indention level of the surrounding C code, or preprocessor directives can have their own indent level. Choose the style that best communicates the intent of your code.
-
-Here is an example for easy reference:
+Most of our style is pretty easy to pick up on. If you are familiar with either C or Python you should not have too much trouble with our local styles.
 
-```c
-/* Enums for foo */
-enum foo_state {
-  FOO_BAR,
-  FOO_BAZ,
-};
-
-/* Returns a value */
-int foo(void) {
-  if (some_condition) {
-    return FOO_BAR;
-  } else {
-    return -1;
-  }
-}
-```
-
-# Auto-formatting with clang-format
-
-[Clang-format](https://clang.llvm.org/docs/ClangFormat.html) is part of LLVM and can automatically format your code for you, because ain't nobody got time to do it manually. We supply a configuration file for it that applies most of the coding conventions listed above. It will only change whitespace and newlines, so you will still have to remember to include optional braces yourself.
-
-Use the [full LLVM installer](http://llvm.org/builds/) to get clang-format on Windows, or use `sudo apt install clang-format` on Ubuntu.
-
-If you run it from the command-line, pass `-style=file` as an option and it will automatically find the .clang-format configuration file in the QMK root directory.
-
-If you use VSCode, the standard C/C++ plugin supports clang-format, alternatively there is a [separate extension](https://marketplace.visualstudio.com/items?itemName=LLVMExtensions.ClangFormat) for it.
-
-Some things (like LAYOUT macros) are destroyed by clang-format, so either don't run it on those files, or wrap the sensitive code in `// clang-format off` and `// clang-format on`.
+* [Coding Conventions - C](coding_conventions_c.md)
+* [Coding Conventions - Python](coding_conventions_python.md)
 
 # General Guidelines
 
diff --git a/docs/python_development.md b/docs/python_development.md
new file mode 100644
index 000000000..b976a7c0e
--- /dev/null
+++ b/docs/python_development.md
@@ -0,0 +1,45 @@
+# Python Development in QMK
+
+This document gives an overview of how QMK has structured its python code. You should read this before working on any of the python code.
+
+## Script directories
+
+There are two places scripts live in QMK: `qmk_firmware/bin` and `qmk_firmware/util`. You should use `bin` for any python scripts that utilize the `qmk` wrapper. Scripts that are standalone and not run very often live in `util`.
+
+We discourage putting anything into `bin` that does not utilize the `qmk` wrapper. If you think you have a good reason for doing so please talk to us about your use case.
+
+## Python Modules
+
+Most of the QMK python modules can be found in `qmk_firmware/lib/python`. This is the path that we append to `sys.path`.
+
+We have a module hierarchy under that path:
+
+* `qmk_firmware/lib/python`
+    * `milc.py` - The CLI library we use. Will be pulled out into its own module in the future.
+    * `qmk` - Code associated with QMK
+        * `cli` - Modules that will be imported for CLI commands.
+        * `errors.py` - Errors that can be raised within QMK apps
+        * `keymap.py` - Functions for working with keymaps
+
+## CLI Scripts
+
+We have a CLI wrapper that you should utilize for any user facing scripts. We think it's pretty easy to use and it gives you a lot of nice things for free.
+
+To use the wrapper simply place a module into `qmk_firmware/lib/python/qmk/cli`, and create a symlink to `bin/qmk` named after your module. Dashes in command names will be converted into dots so you can use hierarchy to manage commands.
+
+When `qmk` is run it checks to see how it was invoked. If it was invoked as `qmk` the module name is take from `sys.argv[1]`. If it was invoked as `qmk-<module-name>` then everything after the first dash is taken as the module name. Dashes and underscores are converted to dots, and then `qmk.cli` is prepended before the module is imported.
+
+The module uses `@cli.entrypoint()` and `@cli.argument()` decorators to define an entrypoint, which is where execution starts.
+
+## Example CLI Script
+
+We have provided a QMK Hello World script you can use as an example. To run it simply run `qmk hello` or `qmk-hello`. The source code is listed below.
+
+```
+from milc import cli
+
+@cli.argument('-n', '--name', default='World', help='Name to greet.')
+@cli.entrypoint('QMK Python Hello World.')
+def main(cli):
+    cli.echo('Hello, %s!', cli.config.general.name)
+```