Issue 256 Work on re-organising the Database Section of Framework

docs/core/architecture.md

@@ -57,7 +57,7 @@ The CiviCRM **data access objects** (DAOs) are PHP classes that
 ([e.g. `CRM/Pledge/DAO`](https://github.com/civicrm/civicrm-core/tree/master/CRM/Pledge/DAO))
 expose the contents
 of the database.  The release script generates each DAO automatically based
-on the matching XML file in the [data schema](/framework/schema-definition.md).  DAO objects tend to be instantiated in BAO classes.
+on the matching XML file in the [data schema](/framework/database/schema-definition.md).  DAO objects tend to be instantiated in BAO classes.
 
 The DAO classes all extend the core
 [DAO base class](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/DAO.php)

docs/extensions/civix.md

@@ -191,7 +191,7 @@ to create a new entity. For this, you can use the command `civix generate:entity
 
 1. Run `civix generate:entity MyEntity` (use CamelCase here). This creates a skeletal file for your XML schema, your BAO, and your API. It does NOT create a skeletal SQL file to create your table or DAO files at this time.
 
-1. Edit the [XML schema definitions](/framework/schema-definition.md) in the `xml` folder to define your desired fields.
+1. Edit the [XML schema definitions](/framework/database/schema-definition.md) in the `xml` folder to define your desired fields.
 
 1. Generate a [DAO file](/core/architecture.md#dao) and SQL to create your table. *(For now, civix does not handle this part. Hopefully these steps will become easier at some point in the future.)*
 

docs/framework/database/index.md

+# CiviCRM Database Information
+
+CiviCRM has a complex Database structure which this chapter will examine in more depth. CiviCRM for various reasons has the Schema struture of its database written out in XML files store in `xml/templates/schema` These files are used to define all Core CiviCRM tables that are installed when CiviCRM is installed. CiviCRM maintains a number of tools and scripts such as `GenCode`, `bin/setup.sh`, `bin/regen.sh`, the purpose of these tools is to update any current database with the lastest information stored in the XML files, Re-create the installation .mysql files that live in `sql/civicrm*.mysql` which are not commited to the Git Repo but found in the Donwloadable package. Also They update the "DAO" Files which contain a reference of what files and what relationships are there in the Datbase.
+
+## Tools
+<Insert documentation on how to use various tools>

docs/framework/database/schema-design.md

+## CiviCRM Schema Design
+
+## Why a Entity Relationship Diagram may *not* be helpful
+
+As some coders are visual learners, ER Diagrams maybe a waste of time for thsoe coders to understand the CivCRM Database modle and structure. Fundamentally CiviCRM's Schema diesgn is way too complex, with at least 150 tables and loads of Foriegn Key Constraints linking tables togeher any such graphic or design document that tries to map out the CiviCRM database design would most likely be more confusion than help, espeically for visual learners.
+
+## Aleternate Methods to ERDs to explore CiviCRM's Database Design
+
+Another method ofr viewing CiviCRM's Database model is to use one of various MySQL coding tools e.g. PHPMyAdmin or [MySql WOrkbench](https://www.mysql.com/products/workbench/). These tools allow coders to inspect the Table Structure more easily and look at all the constraints attached to tables very easily. 
+
+## MySQL Workbench Workflow
+
+1. Look at the list of tables in the database. The table name generally gives you a good idea of the type of data it stores.
+2. Pick one table and look closer. Let's look at `civicrm_country` (a relatively simple example) by opening the "Table inspector" for that table.
+  - We can look at the list of columns in the table.
+  ![Mysql Workbench CiviCRM Country Columns](/img/mysql_workbench_civicrm_country_tables.png)
+  We can see that columns have comments which explain (to some extent) the meaning of the data stored in the columns. From the Column Names and the comments we can start to get some idea of the relationship between tables. For example we can see that there is a column `address_format_id` which indicates to us that the table `civicrm_country` has a relationship to the table `civicrm_address_format`
+  - We can also Look at the Foriegn Keys that are relevant to this table
+  ![Mysql Workbench CiviCRM Country Foreign Keys](/img/mysql_workbench_civicrm_country_foreign_keys.png)
+  Here we can get more of a comprenshive picture of what tables `civicrm_country` relates to. We can tell that because we see not only what tables are referenced by `civicrm_country` but also the tables that reference `civicrm_country`. 
+
+## General Characteristics of CiviCRM Tables
+
+When we look at CiviCRM Tables there are a few patterns that generally hold true:
+- Every table has a id column which is Auto Increment and therefore unique key for that table
+- Columns which reference other tables generally speaking will be named in the format of `other table name` + `_id`. For example in `civicrm_country` there is `address_format_id` which indicates that is references `civicrm_address_format.id`
+- Many-to-many relationships use "join tables" as intermediary tables. For example, a contact can have many activity records, and an activity can have many contact records. So the table `civicrm_activity_contact` is used as the glue because it has foreign keys to both.
+- In some places CiviCRM defines schema using a construct called [pseudoconstants](framework/database/schema-definition/#table-field-pseudoconstant) which produces some slightly more complex logic
+  - Lots of columns reference `civicrm_option_values` when they just need a simple (and user-configurable) list of options. For example, look at `civicrm_contribution` which has a column called `payment_instrument_id`. You'll notice there's no table called `civicrm_payment_instrument`. So in this case the `payment_instrument_id` column actually references the value column in `civicrm_option_values` (but only for records in `civicrm_option_values` with the appropriate `option_group_id`.) Here there is no foreign key, so referential integrity is managed at the application layer, not the database layer.
+  - Some tables use "dynamic foreign keys". For example, look at `civicrm_note` which has columns `entity_id` and `entity_table`. This is because a note can be attached to different entities (e.g. contact, contribution, etc). So two columns are used to indicate what the note references. Here again, the application layer is responsible for ensuring referential integrity, so you won't find any foreign keys.

mkdocs.yml

@@ -67,8 +67,10 @@ pages:
   - Bootstrap Process: framework/bootstrap.md
   - Cache Reference: framework/cache.md
   # Components: /framework/components.md
-  # Database: /framework/database.md
-  - Schema definition: framework/schema-definition.md
+  - Database: 
+    - Overview: framework/database/index.md
+    - XML Schema definition: framework/database/schema-definition.md
+    - Schema Design: framework/database/schema-design.md
   - Resources Reference: framework/resources.md
   - Region Reference: framework/region.md
   - Upgrade Reference: framework/upgrade.md

redirects/internal.txt

@@ -5,3 +5,4 @@ best-practices/documentation-style-guide documentation/style-guide
 extensions/basics extensions
 api/general api
 hooks/hook_civicrm_trigger_info hooks/hook_civicrm_triggerInfo
+framework/schema-definition framework/database/schema-definition