namaste ======= This is the namaste repository - a backend, AMQP-based, event-driven framework written in PHP 7.1. Creation Date: Jun 7, 2017 Last Updated: Oct 20, 2020 Author: mike@givingassistant.org Reference Link for MD Syntax: * https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md Assumptions ----------- * you've already installed git and configured the git globals * you've pulled the source code from the bit-bucket repository * you're installing on Ubuntu 18.04 LTS or a cloud-compatible release version Installation Notes: ------------------- Once you've pulled source code, and upgraded to PHP 7.1 (below), you'll need to make a few changes to the namaste environment. From the root directory in the Namaste source, enter the following commands: * mkdir logs pids * chmod 1777 logs pids Some commands to run for setting up the Namaste environment: * Install Xdebugger 2.7.2 for PHP7.0: https://xdebug.org/wizard.php * do not forget to also configure this in PHPStorm when creating the repos! * cd into the ./lib directory to install composer: * curl -sS https://getcomposer.org/installer -o composer-setup.php * validate the composer download: * php -r "if (hash_file('SHA384', 'composer-setup.php') === '669656bab3166a7aff8a7506b8cb2d1c292f042046c5a994c43155c0be6190fa0355160742ab2e1c88d40d5be660b410') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" * Output: "Installer verified" * Install composer: * sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer * on completion, composer is available globally.. cd into the ./lib directory to install packages needed for the project. * move the composer-setup.php file into the ./lib directory * run: php composer-setup-php -- this will create the composer.phar file in ./lib * visit: https://packagist.org/packages/theseer/phpdox to ensure that the packages listed are version compatible and at the lates revs * php composer.phar self-update -- updates composer package to the latest version. * Note: composer.phar is NOT checked-in to source -- it is _your_ responsibility to keep it current! * php composer.phar update -- installs libraries and dependencies into the lib directory * Note: the vendor sub-directory, where composer stores it's files, is not checked into source. Again, it is _your_ responsibility to ensure it is kept current. * on completion, you should remove composer-setup.php * Oracle Java 8 * Install the Oracle Java8 SDK (required for DynamoDB) * instructions: http://tipsonubuntu.com/2016/07/31/install-oracle-java-8-9-ubuntu-16-04-linux-mint-18/ * Download and install AWS DynamoDB * instructions: https://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/GettingStarted.Download.html * note: the Ddb sdk is installed via composer * Install memcached: * apt install memcached php-memcached PHP 7.4 ------- Don't forget to enable the Apache mod for PHP 7.4! Here's the current list of PHP 7.4 packages I currently have installed: ``` mshallop@clydesdale:~$ apt list "*php*" | grep -i installed WARNING: apt does not have a stable CLI interface. Use with caution in scripts. libapache2-mod-php7.2/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] libapache2-mod-php7.4/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic] php-apcu/bionic,now 5.1.18+4.0.11-1+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php-apcu-bc/bionic,now 1.0.5-1+ubuntu18.04.1+deb.sury.org+20191129 amd64 [installed,automatic] php-common/bionic,bionic,now 2:72+ubuntu18.04.1+deb.sury.org+1 all [installed,automatic] php-igbinary/bionic,now 3.1.0+2.0.8-2+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic] php-memcached/bionic,now 3.1.4+2.2.0-1+ubuntu18.04.1+deb.sury.org+20191129 amd64 [installed] php-mongodb/bionic,now 1.6.1-4+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php-msgpack/bionic,now 2.0.3+0.5.7-2+ubuntu18.04.1+deb.sury.org+20191129 amd64 [installed,automatic] php-pear/bionic,bionic,now 1:1.10.8+submodules+notgz-1+ubuntu18.04.1+deb.sury.org+1 all [installed] php-php-gettext/bionic,bionic,now 1.0.12-0.1 all [installed,automatic] php-phpseclib/bionic,bionic,now 2.0.9-1 all [installed,automatic] php-tcpdf/bionic,bionic,now 6.2.13+dfsg-1ubuntu1 all [installed,automatic] php7.2/bionic,bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 all [installed] php7.2-bcmath/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-bz2/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic] php7.2-cli/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-common/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-curl/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-gd/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-gmp/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic] php7.2-imap/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-json/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-mbstring/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-mysql/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-opcache/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-readline/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-xml/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.2-zip/bionic,now 7.2.27-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic] php7.4/bionic,bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 all [installed] php7.4-bcmath/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-bz2/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-cli/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic] php7.4-common/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-curl/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-dev/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-gd/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-gmp/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-intl/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-json/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic] php7.4-mbstring/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-mysql/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-opcache/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic] php7.4-readline/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed,automatic] php7.4-xml/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] php7.4-zip/bionic,now 7.4.2-6+ubuntu18.04.1+deb.sury.org+1 amd64 [installed] phpmyadmin/bionic,bionic,now 4:4.6.6-5 all [installed] pkg-php-tools/bionic,bionic,now 1.35ubuntu1 all [installed,automatic] ``` To install these tools in a single command: ```$ apt install php-memcached php-mongodb php-pear php7.4-bcmath php7.4-bz2 php7.4-curl php7.4-imap uw-mailutils php7.4-mbstring php7.4-mysql php-apcu php7.4-gd libgd-tools php7.4-xml php7.4-zip php7.4-dev php7.4-intl phpmyadmin``` Finally, install the pecl libraries for mongoDB and the XDebugger: ```$ pecl install mongodb xdebug``` RabbitMQ -------- Install RabbitMQ: RabbitMQ can fail to install based on your current OS version. As of this writing, we're using the following versions: * RabbitMQ 3.7.7 * Erlang 20.2.2 You can go to this site: https://www.rabbitmq.com/download.html and choose the appropriate package to download (Linux/Debian for Ubuntu) and then use your software installer to install the deb package. This will install the compatible versions of both RabbitMQ and Erlang. Or, you can install manually... ```text $ apt install rabbitmq-server ``` Once installed, next, install the RMQ management console: ```text $ rabbitmq-plugins enable rabbitmq_management ``` Test/confirm that you can access the console via: http://localhost:15672 Generally speaking, I've found user-management via the gui to be pretty janky - you'll be better off using the command line: ```text $ rabbitmqctl add_user namaste 4YyrWuKH ``` This creates the namaste user and sets the password. ```text $ rabbitmqctl set_user_tags namaste administrator ``` This sets the namaste account to administrator level. ```text $ rabbitmqctl add_vhost mdev ``` Adds the vhost "mdev" to the configuration. (ymmv) ```text $ rabbitmqctl set_permissions -p mdev namaste ".*" ".*" ".*" ``` This command grants the namaste user full access to the mdev vhost Helpful Links: * https://www.rabbitmq.com/man/rabbitmqctl.1.man.html * https://github.com/rabbitmq/rabbitmq-server/blob/stable/docs/rabbitmq.config.example Upgrade Erlang -------------- **You do not need to do this step if you downloaded and installed from the deb package!** The default version of Erlang, installed with RabbitMQ, under Ubuntu, is sub-optimal for node.js clients. If you're using node.js to connect to RabbitMQ, you should be at Erlang v. 19.x or later. To upgrade Erlang, follow these steps: * lsb_release -c returns the codename of your installation -- for Ubuntu 16.04 or later, this should come back as "xenial". Add the following line to /etc/apt/sources.list: * deb http://packages.erlang-solutions.com/ubuntu xenial trusty contrib (Change "xenial" to the appropriate codename) Execute the following commands: * wget http://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc * sudo apt-key add erlang_solutions.asc Then, update the system: * apt update * apt upgrade * apt dist-upgrade Restart RabbitMQ: * service rabbitmq-server restart And check the console page for the server - the upper-right corner shows the versions of RabbitMQ and Erlang. Your version of Erlang should now be at 19.x or later. (As of this writing, the Erlang version is 20.0 for Ubuntu 16.04) php.ini tweaks: --------------- For most resources, (mongoDB, RabbitMQ, etc.), I like to re-route the logging out put to a single repository. Normally, I use /home/logs however ymmv. ``` error_log = /home/logs/php_errors.log ``` Local Variables: ``` [xdebug] xdebug.remote=1 xdebug.remote_port=9000 xdebug.remote_autostart=0 xdebug.var_display_max_depth=10 xdebug.cli_color=1 ``` mongodb ------- For mongo, you will need to install the pecl libraries containing the mongodb drivers for PHP 7.1. ``` pecl install mongodb ``` Add the following line to the file: /etc/php/7.1/cli/php.ini ``` [mongodb] extension=mongodb.so ``` For Apache2 support, you will need to install the mongodb drivers and restart Apache: ```bash apt install php-mongodb ... systemctl restart apache2 ``` Apache2 support is necessary for the Namaste browser utilities. **User RBAC (Role Based Access Controls) ** You should create a *minimum* of two user account for Namaste mongo when using version 3.6 or later. The first user should be a root user. ```text > use admin > db.createUser({ user: "myRootUser", pwd: "myRootPassword", roles : [ { role : "root", db: "admin" } ] } ) ``` Once the user is created, then you can test your authentication (while still using the admin db): ```text > db.auth("myRootUser", "myRootPassword") 1 ``` A value of 1 will be returned if the information is correct. If not, a 0 is returned. Next, create the user accounts. As of this writing, there are two Namaste mongo collections, one on the admin service, and one in Namaste proper. These are discrete databases and you have the option/ability to create different users for both. Both are defined in the XML configuration ```xml gaOwner zPd7B6^Y 1 givva_namaste ``` Note that the authSource tag uses the name of the Namaste database sans the environment. (Defined in id -> envName) This will be prepended to the database name prior to making a resource connection request. To create users for the Namaste and Admin databases: ```text > use development_givva_namaste > db.createUser( { user : "gaOwner", pwd: "zPd7B6^Y", roles: [ { role : "dbOwner", db: "development_givva_namaste" } ] } ) > use development_givva_namaste_admin > db.createUser( { user : "gaOwner", pwd: "zPd7B6^Y", roles: [ { role : "dbOwner", db: "development_givva_namaste_admin" } ] } ) ``` MariaDB ------- As root: ``` apt install mariadb-server mariadb-client mysql_secure_installation ``` Running mysql_secure_installation will prompt you for the root password - if this is a new, fresh, install, you should just press enter. Create the gaAdmin account -- if you change the name of the account user, make sure you update the env.xml configuration file also! ```mysql CREATE USER 'gaAdmin'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON *.* TO 'gaAdmin'@'%'; FLUSH PRIVILEGES; ``` The database name is derived from the XML file setting under application --> id --> envName which is prepended to the string "givva". In order to execute Namaste db creation scripts, the gaAdmin user account must have all privileges, globally, minus the GRANT option. Once the gaAdmin (or whatever you've named your user) has been created in mySQL, create the databases: ```mysql CREATE DATABASE development_givva_namaste; CREATE DATABASE development_givva_namaste_warehouse; ``` Note that your database name may be different depending on the environment to which you're deploying Namaste. The supported environments are [development | qa | staging | production]. This naming conventions allows us to run databases for different, non-production, environments on the same database instance. **Don't forget to run the ./utilities/mysqlConfig.php program to create your tables, views, and database-objects!** ### Tips... If the database fails to start, and you've changed the location of your datafiles to the */home* directory, check status using the following command: ```text $ systemctl status mariadb ``` And, within the ouput of that you command, you see the following: ```text ... [Warning] Can't create test file /home/data/mysql/master.lower-test ``` Then you'll need to edit the file located in: */lib/systemd/system/mariadb.service* and change the following line: ```text ProtectHome=true ``` to this: ```text ProtectHome=false ``` Next, reload the system configuration files with this command: ```text $ systemctl daemon-reload $ systemctl restart mariadb ``` This will allow you to start-up mariadb using the */home* directory for data storage. mySQL Timeouts -------------- When there is no broker activity requiring interaction with mySQL as a service, the connection in Namaste to the mysql resource may eventually time-out. When this happens, you will need to restart the Namaste brokers (framework). The timeout interval can be mitigated by increasing the timeout value in mySQL/MariaDB to a longer interval value. You can increase the timeout value by adding the following lines to the mariaDB configuration file: ```bash # increase timeout values for broker connections wait_timeout = 86400 interactive_timeout = 86400 ``` Note that you will have to make these changes for every node in the mariaDB cluster. Make sure you restart the mariaDB service in order for these changes to take effect. Apache Virtual Host for Namaste: -------------------------------- You should edit and add this v-host configuration to your Apache configuration (/etc/apache2/sites-available/namaste.conf) so that you can access Namaste tools and utilities over HTTP. ``` # Namaste Virtual Host Configuration ServerName namaste ServerAlias namaste ServerAdmin mike@givingassistant.org DocumentRoot /home/mshallop/code/php/namaste DirectoryIndex index.php Options Indexes FollowSymLinks MultiViews AllowOverride All Satisfy Any Require all granted LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %b" common LogFormat "%{Referer}i -> %U" referer LogFormat "%{User-agent}i" agent CustomLog /home/logs/namaste_log common ErrorLog /home/logs/namaste_error.log CustomLog /home/logs/namaste_access.log combined # Possible values include: debug, info, notice, warn, error, crit, # alert, emerg. LogLevel warn ServerSignature Off ``` Local Tools ----------- Namaste comes with a couple of brilliantly-written apps for debugging. When you've installed and aliased Namaste locally as a HTTP virtual host, you can access these tools from your host alias (in this case: namaste): * cachepeak: memcache reader * http://namaste/utilities/cashpeak.php * dumper: reads namaste log and metrics log * http://namaste/utilities/dumper.php PHPUnit Testing --------------- If you're using PHPStorm, you will want to make a configuration change for PHP Unit testing. Under "Languages and Frameworks" --> PHP --> Test Frameworks Select the "Path to phpunit.phar" option and download phpunit.phar -- for my installation, I installed the phar file into: ./lib/phpunit *(not checked-in to git)* And was able to successfully run unit tests thereafter. Other Recommended Tools and Packages: ------------------------------------- GitKraken - The legendary GIT GUI client for all platforms: * https://www.gitkraken.com/ Robomongo - (Linux) Mongo GUI Tool: * https://robomongo.org/download Adminer -- Database management in a single PHP File: * https://www.adminer.org/ Version 2.0 Features ==================== DynamoDB -------- For this framework, we're using DynamoDB - an AWS nosql storage schema. Review the namaste-readme.txt file in the Ddb directory for help on connecting after you've read the online documentation on AWS. Key Points: * DynamoDB is a NoSQL database, and is schemaless, which means that, other than the primary key attributes, you do not need to define any attributes or data types at table creation time. * Your applications must encode binary values in base64-encoded format before sending them to DynamoDB. Upon receipt of these values, DynamoDB decodes the data into an unsigned byte array and uses that as the length of the binary attribute. * When your application writes data to a DynamoDB table and receives an HTTP 200 response (OK), all copies of the data are updated. The data will eventually be consistent across all storage locations, usually within one second or less. DynamoDB uses eventually consistent reads, unless you specify otherwise. * When you create a table, you specify how much provisioned throughput capacity you want to reserve for reads and writes. DynamoDB will reserve the necessary resources to meet your throughput needs while ensuring consistent, low-latency performance. You can also change your provisioned throughput settings, increasing or decreasing capacity as needed. * DynamoDB is **not** supported under the AWS Resource API - meaning that you cannot use the object-oriented abstraction layer for coding. Helpful Links: * https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.API.html * https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html * https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/BestPractices.html * https://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/GettingStarted.PHP.01.html * https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html * https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html * https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html#Query.Pagination * https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html Command Line Examples: If you want to run aws commands from the command line, you should first run the command: * aws configure and follow the prompts to save your key-id and access-key to a local .ini file. list existing tables: * aws dynamodb list-tables --endpoint-url http://localhost:8000 delete an existing table: * aws dynamodb delete-table --table-name development_gaLogs --endpoint-url http://localhost:8000 list a table's attributes: * aws dynamodb describe-table --table development_gaLogs_log --endpoint-url http://localhost:8000 dump a table using scan: * aws dynamodb scan --table development_gaLogs_log --endpoint-url http://localhost:8000